Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
Authors: Brian R. Bartoldson, Bhavya Kailkhura, Davis Blalock; Volume 24(122):1−77, 2023.
Abstract
In recent years, deep learning has made significant advancements. However, the increasing economic and environmental costs associated with training neural networks have become unsustainable. To tackle this issue, extensive research has focused on *algorithmically-efficient deep learning*, which aims to reduce training costs by making changes to the semantics of the training program, rather than at the hardware or implementation level. This paper provides a structured and comprehensive overview of the research in this field. The paper first formalizes the *algorithmic speedup* problem and then presents a taxonomy built upon fundamental building blocks of algorithmically efficient training. This taxonomy highlights commonalities among seemingly different methods and identifies current research gaps. Additionally, the paper presents best practices for evaluation to enable fair, comprehensive, and reliable comparisons of speedup techniques. To support research and applications, common bottlenecks in the training pipeline are discussed, illustrated through experiments, and taxonomic mitigation strategies are offered. Finally, the paper outlines unsolved research challenges and presents promising directions for future research.
[abs]