AutoML: A survey of the state-of-the-art

• Published in: Knowledge-based systems Vol. 212; p. 106622
• Main Authors: He, Xin, Zhao, Kaiyong, Chu, Xiaowen
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 05.01.2021; Elsevier Science Ltd
• Subjects: Algorithms; Automated machine learning (autoML); Deep learning; Hyperparameter optimization (HPO); Machine learning; Neural architecture search (NAS); Object recognition; Optimization; State-of-the-art reviews; Deep learning; Neural architecture search (NAS); Automated machine learning (autoML); Hyperparameter optimization (HPO)
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2020.106622

Deep learning (DL) techniques have obtained remarkable achievements on various tasks, such as image recognition, object detection, and language modeling. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering its wide application. Meanwhile, automated machine learning (AutoML) is a promising solution for building a DL system without human assistance and is being extensively studied. This paper presents a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. According to the DL pipeline, we introduce AutoML methods – covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS) – with a particular focus on NAS, as it is currently a hot sub-topic of AutoML. We summarize the representative NAS algorithms’ performance on the CIFAR-10 and ImageNet datasets and further discuss the following subjects of NAS methods: one/two-stage NAS, one-shot NAS, joint hyperparameter and architecture optimization, and resource-aware NAS. Finally, we discuss some open problems related to the existing AutoML methods for future research.