Gradient Descent Effects on Differential Neural Architecture Search: A Survey

• Published in: IEEE access Vol. 9; pp. 89602 - 89618
• Main Authors: Santra, Santanu, Hsieh, Jun-Wei, Lin, Chi-Fang
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; back-propagation; Computational modeling; Computer architecture; Computing costs; evolutionary algorithm; Evolutionary algorithms; Gradient descent; Machine learning; neural architecture search; Performance evaluation; Reinforcement learning; Search methods; Search problems; Search process; Task analysis; Training
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2021.3090918

Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.