DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks

• Published in: 2018 IEEE 25th International Conference on High Performance Computing (HiPC) pp. 42 - 51
• Main Authors: Balaprakash, Prasanna, Salim, Michael, Uram, Thomas D., Vishwanath, Venkat, Wild, Stefan M.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2018
• Subjects: Bayesian optimization; Benchmark testing; Computational modeling; Data models; deep learning; Hyperparameter search; Load modeling; model-based search; Search problems; Training
• ISSN: 2640-0316
• DOI: 10.1109/HiPC.2018.00014

Hyperparameters employed by deep learning (DL) methods play a substantial role in the performance and reliability of these methods in practice. Unfortunately, finding performance optimizing hyperparameter settings is a notoriously difficult task. Hyperparameter search methods typically have limited production-strength implementations or do not target scalability within a highly parallel machine, portability across different machines, experimental comparison between different methods, and tighter integration with workflow systems. In this paper, we present DeepHyper, a Python package that provides a common interface for the implementation and study of scalable hyperparameter search methods. It adopts the Balsam workflow system to hide the complexities of running large numbers of hyperparameter configurations in parallel on high-performance computing (HPC) systems. We implement and study asynchronous model-based search methods that consist of sampling a small number of input hyperparameter configurations and progressively fitting surrogate models over the input-output space until exhausting a user-defined budget of evaluations. We evaluate the efficacy of these methods relative to approaches such as random search, genetic algorithms, Bayesian optimization, and hyperband on DL benchmarks on CPU-and GPU-based HPC systems.