A data-driven robust optimization algorithm for black-box cases: An application to hyper-parameter optimization of machine learning algorithms

• Published in: Computers & industrial engineering Vol. 160; p. 107581
• Main Authors: Seifi, Farshad, Azizi, Mohammad Javad, Akhavan Niaki, Seyed Taghi
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.10.2021
• Subjects: Bayesian optimization; Black-box optimization; Data-driven optimization; Deep learning; Gaussian process; Hyper-parameter tuning; Robust optimization; Deep learning; Robust optimization; Gaussian process; Hyper-parameter tuning; Black-box optimization; Bayesian optimization; Data-driven optimization
• ISSN: 0360-8352; 1879-0550
• DOI: 10.1016/j.cie.2021.107581

[Display omitted] •A novel Black-Box data-driven robust optimization approach is proposed.•A Gaussian process is used in a Bayesian optimization framework to design the approach.•The approach is consistent with the data in a predefined confidence level.•A hyper-parameter optimization for deep learning is investigated as an application.•The optimal hyper-parameters are robust with respect to noise. The huge availability of data in the last decade has raised the opportunity for the better use of data in decision-making processes. The idea of using the existing data to achieve a more coherent reality solution has led to a branch of optimization called data-driven optimization. On the one hand, the presence of uncertain variables in these datasets makes it crucial to design robust optimization methods in this area. On the other hand, in many real-world problems, the closed-form of the objective function is not available and a meta-model based framework is necessary. Motivated by the above points, in this paper a Gaussian process is used in a Bayesian optimization framework to design a method that is consistent with the data in a predefined confidence level. The advantage of the proposed method is that it is computationally tractable in addition to being robust and independent of the objective function’s form. As one of the applications of the proposed algorithm, hyper-parameter optimization for deep learning is investigated. The proposed method can help find the optimal hyper-parameters that are robust with respect to noise.