Integrating Bayesian Optimization and Machine Learning for the Optimal Configuration of Cloud Systems

• Published in: IEEE transactions on cloud computing Vol. 12; no. 1; pp. 277 - 294
• Main Authors: Guindani, Bruno, Ardagna, Danilo, Guglielmi, Alessandra, Rocco, Roberto, Palermo, Gianluca
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Acquisition function; Algorithms; Bayes methods; Bayesian analysis; Bayesian optimization; Black boxes; black-box optimization; Closed box; Cloud computing; Computational modeling; Configurations; Costs; Edge computing; Machine learning; Optimization; Software algorithms
• ISSN: 2168-7161; 2372-0018; 2372-0018
• DOI: 10.1109/TCC.2024.3361070

Bayesian Optimization (BO) is an efficient method for finding optimal cloud configurations for several types of applications. On the other hand, Machine Learning (ML) can provide helpful knowledge about the application at hand thanks to its predicting capabilities. This work proposes a general approach based on BO, which integrates elements from ML techniques in multiple ways, to find an optimal configuration of recurring jobs running in public and private cloud environments, possibly subject to black-box constraints, e.g., application execution time or accuracy. We test our approach by considering several use cases, including edge computing, scientific computing, and Big Data applications. Results show that our solution outperforms other state-of-the-art black-box techniques, including classical autotuning and BO- and ML-based algorithms, reducing the number of unfeasible executions and corresponding costs up to 2-4 times.