Hyperparameter Optimization EM Algorithm via Bayesian Optimization and Relative Entropy

• Published in: Entropy (Basel, Switzerland) Vol. 27; no. 7; p. 678
• Main Authors: Zou, Dawei, Ma, Chunhua, Wang, Peng, Geng, Yanqiu
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 25.06.2025; MDPI
• Subjects: Algorithms; Approximation; Automation; Bayesian analysis; Data mining; EM algorithm; Entropy; evidence function; Genetic algorithms; hyperparameters optimization; Machine learning; Maximization; Neural networks; Optimization; Optimization techniques; relative entropy; Statistical analysis; Variables; Germany; evidence function; hyperparameters optimization; EM algorithm; relative entropy
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e27070678

Hyperparameter optimization (HPO), which is also called hyperparameter tuning, is a vital component of developing machine learning models. These parameters, which regulate the behavior of the machine learning algorithm and cannot be directly learned from the given training data, can significantly affect the performance of the model. In the context of relevance vector machine hyperparameter optimization, we have used zero-mean Gaussian weight priors to derive iterative equations through evidence function maximization. For a general Gaussian weight prior and Bayesian linear regression, we similarly derive iterative reestimation equations for hyperparameters through evidence function maximization. Subsequently, after using relative entropy and Bayesian optimization, the aforementioned non-closed-form reestimation equations can be partitioned into E and M steps, providing a clear mathematical and statistical explanation for the iterative reestimation equations of hyperparameters. The experimental result shows the effectiveness of the EM algorithm of hyperparameter optimization, and the algorithm also has the merit of fast convergence, except that the covariance of the posterior distribution is a singular matrix, which affects the increase in the likelihood.