Surrogate maximization/minimization algorithms and extensions

• Published in: Machine learning Vol. 69; no. 1; pp. 1 - 33
• Main Authors: Zhang, Zhihua, Kwok, James T., Yeung, Dit-Yan
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer 01.10.2007; Springer Nature B.V
• Subjects: Applied sciences; Artificial intelligence; Computer science; control theory; systems; Exact sciences and technology; Studies; Gradient; Loglinear model; Minimization; Regression analysis; Jensen inequality; Aggregate model; Logistic regression; Surrogate function,Convexity,Logistic regression,AdaBoost; Supervised learning; Regression model; Classification; Quadratic approximation; Regression function; Convexity; Objective function; EM algorithm; Artificial intelligence; Log-linear model; Step function
• ISSN: 0885-6125; 1573-0565; 1573-0565
• DOI: 10.1007/s10994-007-5022-x

Surrogate maximization (or minimization) (SM) algorithms are a family of algorithms that can be regarded as a generalization of expectation-maximization (EM) algorithms. An SM algorithm aims at turning an otherwise intractable maximization problem into a tractable one by iterating two steps. The S-step computes a tractable surrogate function to substitute the original objective function and the M-step seeks to maximize this surrogate function. Convexity plays a central role in the S-step. SM algorithms enjoy the same convergence properties as EM algorithms. There are mainly three approaches to the construction of surrogate functions, namely, by using Jensen's inequality, first-order Taylor approximation, and the low quadratic bound principle. In this paper, we demonstrate the usefulness of SM algorithms by taking logistic regression models, AdaBoost and the log-linear model as examples. More specifically, by using different surrogate function construction methods, we devise several SM algorithms, including the standard SM, generalized SM, gradient SM, and quadratic SM algorithms, and their two variants called the conditional surrogate maximization (CSM) and surrogate conditional maximization (SCM) algorithms. [PUBLICATION ABSTRACT]