On-line expectation-maximization algorithm for latent data models

• Published in: Journal of the Royal Statistical Society. Series B, Statistical methodology Vol. 71; no. 3; pp. 593 - 613
• Main Authors: Cappé, Olivier, Moulines, Eric
• Format: Journal Article
• Language: English
• Published: Oxford, UK Oxford, UK : Blackwell Publishing Ltd 01.06.2009; Blackwell Publishing Ltd; Blackwell Publishing; Blackwell; Royal Statistical Society; Oxford University Press
• Series: Journal of the Royal Statistical Society Series B
• Subjects: Adaptive algorithms; Algorithms; Approximation; Convergence; Data models; Distribution theory; Exact sciences and technology; Expectation-maximization; General topics; Latent data models; Linear regression; Mathematical analysis; Mathematical models; Mathematics; Matrices; Maximum likelihood estimation; Measure and integration; Mixture of regressions; On-line estimation; Parametric inference; Parametric models; Polyak-Ruppert averaging; Probabilities; Probability and statistics; probability distribution; Probability theory and stochastic processes; Recursion; Regression analysis; Sciences and techniques of general use; Statistical methods; Statistical models; Statistics; Stochastic approximation; Stochastic models; Studies; Conditional distribution; Divergence; Adaptive algorithm; Stochastic approximation; Latent data models; On-line estimation; Linear model; Convergence; Hypothesis test; Statistical test; Sequential method; Marginal distribution; Distribution function; Mixed distribution; Integration; Expectation―maximization; Maximization; Mixture of; Linear regression; Adaptive algorithms; Latent variable model; regressions; Independent variable; Statistical method; Statistical regression; Polyak―Ruppert averaging; Regression model; Maximum likelihood; Expectation; EM algorithm
• ISSN: 1369-7412; 1467-9868; 1467-9868
• DOI: 10.1111/j.1467-9868.2009.00698.x

We propose a generic on-line (also sometimes called adaptive or recursive) version of the expectation-maximization (EM) algorithm applicable to latent variable models of independent observations. Compared with the algorithm of Titterington, this approach is more directly connected to the usual EM algorithm and does not rely on integration with respect to the complete-data distribution. The resulting algorithm is usually simpler and is shown to achieve convergence to the stationary points of the Kullback-Leibler divergence between the marginal distribution of the observation and the model distribution at the optimal rate, i.e. that of the maximum likelihood estimator. In addition, the approach proposed is also suitable for conditional (or regression) models, as illustrated in the case of the mixture of linear regressions model.