Quasi-Bayes properties of a procedure for sequential learning in mixture models

• Published in: Journal of the Royal Statistical Society. Series B, Statistical methodology Vol. 82; no. 4; pp. 1087 - 1114
• Main Authors: Fortini, Sandra, Petrone, Sonia
• Format: Journal Article
• Language: English
• Published: Oxford Wiley 01.09.2020; Oxford University Press
• Subjects: Algorithms; Asymptotic exchangeability; Asymptotic properties; Bayesian analysis; Bayesian non‐parametrics; Bayesian theory; Computer simulation; Conditionally identically distributed sequences; Dirichlet process; Learning; Machine learning; Original Articles; Predictive distributions; Probabilistic models; Questions; Recursive learning; Recursive methods; Regression analysis; Simulation; Statistical analysis; Statistical methods; Statistical models; Statistics
• ISSN: 1369-7412; 1467-9868
• DOI: 10.1111/rssb.12385

Bayesian methods are often optimal, yet increasing pressure for fast computations, especially with streaming data, brings renewed interest in faster, possibly suboptimal, solutions. The extent to which these algorithms approximate Bayesian solutions is a question of interest, but often unanswered. We propose a methodology to address this question in predictive settings, when the algorithm can be reinterpreted as a probabilistic predictive rule. We specifically develop the proposed methodology for a recursive procedure for on-line learning in non-parametric mixture models, which is often referred to as Newton’s algorithm. This algorithm is simple and fast; however, its approximation properties are unclear. By reinterpreting it as a predictive rule, we can show that it underlies a statistical model which is, asymptotically, a Bayesian, exchangeable mixture model. In this sense, the recursive rule provides a quasi-Bayes solution. Although the algorithm offers only a point estimate, our clean statistical formulation enables us to provide the asymptotic posterior distribution and asymptotic credible intervals for the mixing distribution. Moreover, it gives insights for tuning the parameters, as we illustrate in simulation studies, and paves the way to extensions in various directions. Beyond mixture models, our approach can be applied to other predictive algorithms.