Bayesian k -Means as a “Maximization-Expectation” Algorithm

• Published in: Neural computation Vol. 21; no. 4; pp. 1145 - 1172
• Main Authors: Kurihara, Kenichi, Welling, Max
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.04.2009; MIT Press Journals, The
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Bayes Theorem; Bayesian analysis; Biological and medical sciences; Computer science; control theory; systems; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; Learning and adaptive systems; Letters; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Miscellaneous; Models, Neurological; Multivariate analysis; Nervous system; Parameter optimization; Probability and statistics; Sciences and techniques of general use; Statistics; Bayes estimation; Data analysis; Neural computation; Maximization; Cluster analysis (statistics); Tree structure; Neural network; Algorithm; Implementation; Assignment; Tree; Data structure; Random variable
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/neco.2008.12-06-421

We introduce a new class of “maximization-expectation” (ME) algorithms where we maximize over hidden variables but marginalize over random parameters. This reverses the roles of expectation and maximization in the classical expectation-maximization algorithm. In the context of clustering, we argue that these hard assignments open the door to very fast implementations based on data structures such as kd-trees and conga lines. The marginalization over parameters ensures that we retain the ability to infer model structure (i.e., number of clusters). As an important example, we discuss a top-down Bayesian -means algorithm and a bottom-up agglomerative clustering algorithm. In experiments, we compare these algorithms against a number of alternative algorithms that have recently appeared in the literature.