Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

• Published in: ACM transactions on knowledge discovery from data Vol. 10; no. 1; pp. 1 - 51
• Main Authors: Campello, Ricardo J. G. B., Moulavi, Davoud, Zimek, Arthur, Sander, Jörg
• Format: Journal Article
• Language: English
• Published: 01.07.2015
• Subjects: Algorithms; Clustering; Clusters; Data analysis; Data visualization; Density; Estimates; Outliers (statistics)
• ISSN: 1556-4681; 1556-472X; 1556-472X
• DOI: 10.1145/2733381

An integrated framework for density-based cluster analysis, outlier detection, and data visualization is introduced in this article. The main module consists of an algorithm to compute hierarchical estimates of the level sets of a density, following Hartigan’s classic model of density-contour clusters and trees. Such an algorithm generalizes and improves existing density-based clustering techniques with respect to different aspects. It provides as a result a complete clustering hierarchy composed of all possible density-based clusters following the nonparametric model adopted, for an infinite range of density thresholds. The resulting hierarchy can be easily processed so as to provide multiple ways for data visualization and exploration. It can also be further postprocessed so that: (i) a normalized score of “outlierness” can be assigned to each data object, which unifies both the global and local perspectives of outliers into a single definition; and (ii) a “flat” (i.e., nonhierarchical) clustering solution composed of clusters extracted from local cuts through the cluster tree (possibly corresponding to different density thresholds) can be obtained, either in an unsupervised or in a semisupervised way. In the unsupervised scenario, the algorithm corresponding to this postprocessing module provides a global, optimal solution to the formal problem of maximizing the overall stability of the extracted clusters. If partially labeled objects or instance-level constraints are provided by the user, the algorithm can solve the problem by considering both constraints violations/satisfactions and cluster stability criteria. An asymptotic complexity analysis, both in terms of running time and memory space, is described. Experiments are reported that involve a variety of synthetic and real datasets, including comparisons with state-of-the-art, density-based clustering and (global and local) outlier detection methods.