A scalable algorithm for single-linkage hierarchical clustering on distributed-memory architectures

Hierarchical clustering is a fundamental and widely-used clustering algorithm with many advantages over traditional partitional clustering. Due to the explosion in size of modern scientific datasets, there is a pressing need for scalable analytics algorithms, but good scaling is difficult to achieve...

Full description

Saved in:
Bibliographic Details
Published in2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV) pp. 7 - 13
Main Authors Hendrix, William, Palsetia, Diana, Ali Patwary, Md Mostofa, Agrawal, Ankit, Wei-keng Liao, Choudhary, Alok
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.10.2013
Subjects
Arrays
Clustering algorithms
D.1.3 [Programming Techniques]: Concurrent Programming-Parallel Programming
Educational institutions
I.5.3 [Information Systems Applications]: Clustering-Algorithms
Merging
Parallel algorithms
Partitioning algorithms
Vegetation
Online AccessGet full text
DOI10.1109/LDAV.2013.6675153

Cover

More Information
Summary:Hierarchical clustering is a fundamental and widely-used clustering algorithm with many advantages over traditional partitional clustering. Due to the explosion in size of modern scientific datasets, there is a pressing need for scalable analytics algorithms, but good scaling is difficult to achieve for hierarchical clustering due to data dependencies inherent in the algorithm. To the best of our knowledge, no previous work on parallel hierarchical clustering has shown scalability beyond a couple hundred processes. In this paper, we present PINK, a scalable parallel algorithm for single-linkage hierarchical clustering based on decomposing a problem instance into two different types of subproblems. Despite the heterogeneous workloads, our algorithm exhibits good load balancing, as well as low memory requirements and a communication pattern that is both low-volume and deterministic. Evaluating PINK on up to 6050 processes, we find that it achieves speedups up to approximately 6600.
DOI:10.1109/LDAV.2013.6675153