Bandwidth optimal all-reduce algorithms for clusters of workstations

• Published in: Journal of parallel and distributed computing Vol. 69; no. 2; pp. 117 - 124
• Main Authors: Patarasuk, Pitch, Yuan, Xin
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Inc 01.02.2009; Elsevier
• Subjects: All-reduce; Applied sciences; Cluster of workstations; Collective communication; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Exact sciences and technology; Software; Tree topology; All-reduce; Tree topology; Collective communication; Cluster of workstations; Lower bound; Optimal algorithm; Symmetric configuration; Calculator cluster; Distributed system; Topology; Multicore processor; Multiprocessor; Ring; Ethernet; Bandwidth; Switching networks
• ISSN: 0743-7315; 1096-0848
• DOI: 10.1016/j.jpdc.2008.09.002

We consider an efficient realization of the all-reduce operation with large data sizes in cluster environments, under the assumption that the reduce operator is associative and commutative. We derive a tight lower bound of the amount of data that must be communicated in order to complete this operation and propose a ring-based algorithm that only requires tree connectivity to achieve bandwidth optimality. Unlike the widely used butterfly-like all-reduce algorithm that incurs network contention in SMP/multi-core clusters, the proposed algorithm can achieve contention-free communication in almost all contemporary clusters, including SMP/multi-core clusters and Ethernet switched clusters with multiple switches. We demonstrate that the proposed algorithm is more efficient than other algorithms on clusters with different nodal architectures and networking technologies when the data size is sufficiently large.