torcpy: Supporting task parallelism in Python

• Published in: SoftwareX Vol. 12; p. 100517
• Main Authors: Hadjidoukas, P.E., Bartezzaghi, A., Scheidegger, F., Istrate, R., Bekas, C., Malossi, A.C.I.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.07.2020; Elsevier
• Subjects: MPI; Multithreading; Parallelism; Python; Parallelism; Multithreading; MPI; Python
• ISSN: 2352-7110; 2352-7110
• DOI: 10.1016/j.softx.2020.100517

Task-based parallelism has been established as one of the main forms of code parallelization, where asynchronous tasks are launched and distributed across the processing units of a local machine, a cluster or a supercomputer. The tasks can be either completely decoupled, corresponding to a set of independent jobs, or be part of an iterative algorithm where the task results are processed and drive the next step. Typical use cases include the application of the same function to different data, parametric searches and algorithms used in numerical optimization and Bayesian uncertainty quantification. In this work, we introduce torcpy, a platform-agnostic adaptive load balancing library that orchestrates the asynchronous execution of tasks, expressed as callables with arguments, on both shared and distributed memory platforms. The library is implemented on top of MPI and multithreading and provides lightweight support for nested loops and map functions. Experimental results using representative applications demonstrate the flexibility and efficiency of the proposed Python package.