Runtime vs Scheduler: Analyzing Dask's Overheads

• Published in: 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) pp. 1 - 8
• Main Authors: Bohm, Stanislav, Beranek, Jakub
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.11.2020
• Subjects: Computer architecture; dask; Dask central server; Dask overheads; Dask programs; data scientists; distributed task framework; distributed task scheduling; Libraries; parallel programming; Programming; programming effort; Python; Python code; random scheduler; Runtime; runtime overhead; rust; scheduler optimization; scheduling; scheduling algorithm; Servers; Task analysis; work-stealing scheduler; workflow; world task graphs
• DOI: 10.1109/WORKS51914.2020.00006

Dask is a distributed task framework which is commonly used by data scientists to parallelize Python code on computing clusters with little programming effort. It uses a sophisticated work-stealing scheduler which has been handtuned to execute task graphs as efficiently as possible. But is scheduler optimization a worthwhile effort for Dask? Our paper shows on many real world task graphs that even a completely random scheduler is surprisingly competitive with its built-in scheduler and that the main bottleneck of Dask lies in its runtime overhead. We develop a drop-in replacement for the Dask central server written in Rust which is backwards compatible with existing Dask programs. Thanks to its efficient runtime, our server implementation is able to scale up to larger clusters than Dask and consistently outperforms it on a variety of task graphs, despite the fact that it uses a simpler scheduling algorithm.