An SMDP approach for Reinforcement Learning in HPC cluster schedulers

• Published in: Future generation computer systems Vol. 139; pp. 239 - 252
• Main Authors: de Freitas Cunha, Renato Luiz, Chaimowicz, Luiz
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.02.2023
• Subjects: Deep Reinforcement Learning; Machine Learning; Scheduling; Semi-Markov Decision Processes; Simulation; Workload traces; Deep Reinforcement Learning; Scheduling; Simulation; Semi-Markov Decision Processes; Machine Learning; Workload traces
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2022.09.025

Deep reinforcement learning applied to computing systems has shown potential for improving system performance, as well as faster discovery of better allocation strategies. In this paper, we map HPC batch job scheduling to the SMDP formalism, and present an online, deep reinforcement learning-based solution that uses a modification of the Proximal Policy Optimization algorithm for minimizing job slowdown with action masking, supporting large action spaces. In our experiments, we assess the effects of noise in run time estimates in our model, evaluating how it behaves in small (64 processors) and large (16384 processors) clusters. We also show our model is robust to changes in workload and in cluster sizes, showing transfer works with changes of cluster size of up to 10×, and changes from synthetic workload generators to supercomputing workload traces. In our experiments, the proposed model outperforms learning models from the literature and classic heuristics, making it a viable modeling approach for robust, transferable, learning scheduling models. •An SMDP formulation of HPC job scheduling that minimizes average bounded slowdown.•Evaluation of the effects of noise in run time estimates in small and large clusters.•Evaluation of the SMDP model with traces from real supercomputing workloads.•A comparison of agent performance with models from the literature.