Job Scheduling Optimization for Multi-user MapReduce Clusters

• Published in: 2011 Fourth International Symposium on Parallel Architectures, Algorithms and Programming pp. 213 - 217
• Main Authors: Yongcai Tao, Qing Zhang, Lei Shi, Pinhua Chen
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2011
• Subjects: Benchmark testing; Educational institutions; File systems; Hadoop; HDFS; job Scheduling; MapReduce; Scheduling; Scheduling algorithm; Tin
• ISBN: 1457718081; 9781457718083
• ISSN: 2168-3034
• DOI: 10.1109/PAAP.2011.33

A shared MapReduce cluster is beneficial to build data warehouse which can be used by multiple users. FAIR scheduler gives each user the illusion of owning a private cluster. Moreover, it can dynamic redistribute capacity unused by some users to other users. However, when reassigning the slots, FAIR picks the most recently launched tasks to kill without considering the job character and data locality, which increases the network traffic while rescheduling the killed Map/Reduce tasks. The paper, based on FAIR scheduling, proposes an improved FAIR scheduling algorithm, which take into account the job character and data locality while killing tasks to make slots for new users. Performance evaluation results demonstrate that the improved FAIR decreases the data movement, speeds the execution of jobs, consequently improving the system performance.