Stochastic Programming Approach for Resource Selection Under Demand Uncertainty

• Published in: Job Scheduling Strategies for Parallel Processing Vol. 11332; pp. 107 - 126
• Main Authors: Bhuiyan, Tanveer Hossain, Halappanavar, Mahantesh, Friese, Ryan D., Medal, Hugh, de la Torre, Luis, Sathanur, Arun, Tallent, Nathan R.
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2019; Springer International Publishing
• Series: Lecture Notes in Computer Science
• ISBN: 9783030106317; 3030106314
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-10632-4_6

Cost-efficient selection and scheduling of a subset of geographically distributed resources to meet the demands of a scientific workflow is a challenging problem. The problem is exacerbated by uncertainties in demand and availability of resources. In this paper, we present a stochastic optimization based framework for robust decision making in the selection of distributed resources over a planning horizon under demand uncertainty. We present a novel two-stage stochastic programming model for resource selection, and implement an L-shaped decomposition algorithm to solve this model. A Sample Average Approximation algorithm is integrated to enable stochastic optimization to solve problems with a large number of scenarios. Using the metric of stochastic solution, we demonstrate up to 30% cost reduction relative to solutions without explicit consideration of demand uncertainty for a 24-month problem. We also demonstrate up to 54% cost reduction relative to a previously developed solution for a 36-month problem. We further argue that the composition of resources selected is superior to solutions computed without explicit consideration of uncertainties. Given the importance of resource selection and scheduling of complex scientific workflows, especially in the context of commercial cloud computing, we believe that our novel stochastic programming framework will benefit many researchers as well as users of distributed computing resources.