VGrADS enabling e-Science workflows on grids and clouds with fault tolerance

• Published in: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis pp. 1 - 12
• Main Authors: Ramakrishnan, Lavanya, Koelbel, Charles, Kee, Yang-Suk, Wolski, Rich, Nurmi, Daniel, Gannon, Dennis, Obertelli, Graziano, YarKhan, Asim, Mandal, Anirban, Huang, T. Mark, Thyagaraja, Kiran, Zagorodnov, Dmitrii
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 14.11.2009
• Series: ACM Conferences
• Subjects: Cloud computing; Computational modeling; Engines; Fault tolerance; Fault tolerant systems; General and reference > Cross-computing tools and techniques > Performance; Lead; Portals; Schedules; Software and its engineering > Software organization and properties > Contextual software domains > Operating systems > Process management; Software and its engineering > Software organization and properties > Extra-functional properties > Software performance; Software and its engineering > Software organization and properties > Software system structures > Distributed systems organizing principles; Virtual machines; Weather forecasting
• ISBN: 1605587443; 9781605587448
• ISSN: 2167-4329
• DOI: 10.1145/1654059.1654107

Today's scientific workflows use distributed heterogeneous resources through diverse grid and cloud interfaces that are often hard to program. In addition, especially for time-sensitive critical applications, predictable quality of service is necessary across these distributed resources. VGrADS' virtual grid execution system (vgES) provides an uniform qualitative resource abstraction over grid and cloud systems. We apply vgES for scheduling a set of deadline sensitive weather forecasting workflows. Specifically, this paper reports on our experiences with (1) virtualized reservations for batchqueue systems, (2) coordinated usage of TeraGrid (batch queue), Amazon EC2 (cloud), our own clusters (batch queue) and Eucalyptus (cloud) resources, and (3) fault tolerance through automated task replication. The combined effect of these techniques was to enable a new workflow planning method to balance performance, reliability and cost considerations. The results point toward improved resource selection and execution management support for a variety of e-Science applications over grids and cloud systems.