Faults in Large Distributed Systems and What We Can Do About Them

• Published in: Euro-Par 2005 Parallel Processing pp. 442 - 453
• Main Authors: Kola, George, Kosar, Tevfik, Livny, Miron
• Format: Book Chapter Conference Proceeding
• Language: English
• Published: Berlin, Heidelberg Springer Berlin Heidelberg 2005; Springer
• Series: Lecture Notes in Computer Science
• Subjects: Applied sciences; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Cost Curve; Exact sciences and technology; Failure Detection; Fault Model; Incorrect Result; Software; Storage Server; High performance; Model-based reasoning; Mechanism synthesis; Scientific computation; Distributed algorithm; Grid; Distributed system; Localization; Failure rate; Distributed computing; Modeling
• ISBN: 3540287000; 9783540287001
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/11549468_51

Scientists are increasingly using large distributed systems built from commodity off-the-shelf components to perform scientific computation. Grid computing has expanded the scale of such systems by spanning them across organizations. While such systems are cost-effective, the usage of large number of commodity components causes high fault and failure rates. Some of these faults result in silent data corruption leaving users with possibly incorrect results. In this work, we analyzed the faults and failures that occurred in Condor pools at UW-Madison having a few thousand CPUs and in two large distributed applications: US-CMS and BMRB BLAST, each of which used hundreds of thousands of CPU hours. We propose ‘silent-fail-stutter’ fault-model to correctly model the silent failures and detail how to handle them. Based on the model, we have designed mechanisms that automatically detect and handle silent failures and ensure that users get correct results. Our mechanisms perform automated fault location and can transparently adapt applications to avoid faulty machines. We also designed a data provenance mechanism that tracks the origin of the results, enabling scientists to selectively purge results from faulty components.