StreamDFP: A General Stream Mining Framework for Adaptive Disk Failure Prediction

• Published in: IEEE transactions on computers Vol. 72; no. 2; pp. 520 - 534
• Main Authors: Han, Shujie, Lee, Patrick P. C., Shen, Zhirong, He, Cheng, Liu, Yi, Huang, Tao
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation; Adaptation models; Algorithms; and online transfer learning; concept drift; Data transmission; Disk failure prediction; Drift; Failure; Fault tolerance; Machine learning; Machine learning algorithms; Prediction algorithms; Predictions; Predictive models; Production; Random forests; Storage systems; stream mining; Training
• ISSN: 0018-9340; 1557-9956
• DOI: 10.1109/TC.2022.3160365

We explore machine learning for accurately predicting imminent disk failures and hence providing proactive fault tolerance for modern large-scale storage systems. Current disk failure prediction approaches are mostly offline and assume that the disk logs required for training learning models are available a priori. However, disk logs are often continuously generated as an evolving data stream, in which the statistical patterns vary over time (also known as concept drift). Such a challenge motivates the need of online techniques that perform training and prediction on the incoming stream of disk logs in real time, while being adaptive to concept drift. We first measure and demonstrate the existence of concept drift on various disk models in production. Motivated by our study, we design StreamDFP , a general stream mining framework for disk failure prediction with concept-drift adaptation based on three key techniques, namely online labeling, concept-drift-aware training, and general prediction, with a primary objective of supporting various machine learning algorithms. We extend StreamDFP to support online transfer learning for minority disk models with concept-drift adaptation. Our evaluation shows that StreamDFP improves the prediction accuracy significantly compared to without concept-drift adaptation under various settings, and achieves reasonably high stream processing performance.