Prediction of Disk Failure Based on Classification Intensity Resampling

• Published in: Information (Basel) Vol. 15; no. 6; p. 322
• Main Authors: Wu, Sheng, Guan, Jihong
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.06.2024
• Subjects: Accuracy; Algorithms; Big Data; bucket undersampling; Classification; classification intensity; Classifiers; Data centers; Datasets; Decision trees; Disk drives; Disks; Ensemble learning; Failure; Failure rates; Fault diagnosis; Feature selection; Hard disks; Hardware; imbalanced data; Information systems; Machine learning; Neural networks; Resampling; Research methodology; secondary screening; SMOTE oversampling; Support vector machines
• ISSN: 2078-2489; 2078-2489
• DOI: 10.3390/info15060322

With the rapid growth of the data scale in data centers, the high reliability of storage is facing various challenges. Specifically, hardware failures such as disk faults occur frequently, causing serious system availability issues. In this context, hardware fault prediction based on AI and big data technologies has become a research hotspot, aiming to guide operation and maintenance personnel to implement preventive replacement through accurate prediction to reduce hardware failure rates. However, existing methods still have weaknesses in terms of accuracy due to the impacts of data quality issues such as the sample imbalance. This article proposes a disk fault prediction method based on classification intensity resampling, which fills the gap between the degree of data imbalance and the actual classification intensity of the task by introducing a base classifier to calculate the classification intensity, thus better preserving the data features of the original dataset. In addition, using ensemble learning methods such as random forests, combined with resampling, an integrated classifier for imbalanced data is developed to further improve the prediction accuracy. Experimental verification shows that compared with traditional methods, the F1-score of disk fault prediction is improved by 6%, and the model training time is also greatly reduced. The fault prediction method proposed in this paper has been applied to approximately 80 disk drives and nearly 40,000 disks in the production environment of a large bank’s data center to guide preventive replacements. Compared to traditional methods, the number of preventive replacements based on our method has decreased by approximately 21%, while the overall disk failure rate remains unchanged, thus demonstrating the effectiveness of our method.