Coverage-based regression test case selection, minimization and prioritization: a case study on an industrial system

• Published in: Software testing, verification & reliability Vol. 25; no. 4; pp. 371 - 396
• Main Authors: Nardo, Daniel Di, Alshahwan, Nadia, Briand, Lionel, Labiche, Yvan
• Format: Journal Article
• Language: English
• Published: Chichester Blackwell Publishing Ltd 01.06.2015; Wiley Subscription Services, Inc
• Subjects: Cost engineering; Criteria; Fault detection; Faults; industrial case study; Minimization; Optimization; Regression; regression testing; test case prioritization; test case selection; test suite minimization; Tradeoffs
• ISSN: 0960-0833; 1099-1689
• DOI: 10.1002/stvr.1572

SummaryThis paper presents a case study of coverage‐based regression testing techniques on a real world industrial system with real regression faults. The study evaluates four common prioritization techniques, a test selection technique, a test suite minimization technique and a hybrid approach that combines selection and minimization. The study also examines the effects of using various coverage criteria on the effectiveness of the studied approaches. The results show that prioritization techniques that are based on additional coverage with finer grained coverage criteria perform significantly better in fault detection rates. The study also reveals that using modification information in prioritization techniques does not significantly enhance fault detection rates. The results show that test selection does not provide significant savings in execution cost (<2%), which might be attributed to the nature of the changes made to the system. Test suite minimization using finer grained coverage criteria could provide significant savings in execution cost (79.5%) while maintaining a fault detection capability level above 70%, thus representing a possible trade‐off. The hybrid technique did not provide a significant improvement over traditional minimization techniques. Copyright © 2015 John Wiley & Sons, Ltd. This paper presents a case study of coverage‐based regression testing techniques on a real‐world industrial system with real regression faults. The results show that prioritization techniques that are based on additional coverage with finer‐grained coverage criteria perform significantly better in fault detection rates, test selection does not provide significant savings in execution cost (<2%), and test suite minimization using finer‐grained coverage criteria could provide significant savings in execution cost (79.5%) while maintaining a fault detection capability level above 70%.