Detecting Compiler Error Recovery Defects via Program Mutation Exploration

• Published in: IEEE transactions on software engineering Vol. 51; no. 2; pp. 389 - 412
• Main Authors: Tang, Yixuan, Zhang, Jingxuan, Li, Xiaochen, Huang, Zhiqiu, Jiang, He
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2025; IEEE Computer Society
• Subjects: Codes; Compiler testing; Compilers; Computer languages; Computer programming; Configurations; Defects; differential testing; Error detection; Error recovery; Internet; Java; Mutation; Periodic structures; program mutation; Program processors; Programming; Recovery; Software; Software development; Software development management; test program generation; Testing
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2024.3510912

Compiler error recovery diagnostics facilitates software development as it provides the possible causes and suggestions on potential programming errors. However, due to compiler bugs, error recovery diagnostics could be erroneous, spurious, missing, or even crashing for mature production compilers like GCC and Clang. Compiler testing is one of the most widely used ways of ensuring its quality. However, existing compiler diagnostics testing approaches (e.g., DIPROM) only consider the typically syntactically valid test programs as inputs, which are unlikely to trigger compiler error recovery defects. Therefore, in this paper, we propose the first mutation based approach for Compiler Error Recovery diagnostics Testing, called CERTest. Specifically, CERTest first explores the mutation space for a given seed program, and leverages a series of mutation configurations (which are referred as a series of mutators applying for a seed) to iteratively mutate the structures of the seed, so as to generate error-sensitive program variants for triggering compiler error recovery mechanisms. To effectively construct error-sensitive structures, CERTest then applies a novel furthest-first based selection approach to select a set of representative mutation configurations to generate program variants in each iteration. With the generated program variants, CERTest finally leverages differential testing to detect error recovery defects in different compilers. The experiments on GCC and Clang demonstrate that CERTest outperforms five state-of-the-art approaches (i.e., DIPROM, Ccoft , Clang-fuzzer , AFL++, and HiCOND) by up to 13.10%<inline-formula><tex-math notation="LaTeX">\sim</tex-math> <mml:math><mml:mo>∼</mml:mo></mml:math><inline-graphic xlink:href="jiang-ieq1-3510912.gif"/> </inline-formula>221.61% on average in the term of bug-finding capability, and CERTest detects 9 new error recovery defects, 5 of which have been confirmed or fixed by developers.