FuzzBoost: Reinforcement Compiler Fuzzing

• Published in: Information and Communications Security Vol. 13407; pp. 359 - 375
• Main Authors: Li, Xiaoting, Liu, Xiao, Chen, Lingwei, Prajapati, Rupesh, Wu, Dinghao
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2022; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Compilers; Fuzzing; Reinforcement learning
• ISBN: 9783031157769; 3031157761
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-15777-6_20

Enforcing the correctness of compilers is important for the current computing systems. Fuzzing is an efficient way to find security vulnerabilities in software by repeatedly testing programs with enormous modified, or fuzzed input data. However, in the context of compilers, fuzzing is challenging because the inputs are pieces of code that are required to be both syntactically and semantically valid to pass front-end checks. Also, the fuzzed inputs are expected to be distinct enough to trigger abnormal crashes, memory leaks, or failing assertions that have not been detected before. In this paper, we formalize compiler fuzzing as a reinforcement learning problem and propose an automatic code synthesis framework called FuzzBoost to empower the input code mutations in the fuzzing process. In our learning system, we incorporate the deep Q-learning algorithm to perform multi-step code mutations in each training episode, and design a reward policy to assess the testing coverage information collected at runtime. By interacting with the system, the fuzzing agent learns to predict code mutation actions that maximizing the fuzzing rewards. We validate the effectiveness of our proposed approach and the preliminary evidence shows that our reinforcement fuzzing method can outperform the fuzzing baseline on production compilers. Our results also show that a pre-trained model can boost the fuzzing process for seed programs with similar patterns.