Using symbolic evaluation to understand behavior in configurable software systems

• Published in: 2010 ACM/IEEE 32nd International Conference on Software Engineering Vol. 1; pp. 445 - 454
• Main Authors: Reisner, Elnatan, Song, Charles, Ma, Kin-Keung, Foster, Jeffrey S., Porter, Adam
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 01.05.2010; IEEE
• Series: ACM Conferences
• Subjects: Computing methodologies > Symbolic and algebraic manipulation; Concrete; empirical software engineering; Libraries; Measurement; Servers; Software and its engineering > Software creation and management > Software verification and validation; Software and its engineering > Software creation and management > Software verification and validation > Software defect analysis > Software testing and debugging; software configurations; Software systems; software testing and analysis; Testing; Theory of computation > Models of computation > Abstract machines; Theory of computation > Semantics and reasoning > Program reasoning > Abstraction; software configurations; empirical software engineering; software testing and analysis
• ISBN: 9781605587196; 1605587192
• ISSN: 0270-5257
• DOI: 10.1145/1806799.1806864

Many modern software systems are designed to be highly configurable, which increases flexibility but can make programs hard to test, analyze, and understand. We present an initial empirical study of how configuration options affect program behavior. We conjecture that, at certain levels of abstraction, configuration spaces are far smaller than the worst case, in which every configuration is distinct. We evaluated our conjecture by studying three configurable software systems: vsftpd, ngIRCd, and grep. We used symbolic evaluation to discover how the settings of run-time configuration options affect line, basic block, edge, and condition coverage for our subjects under a given test suite. Our results strongly suggest that for these subject programs, test suites, and configuration options, when abstracted in terms of the four coverage criteria above, configuration spaces are in fact much smaller than combinatorics would suggest and are effectively the composition of many small, self-contained groupings of options.