Algorithm selection for SMT MachSMT: machine learning driven algorithm selection for SMT solvers

• Published in: International journal on software tools for technology transfer Vol. 25; no. 2; pp. 219 - 239
• Main Authors: Scott, Joseph, Niemetz, Aina, Preiner, Mathias, Nejati, Saeed, Ganesh, Vijay
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.04.2023
• Subjects: Computer Science; Software Engineering; Software Engineering/Programming and Operating Systems; Theory of Computation; Algorithm selection; SMT solvers; Machine learning
• ISSN: 1433-2779; 1433-2787
• DOI: 10.1007/s10009-023-00696-0

This paper presents MachSMT, an algorithm selection tool for Satisfiability Modulo Theories (SMT) solvers. MachSMT supports the entirety of the SMT-LIB language and standardized SMT-LIB theories, and is easy to extend with support for new theories. MachSMT deploys machine learning methods to construct both empirical hardness models and pairwise ranking comparators over state-of-the-art SMT solvers. Given an input formula in SMT-LIB format, MachSMT leverages these learnt models to output a ranking of solvers based on predicted runtimes. We provide an extensive empirical evaluation of MachSMT to demonstrate the efficiency and efficacy of MachSMT over three broad usage scenarios on theories and theory combinations of practical relevance (e.g., bit-vectors, (non)linear integer and real arithmetic, arrays, and floating-point arithmetic). First, we deploy MachSMT on state-of-the-art solvers in SMT-COMP 2019 and 2020. We observe MachSMT frequently improves on the best performing solvers in the competition, winning 57 divisions outright, with up to a 99.4 % improvement in PAR-2 score. Second, we evaluate MachSMT to select configurations from a single underlying solver. We observe that MachSMT solves 898 more benchmarks and up to a 93.4 % improvement in PAR-2 score across 23 configurations of the SMT solver cvc5. Last, we evaluate MachSMT on domain-specific problems, namely network verification with simple domain-specific features, and observe an improvement of 77.3 % in PAR-2  score.