AnnotationGym: A Generic Framework for Automatic Source Code Annotation

• Published in: IEEE access Vol. 13; pp. 155321 - 155339
• Main Authors: Shahzad, Hafsah, Sanaullah, Ahmed, Arora, Sanjay, Drepper, Ulrich, Herbordt, Martin C.
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Annotations; Benchmark testing; Best practice; Code annotation; Codes; compiler hints; Compilers; Field programmable gate arrays; Hardware; Machine learning; Optimization; Optimization algorithms; reinforcement learning; Runtime; Semantics; Source code; Source coding; source-to-source code optimization; Space exploration
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2025.3605852

A common approach to code optimization is to insert compiler hints in the source code using annotations. Two major challenges with using annotations effectively are their complexity and lack of portability. This means, first, that significant developer expertise is required, and, second, that the supported annotations, as well as their syntax and use, can vary substantially. Moreover, there is not currently any tool that can output performant annotation-inserted codes for different back-ends. To address these challenges, we present AnnotationGym, an easy-to-use, open-source, generic infrastructure that supplements or replaces the developer in annotating source code. It demonstrates a novel application of AI methods to code annotation. In addition to improving code performance, the flexibility of AnnotationGym enables easy comparisons of performance and optimization strategies among compilers and target architectures and thus provides an extensible platform to facilitate further progress in this field. AnnotationGym automatically extracts structured information about the target code and compiler to generate a list of possible annotations. AI-based optimization algorithms then traverse this space to determine the best set of annotations depending on the developer goals. To demonstrate its effectiveness, we run AnnotationGym on popular, representative workloads from the Polybench suite, as well as targeting various compilers (GCC, AMD HLS, Intel HLS), optimization algorithms (Reinforcement Learning, Bayesian Optimization), and architectures (CPU, FPGA). We also test our approach on FPGA codes derived, e.g., from the Rodinia and OpenDwarfs benchmarks and that are hand-optimized using standard best practices. An interesting finding is that the best overall performance obtained by AnnotationGym was generally with unoptimized codes.