An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs

• Published in: Applied sciences Vol. 14; no. 3; p. 1046
• Main Authors: Liu, Mingxing, Wang, Junfeng, Lin, Tao, Ma, Quan, Fang, Zhiyang, Wu, Yanqun
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.01.2024
• Subjects: Algorithms; Automobile industry; Aviation; Case studies; Chatbots; code generation; Computational linguistics; Datasets; Efficiency; Engineering; generative pre-training (GPT); Language; Language processing; large language models (LLMs); Natural language interfaces; Natural language processing; Nuclear energy; Nuclear industry; prompt engineering; Python; safety-critical software; Software; Software development; User interface
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app14031046

In the digital era of increasing software complexity, improving the development efficiency of safety-critical software is a challenging task faced by academia and industry in domains such as nuclear energy, aviation, the automotive industry, and rail transportation. Recently, people have been excited about using pre-trained large language models (LLMs) such as ChatGPT and GPT-4 to generate code. Professionals in the safety-critical software field are intrigued by the code generation capabilities of LLMs. However, there is currently a lack of systematic case studies in this area. Aiming at the need for automated code generation in safety-critical domains such as nuclear energy and the automotive industry, this paper conducts a case study on generating safety-critical software code using GPT-4 as the tool. Practical engineering cases from the industrial domain are employed. We explore different approaches, including code generation based on overall requirements, specific requirements, and augmented prompts. We propose a novel prompt engineering method called Prompt-FDC that integrates basic functional requirements, domain feature generalization, and domain constraints. This method improves code completeness from achieving 30% functions to 100% functions, increases the code comment rate to 26.3%, and yields better results in terms of code compliance, readability, and maintainability. The code generation approach based on LLMs also introduces a new software development process and V-model lifecycle for safety-critical software. Through systematic case studies, we demonstrate that, with appropriate prompt methods, LLMs can auto-generate safety-critical software code that meets practical engineering application requirements. It is foreseeable that LLMs can be applied to various engineering domains to improve software safety and development efficiency.