Is ChatGPT ‘ready’ to be a learning tool for medical undergraduates and will it perform equally in different subjects? Comparative study of ChatGPT performance in tutorial and case-based learning questions in physiology and biochemistry

• Published in: Medical teacher Vol. 46; no. 11; pp. 1441 - 1447
• Main Authors: Luke, W. A. Nathasha V., Seow Chong, Lee, Ban, Kenneth H., Wong, Amanda H., Zhi Xiong, Chen, Shuh Shing, Lee, Taneja, Reshma, Samarasekera, Dujeepa D., Yap, Celestial T.
• Format: Journal Article
• Language: English
• Published: England Taylor & Francis Ltd 01.11.2024
• Subjects: Answers; Biochemistry; Biochemistry - education; Blooms taxonomy; Chatbots; Classification; College students; Comparative Analysis; Comparative Education; Comparative studies; Computer-Assisted Instruction - methods; Education, Medical, Undergraduate - methods; Educational Measurement - methods; Examiners; Humans; Influence of Technology; Learning; Physiology; Physiology - education; Problem-Based Learning - methods; Students, Medical - psychology; Taxonomy; Teaching; Technology; Undergraduate students; GPT-3.5; ChatGPT; physiology biochemistry; LLM (large language model); GPT-4 generative AI (artificial intelligence)
• ISSN: 0142-159X; 1466-187X; 1466-187X
• DOI: 10.1080/0142159X.2024.2308779

Generative AI will become an integral part of education in future. The potential of this technology in different disciplines should be identified to promote effective adoption. This study evaluated the performance of ChatGPT in tutorial and case-based learning questions in physiology and biochemistry for medical undergraduates. Our study mainly focused on the performance of GPT-3.5 version while a subgroup was comparatively assessed on GPT-3.5 and GPT-4 performances. Answers were generated in GPT-3.5 for 44 modified essay questions (MEQs) in physiology and 43 MEQs in biochemistry. Each answer was graded by two independent examiners. Subsequently, a subset of 15 questions from each subject were selected to represent different score categories of the GPT-3.5 answers; responses were generated in GPT-4, and graded. The mean score for physiology answers was 74.7 (SD 25.96). GPT-3.5 demonstrated a statistically significant (  = .009) superior performance in lower-order questions of Bloom's taxonomy in comparison to higher-order questions. Deficiencies in the application of physiological principles in clinical context were noted as a drawback. Scores in biochemistry were relatively lower with a mean score of 59.3 (SD 26.9) for GPT-3.5. There was no statistically significant difference in the scores for higher and lower-order questions of Bloom's taxonomy. The deficiencies highlighted were lack of in-depth explanations and precision. The subset of questions where the GPT-4 and GPT-3.5 were compared demonstrated a better overall performance in GPT-4 responses in both subjects. This difference between the GPT-3.5 and GPT-4 performance was statistically significant in biochemistry but not in physiology. The differences in performance across the two versions, GPT-3.5 and GPT-4 across the disciplines are noteworthy. Educators and students should understand the strengths and limitations of this technology in different fields to effectively integrate this technology into teaching and learning.