Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis

• Published in: Ocular immunology and inflammation Vol. 32; no. 8; p. 1526
• Main Authors: Rojas-Carabali, William, Cifuentes-González, Carlos, Wei, Xin, Putera, Ikhwanuliman, Sen, Alok, Thng, Zheng Xian, Agrawal, Rajdeep, Elze, Tobias, Sobrin, Lucia, Kempen, John H, Lee, Bernett, Biswas, Jyotirmay, Nguyen, Quan Dong, Gupta, Vishali, de-la-Torre, Alejandra, Agrawal, Rupesh
• Format: Journal Article
• Language: English
• Published: England 01.10.2024
• Subjects: Artificial Intelligence; Diagnosis, Differential; Diagnostic Techniques, Ophthalmological - standards; Disease Management; Female; Humans; Male; Ophthalmologists - standards; Ophthalmology - standards; Reproducibility of Results; Uveitis - diagnosis; Uveitis - therapy; chatbot; diagnosis; ophthalmology; natural language processing; Artificial intelligence
• ISSN: 1744-5078; 1744-5078
• DOI: 10.1080/09273948.2023.2253471

Accurate diagnosis and timely management are vital for favorable uveitis outcomes. Artificial Intelligence (AI) holds promise in medical decision-making, particularly in ophthalmology. Yet, the diagnostic precision and management advice from AI-based uveitis chatbots lack assessment. We appraised diagnostic accuracy and management suggestions of an AI-based chatbot, ChatGPT, versus five uveitis-trained ophthalmologists, using 25 standard cases aligned with new Uveitis Nomenclature guidelines. Participants predicted likely diagnoses, two differentials, and next management steps. Comparative success rates were computed. Ophthalmologists excelled (60-92%) in likely diagnosis, exceeding AI (60%). Considering fully and partially accurate diagnoses, ophthalmologists achieved 76-100% success; AI attained 72%. Despite an 8% AI improvement, its overall performance lagged. Ophthalmologists and AI agreed on diagnosis in 48% cases, with 91.6% exhibiting concurrence in management plans. The study underscores AI chatbots' potential in uveitis diagnosis and management, indicating their value in reducing diagnostic errors. Further research is essential to enhance AI chatbot precision in diagnosis and recommendations.