Statistical considerations for testing an AI algorithm used for prescreening lung CT images

• Published in: Contemporary clinical trials communications Vol. 16; p. 100434
• Main Authors: Obuchowski, Nancy A., Bullen, Jennifer A.
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier Inc 01.12.2019; Elsevier
• Subjects: Area under the ROC curve; Artificial intelligence; Computer-aided detection; Diagnostic accuracy; Diagnostic accuracy studies; Other; Prescreening; Area under the ROC curve; Prescreening; Diagnostic accuracy studies; Artificial intelligence; Computer-aided detection; Diagnostic accuracy; diagnostic accuracy; computer-aided detection; prescreening; diagnostic accuracy studies; area under the ROC curve
• ISSN: 2451-8654; 2451-8654
• DOI: 10.1016/j.conctc.2019.100434

Artificial intelligence, as applied to medical images to detect, rule out, diagnose, and stage disease, has seen enormous growth over the last few years. There are multiple use cases of AI algorithms in medical imaging: first-reader (or concurrent) mode, second-reader mode, triage mode, and more recently prescreening mode as when an AI algorithm is applied to the worklist of images to identify obvious negative cases so that human readers do not need to review them and can focus on interpreting the remaining cases. In this paper we describe the statistical considerations for designing a study to test a new AI prescreening algorithm for identifying normal lung cancer screening CTs. We contrast agreement vs. accuracy studies, and retrospective vs. prospective designs. We evaluate various test performance metrics with respect to their sensitivity to changes in the AI algorithm's performance, as well as to shifts in reader behavior to a revised worklist. We consider sample size requirements for testing the AI prescreening algorithm.