Evaluation of MRI-based machine learning approaches for computer-aided diagnosis of dementia in a clinical data warehouse

• Published in: Medical image analysis Vol. 89; p. 102903
• Main Authors: Bottani, Simona, Burgos, Ninon, Maire, Aurélien, Saracino, Dario, Ströer, Sebastian, Dormont, Didier, Colliot, Olivier
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.10.2023; Elsevier
• Subjects: Artificial Intelligence; Clinical data warehouse; Computer Science; Deep learning; Dementia; MRI; Neuroimaging; Shortcut learning; Neuroimaging; Deep learning; Clinical data warehouse; MRI; Shortcut learning; Dementia
• ISSN: 1361-8415; 1361-8423; 1361-8423
• DOI: 10.1016/j.media.2023.102903

A variety of algorithms have been proposed for computer-aided diagnosis of dementia from anatomical brain MRI. These approaches achieve high accuracy when applied to research data sets but their performance on real-life clinical routine data has not been evaluated yet. The aim of this work was to study the performance of such approaches on clinical routine data, based on a hospital data warehouse, and to compare the results to those obtained on a research data set. The clinical data set was extracted from the hospital data warehouse of the Greater Paris area, which includes 39 different hospitals. The research set was composed of data from the Alzheimer’s Disease Neuroimaging Initiative data set. In the clinical set, the population of interest was identified by exploiting the diagnostic codes from the 10th revision of the International Classification of Diseases that are assigned to each patient. We studied how the imbalance of the training sets, in terms of contrast agent injection and image quality, may bias the results. We demonstrated that computer-aided diagnosis performance was strongly biased upwards (over 17 percent points of balanced accuracy) by the confounders of image quality and contrast agent injection, a phenomenon known as the Clever Hans effect or shortcut learning. When these biases were removed, the performance was very poor. In any case, the performance was considerably lower than on the research data set. Our study highlights that there are still considerable challenges for translating dementia computer-aided diagnosis systems to clinical routine. [Display omitted] •We studied ML-based detection of dementia using brain MRI clinical routine data.•We studied patients from a clinical data warehouse identified with ICD-10 codes.•We uncovered shortcut learning (performance driven by irrelevant characteristics).•Performance was considerably lower on real-life data compared with research data.•Our work demonstrates the difficulty of translating algorithms to clinical routine.