Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

• Published in: BMJ (Online) Vol. 375; p. n2281
• Main Authors: Andaur Navarro, Constanza L, Damen, Johanna A A, Takada, Toshihiko, Nijman, Steven W J, Dhiman, Paula, Ma, Jie, Collins, Gary S, Bajpai, Ram, Riley, Richard D, Moons, Karel G M, Hooft, Lotty
• Format: Journal Article
• Language: English
• Published: England British Medical Journal Publishing Group 20.10.2021; BMJ Publishing Group LTD; BMJ Publishing Group Ltd
• Subjects: Artificial intelligence; Bias; Clinical Decision Rules; Clinical medicine; Data Interpretation, Statistical; Humans; Learning algorithms; Machine Learning; Medical research; Models, Statistical; Multivariate Analysis; Prediction models; Risk; Statistical methods; Systematic review
• ISSN: 1756-1833; 0959-8138; 1756-1833
• DOI: 10.1136/bmj.n2281

AbstractObjectiveTo assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties.DesignSystematic review.Data sourcesPubMed from 1 January 2018 to 31 December 2019.Eligibility criteriaArticles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes.Review methodsMethodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall).Results152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively.ConclusionMost studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice.Systematic review registrationPROSPERO CRD42019161764.