An interpretable machine learning framework for opioid overdose surveillance from emergency medical services records

• Published in: PloS one Vol. 19; no. 1; p. e0292170
• Main Authors: Graham, S. Scott, Shifflet, Savannah, Amjad, Maaz, Claborn, Kasey
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 30.01.2024; Public Library of Science (PLoS)
• Subjects: Analysis; Annotations; Benchmarks; Biology and Life Sciences; Classifiers; Computer and Information Sciences; Datasets; Drug overdose; Drugs; Emergency medical care; Emergency medical services; Emergency service; Emergency services; Harm reduction; Health services; Health surveillance; Hispanic Americans; Hospitals; Learning algorithms; Machine learning; Medical diagnosis; Medicine and Health Sciences; Narcotics; Neural networks; Opioids; Overdose; Paramedics; People and places; Public health; Statistical models; Unstructured data; Variables; United States; United States > US; Texas
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0292170

The goal of this study is to develop and validate a lightweight, interpretable machine learning (ML) classifier to identify opioid overdoses in emergency medical services (EMS) records. We conducted a comparative assessment of three feature engineering approaches designed for use with unstructured narrative data. Opioid overdose annotations were provided by two harm reduction paramedics and two supporting annotators trained to reliably match expert annotations. Candidate feature engineering techniques included term frequency-inverse document frequency (TF-IDF), a highly performant approach to concept vectorization, and a custom approach based on the count of empirically-identified keywords. Each feature set was trained using four model architectures: generalized linear model (GLM), Naïve Bayes, neural network, and Extreme Gradient Boost (XGBoost). Ensembles of trained models were also evaluated. The custom feature models were also assessed for variable importance to aid interpretation. Models trained using TF-IDF feature engineering ranged from AUROC = 0.59 (95% CI: 0.53–0.66) for the Naïve Bayes to AUROC = 0.76 (95% CI: 0.71–0.81) for the neural network. Models trained using concept vectorization features ranged from AUROC = 0.83 (95% 0.78–0.88)for the Naïve Bayes to AUROC = 0.89 (95% CI: 0.85–0.94) for the ensemble. Models trained using custom features were the most performant, with benchmarks ranging from AUROC = 0.92 (95% CI: 0.88–0.95) with the GLM to 0.93 (95% CI: 0.90–0.96) for the ensemble. The custom features model achieved positive predictive values (PPV) ranging for 80 to 100%, which represent substantial improvements over previously published EMS encounter opioid overdose classifiers. The application of this approach to county EMS data can productively inform local and targeted harm reduction initiatives.