Expert-augmented machine learning

• Published in: Proceedings of the National Academy of Sciences - PNAS Vol. 117; no. 9; pp. 4571 - 4577
• Main Authors: Gennatas, Efstathios D., Friedman, Jerome H., Ungar, Lyle H., Pirracchio, Romain, Eaton, Eric, Reichmann, Lara G., Interian, Yannet, Luna, José Marcio, Simone, Charles B., Auerbach, Andrew, Delgado, Elier, van der Laan, Mark J., Solberg, Timothy D., Valdes, Gilmer
• Format: Journal Article
• Language: English
• Published: United States National Academy of Sciences 03.03.2020
• Subjects: Automation; Biological Sciences; Human performance; Learning algorithms; Machine learning; Physical Sciences; Risk assessment; machine learning; computational medicine; medicine
• ISSN: 0027-8424; 1091-6490; 1091-6490
• DOI: 10.1073/pnas.1906831117

Machine learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption is limited by the level of trust afforded by given models. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of humans and machines. Here, we present expertaugmented machine learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We used a large dataset of intensive-care patient data to derive 126 decision rules that predict hospital mortality. Using an online platform, we asked 15 clinicians to assess the relative risk of the subpopulation defined by each rule compared to the total sample. We compared the clinician-assessed risk to the empirical risk and found that, while clinicians agreed with the data in most cases, there were notable exceptions where they overestimated or underestimated the true risk. Studying the rules with greatest disagreement, we identified problems with the training data, including one miscoded variable and one hidden confounder. Filtering the rules based on the extent of disagreement between clinician-assessed risk and empirical risk, we improved performance on out-of-sample data and were able to train with less data. EAML provides a platform for automated creation of problemspecific priors, which help build robust and dependable machinelearning models in critical applications.