Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structure-based virtual screening

• Published in: PloS one Vol. 14; no. 8; p. e0220113
• Main Authors: Chen, Lieyang, Cruz, Anthony, Ramsey, Steven, Dickson, Callum J., Duca, Jose S., Hornak, Viktor, Koes, David R., Kurtzman, Tom
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 20.08.2019; Public Library of Science (PLoS)
• Subjects: Artificial intelligence; Artificial neural networks; Bias; Binding; Binding sites; Biochemistry; Biology and Life Sciences; Biomedical research; Chemistry; Computer and Information Sciences; Coordination compounds; Databases, Pharmaceutical; Datasets; Decoys; Deep Learning; Docking; Drug Evaluation, Preclinical - methods; Enrichment; Health screening; Learning algorithms; Ligands; Machine learning; Molecular modelling; Neural networks; Pharmaceutical Preparations - chemistry; Pharmaceutical Preparations - metabolism; Physical Sciences; Physics; Protein binding; Proteins; Proteins - metabolism; R&D; Recognition; Research & development; Research and Analysis Methods; Screening; Success; Three dimensional models; User-Computer Interface; New York; United States > US; Massachusetts
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0220113

Recently much effort has been invested in using convolutional neural network (CNN) models trained on 3D structural images of protein-ligand complexes to distinguish binding from non-binding ligands for virtual screening. However, the dearth of reliable protein-ligand x-ray structures and binding affinity data has required the use of constructed datasets for the training and evaluation of CNN molecular recognition models. Here, we outline various sources of bias in one such widely-used dataset, the Directory of Useful Decoys: Enhanced (DUD-E). We have constructed and performed tests to investigate whether CNN models developed using DUD-E are properly learning the underlying physics of molecular recognition, as intended, or are instead learning biases inherent in the dataset itself. We find that superior enrichment efficiency in CNN models can be attributed to the analogue and decoy bias hidden in the DUD-E dataset rather than successful generalization of the pattern of protein-ligand interactions. Comparing additional deep learning models trained on PDBbind datasets, we found that their enrichment performances using DUD-E are not superior to the performance of the docking program AutoDock Vina. Together, these results suggest that biases that could be present in constructed datasets should be thoroughly evaluated before applying them to machine learning based methodology development.