Assessing Biases in the Evaluation of Classification Assays for HIV Infection Recency

• Published in: PloS one Vol. 10; no. 10; p. e0139735
• Main Authors: Patterson-Lomba, Oscar, Wu, Julia W., Pagano, Marcello
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 05.10.2015; Public Library of Science (PLoS)
• Subjects: Acquired immune deficiency syndrome; AIDS; Algorithms; Analysis; Assaying; Biomarkers; Care and treatment; Classification; Computer simulation; Datasets; Development and progression; Diagnosis; Disease prevention; Disease transmission; Epidemics; Epidemiology; Evaluation; Forecasts and trends; Genetic diversity; Health aspects; HIV; HIV infections; HIV Infections - diagnosis; HIV Infections - epidemiology; HIV tests; HIV-1 - isolation & purification; Human immunodeficiency virus; Humans; Incidence; Infections; Life assessment; Medical treatment; Models, Theoretical; Occupational health; Performance measurement; Public health; Random variables; Sensitivity and Specificity; Virology; United States; United States > US
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0139735

Identifying recent HIV infection cases has important public health and clinical implications. It is essential for estimating incidence rates to monitor epidemic trends and evaluate the effectiveness of interventions. Detecting recent cases is also important for HIV prevention given the crucial role that recently infected individuals play in disease transmission, and because early treatment onset can improve the clinical outlook of patients while reducing transmission risk. Critical to this enterprise is the development and proper assessment of accurate classification assays that, based on cross-sectional samples of viral sequences, help determine infection recency status. In this work we assess some of the biases present in the evaluation of HIV recency classification algorithms that rely on measures of within-host viral diversity. Particularly, we examine how the time since infection (TSI) distribution of the infected subjects from which viral samples are drawn affect performance metrics (e.g., area under the ROC curve, sensitivity, specificity, accuracy and precision), potentially leading to misguided conclusions about the efficacy of classification assays. By comparing the performance of a given HIV recency assay using six different TSI distributions (four simulated TSI distributions representing different epidemic scenarios, and two empirical TSI distributions), we show that conclusions about the overall efficacy of the assay depend critically on properties of the TSI distribution. Moreover, we demonstrate that an assay with high overall classification accuracy, mainly due to properly sorting members of the well-represented groups in the validation dataset, can still perform notoriously poorly when sorting members of the less represented groups. This is an inherent issue of classification and diagnostics procedures that is often underappreciated. Thus, this work underscores the importance of acknowledging and properly addressing evaluation biases when proposing new HIV recency assays.