The proposed ‘concordance-statistic for benefit’ provided a useful metric when modeling heterogeneous treatment effects

• Published in: Journal of clinical epidemiology Vol. 94; pp. 59 - 68
• Main Authors: van Klaveren, David, Steyerberg, Ewout W., Serruys, Patrick W., Kent, David M.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.02.2018; Elsevier Limited
• Subjects: Acute ischemic stroke; Calibration; Cardiovascular disease; Clinical decision making; Clinical trials; Concordance; Coronary artery disease; Coronary vessels; Data processing; Decision making; Discrimination; Epidemiology; Heart diseases; Heart surgery; Individualized treatment decisions; Mathematical models; Medical screening; Patients; Performance measurement; Prediction models; Risk; Statistical analysis; Stroke; Syntax; t-Plasminogen activator; Thrombolysis; Treatment benefit; Discrimination; Prediction models; Acute ischemic stroke; Treatment benefit; Concordance; Individualized treatment decisions; Coronary artery disease
• ISSN: 0895-4356; 1878-5921; 1878-5921
• DOI: 10.1016/j.jclinepi.2017.10.021

Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit–the difference between outcome risk with vs. without therapy. We aimed to define performance metrics for a model's ability to predict treatment benefit. We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial and of three recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The ‘c-for-benefit’ represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit. Compared to a model without treatment interactions, the SYNTAX score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 vs. 0.552), despite having similar risk discrimination (c-statistic 0.725 vs. 0.719). However, for the simplified stroke–thrombolytic predictive instrument (TPI) vs. the original stroke-TPI, the c-for-benefit (0.584 vs. 0.578) was similar. The proposed methodology has the potential to measure a model's ability to predict treatment benefit not captured with conventional performance metrics.