Establishing a Validation Framework of Treatment Discontinuation in Claims Data Using Natural Language Processing and Electronic Health Records

• Published in: Clinical pharmacology and therapeutics Vol. 118; no. 1; p. 138
• Main Authors: Yang, Chun-Ting, Ngan, Kerry, Kim, Dae Hyun, Yang, Jie, Liu, Jun, Lin, Kueiyu Joshua
• Format: Journal Article
• Language: English
• Published: United States 01.07.2025
• Subjects: Aged; Aged, 80 and over; Algorithms; Anticoagulants - therapeutic use; Antipsychotic Agents - administration & dosage; Antipsychotic Agents - therapeutic use; Electronic Health Records - statistics & numerical data; Female; Humans; Insurance Claim Review - statistics & numerical data; Male; Medicare Part D - statistics & numerical data; Natural Language Processing; United States; Warfarin - therapeutic use; United States
• ISSN: 1532-6535; 0009-9236; 1085-8733; 1532-6535
• DOI: 10.1002/cpt.3650

Measuring medication discontinuation in claims data primarily relies on the gaps between prescription fills, but such definitions are rarely validated. This study aimed to establish a natural language processing (NLP)-based validation framework to evaluate the performance of claims-based discontinuation algorithms for commonly used medications against NLP-based reference standards from electronic health records (EHRs). A total of 36,656 patients receiving antipsychotic medications (APMs), benzodiazepines (BZDs), warfarin, or direct oral anticoagulants (DOACs) were identified from the Mass General Brigham EHRs in 2007-2020. These EHR data were linked with 97,900 Medicare Part D claims. An NLP-aided chart review was applied to determine medication discontinuation from EHR (reference standard). In claims data, discontinuation was defined by having a prescription gap larger than 15-90 days (claims-based algorithms). Sensitivity, specificity, and predictive values of claims-based algorithms against the reference standard were measured. The sensitivity and specificity of 90-day-gap-based algorithms were 0.46 and 0.79 for haloperidol, 0.41 and 0.85 for atypical APMs, 0.47 and 0.75 for BZDs, 0.33 and 0.80 for warfarin, and 0.38 and 0.87 for DOACs, respectively. The corresponding estimates for 15-day-gap-based algorithms were 0.68 and 0.55 for haloperidol, 0.59 and 0.62 for atypical APMs, 0.71 and 0.45 for BZDs, 0.61 and 0.49 for warfarin, and 0.58 and 0.64 for DOACs, respectively. Positive predictive values were primarily affected by medication discontinuation rates and less by gap lengths. The overall accuracy of claims-based discontinuation algorithms differs by medications. This study demonstrates the scalability and utility of the NLP-based validation framework for multiple medications.