Reporting Quality of Discrete Event Simulations in Healthcare—Results From a Generic Reporting Checklist

• Published in: Value in health Vol. 23; no. 4; pp. 506 - 514
• Main Authors: Zhang, Xiange, Lhachimi, Stefan K., Rogowski, Wolf H.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.04.2020; Elsevier Science Ltd
• Subjects: Access; Appraisal; Checklists; Cost analysis; Decision making; Discrete element method; discrete event simulation; Economics; Evaluation; Generalizability; Health care; healthcare decision modeling; Internal Medicine; Medical decision making; Mental task performance; Pharmacology; Quality of care; reporting quality checklist; Simulation; Sponsorship; Stakeholders; Uncertainty; Validity; healthcare decision modeling; discrete event simulation; reporting quality checklist
• ISSN: 1098-3015; 1524-4733; 1524-4733
• DOI: 10.1016/j.jval.2020.01.005

The aims of this study were to formulate a generic reporting checklist for healthcare-related discrete event simulation (DES) studies and to critically appraise the existing studies. Based on the principles of accessibility and generality, assessment items were derived from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)–Society for Medical Decision Making (SMDM) Task Force reports. The resulting checklist was applied to all 211 DES studies identified in a previous review. The proportion of fulfilled checklist items served as an indicator of reporting quality. A logistic regression was conducted to investigate whether study characteristics (eg, publication before or after the publication of the ISPOR-SMDM reports) increased the likelihood of fulfilling more than the mean number of items fulfilled by the appraised DES studies. An 18-item checklist was formulated covering model conceptualization, parameterization and uncertainty assessment, validation, generalizability, and stakeholder involvement. The reporting quality of the DES models fluctuated around the mean of 63.7% (SD 11.0%) over the period studied. A modest nonsignificant improvement in reporting quality was found after the publication of the ISPOR-SMDM reports (64.5% vs 62.9%). Items with the lowest performance were related to predictive validation (2.8% of studies), cross validation (8.5%), face validity assessment (26.5%), and stakeholder involvement (27.5%). Models applied to health economic evaluation (HEE), country under study, and industry sponsorship were significantly associated with the odds of achieving above-average reporting quality. The checklist is applicable across various model-based analyses beyond HEEs. Adherence to the ISPOR-SMDM guidelines should be improved, particularly regarding model validation. •Discrete event simulation (DES) is a versatile, individual-level modeling technique that is receiving increasing attention from healthcare researchers and decision makers and has been applied to a broad range of areas beyond health economic evaluation (HEE). To our knowledge, this study is the first in at least the last decade to be dedicated to the appraisal of reporting quality among healthcare-related DES models across this wide spectrum of application areas.•Reporting guidelines and checklists for healthcare modeling studies mainly focus on HEEs, disregarding more than half of the literature, which is focused on other applications such as operations research. To fill a gap in the literature and facilitate a broad approach for assessing modeling reports, an 18-item checklist was developed. It was based on reports from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Modeling Good Research Practices Task Force.•All 211 DES studies included in a previous review were appraised. The present study found that models applied to HEE had a higher mean quality score than other models. It also identified areas with room for improvement, such as model validation. Adherence to the ISPOR-SMDM guidance on model reporting quality should be further enhanced.