Verification of modified receiver-operating characteristic software using simulated rating data

• Published in: Radiological physics and technology Vol. 11; no. 4; pp. 406 - 414
• Main Authors: Shiraishi, Junji, Fukuoka, Daisuke, Iha, Reimi, Inada, Haruka, Tanaka, Rie, Hara, Takeshi
• Format: Journal Article
• Language: English
• Published: Singapore Springer Singapore 01.12.2018; Springer Nature B.V
• Subjects: Computer simulation; Curve fitting; Diagnostic software; Diagnostic systems; Imaging; Medical and Radiation Physics; Medicine; Medicine & Public Health; Nuclear Medicine; Operating systems; Parameters; Program verification (computers); Radiology; Radiotherapy; Software; Statistical analysis; Statistical methods; Statistical tests; Windows (computer programs); Binormal distribution; Observer study; Computer software; Multi-reader multi-case; Simulation data; Receiver-operating characteristic analysis (ROC)
• ISSN: 1865-0333; 1865-0341; 1865-0341
• DOI: 10.1007/s12194-018-0479-9

ROCKIT, which is a receiver-operating characteristic (ROC) curve-fitting software package, was developed by Metz et al. In the early 1990s, it is a very frequently used ROC software throughout the world. In addition to ROCKIT, DBM-MRMC software was developed for multi-reader multi-case analysis of the difference in average area under ROC curves (AUCs). Because this old software cannot run on a PC with Windows 7 or a more recent operating system, we developed new software that employs the same basic algorithms with minor modifications. In this study, we verified our modified software and tested the differences between the index of diagnostic accuracies using simulated rating data. In our simulation model, all data were generated using target AUCs and a binormal parameter b . In ROC curve fitting with simulated rating data, we varied four factors: the total number of case samples, the ratio of positive-to-negative cases, a binormal parameter b , and the preset AUC. To investigate the differences between the statistical test results obtained from our software and the existing software, we generated simulated rating data sets with three levels of case difficulty and three degrees of difference in AUCs obtained from two modalities. As a result of the simulation, the AUCs estimated by the new and existing software were highly correlated ( R  > 0.98), and there were high agreements (85% or more) in the statistical test results. In conclusion, we believe that our modified software is as capable as the existing software.