Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images

• Published in: Journal of the American Academy of Dermatology Vol. 78; no. 2; pp. 270 - 277.e1
• Main Authors: Marchetti, Michael A., Codella, Noel C.F., Dusza, Stephen W., Gutman, David A., Helba, Brian, Kalloo, Aadi, Mishra, Nabin, Carrera, Cristina, Celebi, M. Emre, DeFazio, Jennifer L., Jaimes, Natalia, Marghoob, Ashfaq A., Quigley, Elizabeth, Scope, Alon, Yélamos, Oriol, Halpern, Allan C.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.02.2018
• Subjects: Algorithms; computer algorithm; computer vision; Congresses as Topic; Cross-Sectional Studies; dermatologist; Dermatologists; Dermoscopy; Diagnosis, Computer-Assisted; Humans; International Skin Imaging Collaboration; International Symposium on Biomedical Imaging; Lentigo - diagnostic imaging; Machine Learning; melanoma; Melanoma - diagnosis; Melanoma - pathology; Nevus - diagnostic imaging; reader study; ROC Curve; skin cancer; Skin Neoplasms - diagnostic imaging; Skin Neoplasms - pathology; International Symposium on Biomedical Imaging; skin cancer; reader study; dermatologist; ISIC; ROC; melanoma; SVM; machine learning; ISBI; computer vision; International Skin Imaging Collaboration; computer algorithm
• ISSN: 0190-9622; 1097-6787; 1097-6787
• DOI: 10.1016/j.jaad.2017.08.016

Computer vision may aid in melanoma detection. We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images. We conducted a cross-sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into “fusion” algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant. The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P = .68) but lower than the best-performing fusion algorithm (59% vs. 76%, P = .02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P = .001). The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice. Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists.