Multimodal convolutional neural network–based algorithm for real-time detection and differentiation of malignant and inflammatory biliary strictures in cholangioscopy: a proof-of-concept study (with video)

• Published in: Gastrointestinal endoscopy Vol. 101; no. 4; pp. 830 - 842.e2
• Main Authors: Ziegler, Joceline, Dobsch, Philipp, Rozema, Marten, Zuber-Jerger, Ina, Weigand, Kilian, Reuther, Stefan, Müller, Martina, Kandulski, Arne
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.04.2025
• Subjects: Aged; Algorithms; Bile Duct Neoplasms - complications; Bile Duct Neoplasms - diagnosis; Bile Duct Neoplasms - diagnostic imaging; Biliary Tract Neoplasms - complications; Biliary Tract Neoplasms - diagnosis; Biliary Tract Neoplasms - diagnostic imaging; Cholangiocarcinoma - diagnosis; Cholangitis - diagnosis; Cholangitis - diagnostic imaging; Cholestasis - diagnostic imaging; Cholestasis - etiology; Constriction, Pathologic - diagnostic imaging; Convolutional Neural Networks; Deep Learning; Diagnosis, Computer-Assisted - methods; Diagnosis, Differential; Endoscopy, Digestive System - methods; Female; Humans; Male; Middle Aged; Neural Networks, Computer; Proof of Concept Study; ROC Curve; Sensitivity and Specificity; BTC; CADe; UMAP; PSC; CNN; NPV; PPV; DL; CADx; dSOC; AUC
• ISSN: 0016-5107; 1097-6779; 1097-6779
• DOI: 10.1016/j.gie.2024.09.001

Deep learning algorithms gained attention for detection (computer-aided detection [CADe]) of biliary tract cancer in digital single-operator cholangioscopy (dSOC). We developed a multimodal convolutional neural network (CNN) for detection (CADe), characterization and discriminating (computer-aided diagnosis [CADx]) between malignant, inflammatory, and normal biliary tissue in raw dSOC videos. In addition, clinical metadata were included in the CNN algorithm to overcome limitations of image-only models. Based on dSOC videos and images of 111 patients (total of 15,158 still frames), a real-time CNN-based algorithm for CADe and CADx was developed and validated. We established an image-only model and metadata injection approach. In addition, frame-wise and case-based predictions on complete dSOC video sequences were validated. Model embeddings were visualized, and class activation maps highlighted relevant image regions. The concatenation-based CADx approach achieved a per-frame area under the receiver-operating characteristic curve of .871, sensitivity of .809 (95% CI, .784-.832), specificity of .773 (95% CI, .761-.785), positive predictive value of .450 (95% CI, .423-.467), and negative predictive value of .946 (95% CI, .940-.954) with respect to malignancy on 5715 test frames from complete videos of 20 patients. For case-based diagnosis using average prediction scores, 6 of 8 malignant cases and all 12 benign cases were identified correctly. Our algorithm distinguishes malignant and inflammatory bile duct lesions in dSOC videos, indicating the potential of CNN-based diagnostic support systems for both CADe and CADx. The integration of non-image data can improve CNN-based support systems, targeting current challenges in the assessment of biliary strictures. [Display omitted]