Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification

• Published in: IEEE transactions on ultrasonics, ferroelectrics, and frequency control Vol. 72; no. 10; pp. 1388 - 1400
• Main Authors: Painchaud, Nathan, Stym-Popper, Jeremie, Courand, Pierre-Yves, Thome, Nicolas, Jodoin, Pierre-Marc, Duchateau, Nicolas, Bernard, Olivier
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; Algorithms; Artificial Intelligence; Cardiac function; Cardiac ultrasound; Computer Science; Computer Vision and Pattern Recognition; Data mining; Data models; Deep Learning; Echocardiography; Echocardiography - methods; Feature extraction; Female; health records; Humans; Hypertension; Hypertension - diagnostic imaging; Hypertension - physiopathology; Image Interpretation, Computer-Assisted - methods; Knowledge management; Male; Medical Imaging; Medical records; Medical services; Middle Aged; multimodal; Pathology; representation learning; Representations; Sequences; Signal and Image Processing; stratification; Training; Transformers; Training; Hypertension; Deep learning; Adaptation models; Pathology; Medical services; Feature extraction; Transformers; Data models; Data mining
• ISSN: 0885-3010; 1525-8955; 1525-8955
• DOI: 10.1109/TUFFC.2025.3600902

Deep learning enables automatic and robust extraction of cardiac function descriptors from echocardiographic sequences, such as ejection fraction (EF) or strain. These descriptors provide fine-grained information that physicians consider, in conjunction with more global variables from the clinical record, to assess patients' condition. Drawing on novel Transformer models applied to tabular data, we propose a method that considers all descriptors extracted from medical records and echocardiograms to learn the representation of a cardiovascular pathology with a difficult-to-characterize continuum, namely hypertension. Our method first projects each variable into its own representation space using modality-specific approaches. These standardized representations of multimodal data are then fed to a Transformer encoder, which learns to merge them into a comprehensive representation of the patient through the task of predicting a clinical rating. This stratification task is formulated as an ordinal classification to enforce a pathological continuum in the representation space. We observe the major trends along this continuum on a cohort of 239 hypertensive patients, providing unprecedented details in the description of hypertension's impact on various cardiac function descriptors. Our analysis shows that: 1) the XTab foundation model's architecture allows to reach high performance (96.8% AUROC) even with limited data (less than 200 training samples); 2) stratification across the population is reproducible between trainings [within 5.7% of mean absolute error (MAE)]; and 3) patterns emerge in descriptors, some of which align with established physiological knowledge about hypertension, while others could pave the way for a more comprehensive understanding of this pathology. The code is available at https://github.com/creatis-myriad/didactic