Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single ca...

Full description

Saved in:
Bibliographic Details
Published inProgress in biomedical engineering (Bristol) Vol. 5; no. 2; pp. 22001 - 22021
Main Authors Cui, Can, Yang, Haichun, Wang, Yaohong, Zhao, Shilin, Asad, Zuhayr, Coburn, Lori A, Wilson, Keith T, Landman, Bennett A, Huo, Yuankai
Format Journal Article
LanguageEnglish
Published England IOP Publishing 01.04.2023
Subjects
Online AccessGet full text
ISSN2516-1091
2516-1091
DOI10.1088/2516-1091/acc2fe

Cover

Abstract The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.
AbstractList The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.
The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.
Author Asad, Zuhayr
Coburn, Lori A
Wilson, Keith T
Cui, Can
Zhao, Shilin
Landman, Bennett A
Yang, Haichun
Huo, Yuankai
Wang, Yaohong
Author_xml – sequence: 1
  givenname: Can
  orcidid: 0000-0002-2159-5387
  surname: Cui
  fullname: Cui, Can
  organization: Vanderbilt University Department of Computer Science, Nashville, TN 37235, United States of America
– sequence: 2
  givenname: Haichun
  surname: Yang
  fullname: Yang, Haichun
  organization: Vanderbilt University Medical Center Department of Pathology, Microbiology and Immunology, Nashville, TN 37215, United States of America
– sequence: 3
  givenname: Yaohong
  surname: Wang
  fullname: Wang, Yaohong
  organization: Vanderbilt University Medical Center Department of Pathology, Microbiology and Immunology, Nashville, TN 37215, United States of America
– sequence: 4
  givenname: Shilin
  surname: Zhao
  fullname: Zhao, Shilin
  organization: Vanderbilt University Medical Center Department of Biostatistics, Nashville, TN 37215, United States of America
– sequence: 5
  givenname: Zuhayr
  surname: Asad
  fullname: Asad, Zuhayr
  organization: Vanderbilt University Department of Computer Science, Nashville, TN 37235, United States of America
– sequence: 6
  givenname: Lori A
  surname: Coburn
  fullname: Coburn, Lori A
  organization: Veterans Affairs Tennessee Valley Healthcare System , Nashville, TN 37212, United States of America
– sequence: 7
  givenname: Keith T
  surname: Wilson
  fullname: Wilson, Keith T
  organization: Veterans Affairs Tennessee Valley Healthcare System , Nashville, TN 37212, United States of America
– sequence: 8
  givenname: Bennett A
  surname: Landman
  fullname: Landman, Bennett A
  organization: Vanderbilt University Department of Electrical and Computer Engineering, Nashville, TN 37235, United States of America
– sequence: 9
  givenname: Yuankai
  surname: Huo
  fullname: Huo, Yuankai
  organization: Vanderbilt University Department of Electrical and Computer Engineering, Nashville, TN 37235, United States of America
BackLink https://www.ncbi.nlm.nih.gov/pubmed/37360402$$D View this record in MEDLINE/PubMed
BookMark eNp9kE1PxCAQhonR-LF692S46cHqAFu29Wb8TjbxomfC0mGDaUuFVuO_l7VqjDGGAwx5Xph5dsh661skZJ_BCYOiOOU5kxmDkp1qY7jFNbL9fbX-47xF9mJ8AoB0w0oOm2RLzISEKfBtoi4RO9oMde8aX-ma2iE631JvqWv0EqluK5o-zsaq0r2mrqWVi6hjqp1etj66-MF1wY_VGdU04IvD112yYXUdce9zn5DH66uHi9tsfn9zd3E-z8yUQZ9Ni9RPDjMJDE0140JODZrFIs2Z6zItiYWRHLgR1iIwW1a5LFgpNXIBiGJCjsZ3Uw_PA8ZeNS4arGvdoh-i4oUAzoTMZwk9-ESHRYOV6kKaLbypLykJgBEwwccY0H4jDNRKvVq5VSu3alSfIvJXxLhe98lkH7Sr_wsej0HnO_Xkh9AmS__hh3_gXVguVK64As4BmOoqK94BaCujVQ
CODEN PBERB8
CitedBy_id crossref_primary_10_1007_s10278_024_01036_7
crossref_primary_10_1016_j_compbiomed_2024_109399
crossref_primary_10_3389_frai_2025_1481338
crossref_primary_10_1016_j_jbi_2025_104796
crossref_primary_10_1007_s11042_023_14560_5
crossref_primary_10_1016_j_compbiomed_2024_109438
crossref_primary_10_1016_j_cmpb_2024_108568
crossref_primary_10_1002_qub2_93
crossref_primary_10_3390_diagnostics14151634
crossref_primary_10_1136_bmjopen_2024_094110
crossref_primary_10_3389_fmed_2024_1450103
crossref_primary_10_1109_ACCESS_2023_3331315
crossref_primary_10_3390_bioengineering11090872
crossref_primary_10_1007_s10462_023_10582_5
crossref_primary_10_3390_jcm14020550
crossref_primary_10_1115_1_4063954
crossref_primary_10_1007_s11227_025_07083_3
crossref_primary_10_1016_j_inffus_2025_103033
crossref_primary_10_1109_ACCESS_2024_3385340
crossref_primary_10_1016_j_engappai_2025_110554
crossref_primary_10_1190_geo2024_0113_1
crossref_primary_10_1016_j_microc_2025_112870
crossref_primary_10_37394_23205_2023_22_37
crossref_primary_10_1016_j_compbiomed_2023_107373
crossref_primary_10_1109_ACCESS_2024_3412961
crossref_primary_10_1002_sim_10350
crossref_primary_10_1016_j_inffus_2023_102134
crossref_primary_10_1111_jdv_20166
crossref_primary_10_1007_s13198_023_02180_z
crossref_primary_10_1109_ACCESS_2024_3427731
crossref_primary_10_3390_diagnostics13020217
crossref_primary_10_3348_kjr_2024_1086
crossref_primary_10_1016_j_inffus_2024_102690
crossref_primary_10_1109_TVCG_2023_3326929
crossref_primary_10_1007_s10462_023_10588_z
crossref_primary_10_1097_RLI_0000000000001106
crossref_primary_10_1109_TAI_2024_3397246
crossref_primary_10_35516_jjps_v17i3_2410
crossref_primary_10_2196_59452
crossref_primary_10_1016_j_compbiomed_2025_109721
crossref_primary_10_1177_20552076241253757
crossref_primary_10_1109_JBHI_2023_3279433
crossref_primary_10_1016_j_compgeo_2024_106697
crossref_primary_10_1088_1361_6501_ad14e2
crossref_primary_10_7717_peerj_cs_2338
crossref_primary_10_1097_WCO_0000000000001280
crossref_primary_10_1002_advs_202407060
crossref_primary_10_1093_bib_bbae449
crossref_primary_10_1088_1361_6501_ad6927
Cites_doi 10.1038/s41586-021-03512-4
10.1117/12.2581127
10.1016/j.eswa.2022.118051
10.1109/JBHI.2021.3062002
10.1016/j.media.2018.06.001
10.1038/s41746-020-00341-z
10.1093/bioinformatics/btw033
10.1212/WNL.0b013e3181cb3e25
10.1016/j.inffus.2021.07.016
10.1109/TMI.2022.3142321
10.1038/s41597-021-00815-z
10.1007/s00371-022-02492-4
10.1109/CVPR52688.2022.01764
10.1109/BIBM49941.2020.9313289
10.1038/s41597-019-0322-0
10.1038/s41551-020-00682-w
10.1093/jamia/ocv080
10.1097/IIO.0000000000000246
10.1109/CVPR.2018.00943
10.1038/s41597-020-00622-y
10.17925/EE.2021.17.1.5
10.1109/iccvw54120.2021.00368
10.1007/978-3-030-87240-3
10.1109/TCBB.2018.2806438
10.1002/hbm.24428
10.1016/j.blre.2020.100693
10.1109/RBME.2009.2034865
10.1111/cup.12485
10.1038/s41568-021-00408-3
10.1080/21681163.2017.1356750
10.1109/cvpr46437.2021.01549
10.1109/ISBI52829.2022.9761567
10.1186/s12911-020-01340-6
10.1016/j.compbiomed.2022.106241
10.1109/CVPR.2017.369
10.1038/s41598-019-47765-6
10.1093/bib/bbab569
10.1007/978-3-030-58577-8_7
10.1016/j.patcog.2013.11.032
10.1109/TPAMI.2018.2798607
10.1109/CVPR.2019.01075
10.1016/j.bspc.2021.103015
10.18653/v1/D19-1514
10.1109/EMBC44109.2020.9176360
10.1111/1751-2980.12992
10.1016/j.acra.2015.05.007
10.1016/j.inffus.2022.01.001
10.1016/j.inffus.2020.09.006
10.1186/s12874-018-0482-1
10.1007/s00429-013-0687-3
10.18653/v1/w19-5034
10.1038/mp.2013.78
10.1016/j.neucom.2020.05.087
10.1016/j.media.2019.01.012
10.3389/fonc.2021.788740
10.1007/978-3-030-87237-3
10.1109/cvprw.2009.5206848
10.1016/S1474-4422(13)70024-3
10.21147/j.issn.1000-9604.2021.05.03
10.1007/978-3-319-66185-8_46
10.1109/TKDE.2019.2932666
10.1093/biomet/28.3-4.321
10.7326/0003-4819-140-3-200402030-00010
10.1007/978-3-030-60946-7_11
10.1186/s13014-019-1385-y
10.1007/978-3-031-18814-5_1
10.18653/v1/d17-1115
10.1145/3394171.3413977
10.1093/bioinformatics/btab185
10.1038/s41598-020-78888-w
10.18653/v1/d19-1371
10.1109/ISBI48211.2021.9433823
10.1007/s12021-013-9204-3.A
10.1109/TCSVT.2021.3063952
10.1016/j.media.2020.101789
10.1109/CVPR46437.2021.01409
10.1007/s11684-020-0782-9
10.1016/j.ejca.2021.10.007
10.1093/bioinformatics/btz318
10.1007/s00330-021-07854-5
10.1007/978-3-030-87240-3_66
10.1111/exd.13777
10.1186/gb-2006-7-10-r100
10.1109/tmi.2020.3021387
10.1007/978-3-030-59713-9_51
10.1109/ICCV48922.2021.00346
10.1162/neco_a_01273
10.1109/JBHI.2018.2824327
10.1093/bioinformatics/btz342
10.1038/s41746-021-00438-z
10.1016/j.jbi.2019.103337
10.1073/pnas.1717139115
10.1007/978-3-030-59713-9_24
ContentType Journal Article
Copyright 2023 The Author(s). Published by IOP Publishing Ltd
Copyright_xml – notice: 2023 The Author(s). Published by IOP Publishing Ltd
DBID O3W
TSCCA
AAYXX
CITATION
NPM
7X8
DOI 10.1088/2516-1091/acc2fe
DatabaseName Institute of Physics Open Access Journal Titles
IOPscience (Open Access)
CrossRef
PubMed
MEDLINE - Academic
DatabaseTitle CrossRef
PubMed
MEDLINE - Academic
DatabaseTitleList CrossRef
MEDLINE - Academic
PubMed
Database_xml – sequence: 1
  dbid: NPM
  name: PubMed
  url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
  sourceTypes: Index Database
– sequence: 2
  dbid: O3W
  name: Institute of Physics Open Access Journal Titles
  url: http://iopscience.iop.org/
  sourceTypes:
    Enrichment Source
    Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2516-1091
ExternalDocumentID 37360402
10_1088_2516_1091_acc2fe
prgbacc2fe
Genre Journal Article
GrantInformation_xml – fundername: CAREER
  grantid: 1452485; I01BX004366; I01CX002171
– fundername: NSF
  grantid: G-1903-03793
– fundername: Leona M. and Harry B. Helmsley Charitable Trust
  funderid: http://dx.doi.org/10.13039/100007028
– fundername: BLRD VA
  grantid: I01 BX004366
– fundername: CSRD VA
  grantid: I01 CX002171
GroupedDBID AAGCD
ABVAM
ACHIP
AKOSD
AKPSB
ALMA_UNASSIGNED_HOLDINGS
AOAED
CJUJL
IJHAN
IOP
KOT
M~E
N5L
O3W
ROL
RPA
TSCCA
AAYXX
ABJNI
ADEQX
AEINN
CITATION
NPM
7X8
ID FETCH-LOGICAL-c410t-48604507601ecd72364cecbb0885a9a9a6e8c6202c3ffe01f9d568196ae230ee3
IEDL.DBID IOP
ISSN 2516-1091
IngestDate Thu Sep 04 23:41:29 EDT 2025
Wed Feb 19 02:23:21 EST 2025
Wed Oct 01 03:24:41 EDT 2025
Thu Apr 24 22:59:42 EDT 2025
Wed May 17 01:26:54 EDT 2023
Wed Aug 21 03:33:40 EDT 2024
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 2
Keywords multimodal learning
medical data
fusion method
diagnosis and prognosis
Language English
License Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c410t-48604507601ecd72364cecbb0885a9a9a6e8c6202c3ffe01f9d568196ae230ee3
Notes PRGB-100107.R1
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ORCID 0000-0002-2159-5387
OpenAccessLink https://proxy.k.utb.cz/login?url=https://iopscience.iop.org/article/10.1088/2516-1091/acc2fe
PMID 37360402
PQID 2830213657
PQPubID 23479
PageCount 21
ParticipantIDs pubmed_primary_37360402
crossref_primary_10_1088_2516_1091_acc2fe
crossref_citationtrail_10_1088_2516_1091_acc2fe
proquest_miscellaneous_2830213657
iop_journals_10_1088_2516_1091_acc2fe
ProviderPackageCode CITATION
AAYXX
PublicationCentury 2000
PublicationDate 2023-04-01
PublicationDateYYYYMMDD 2023-04-01
PublicationDate_xml – month: 04
  year: 2023
  text: 2023-04-01
  day: 01
PublicationDecade 2020
PublicationPlace England
PublicationPlace_xml – name: England
PublicationTitle Progress in biomedical engineering (Bristol)
PublicationTitleAbbrev PRGB
PublicationTitleAlternate Prog. Biomed. Eng
PublicationYear 2023
Publisher IOP Publishing
Publisher_xml – name: IOP Publishing
References Huang (prgbacc2febib87) 2020; 10
Rotemberg (prgbacc2febib43) 2021; 8
Lu (prgbacc2febib69) 2022
Schneider (prgbacc2febib13) 2021; 160
Neumann (prgbacc2febib54) 2019
El-Sappagh (prgbacc2febib24) 2020; 412
Zhou (prgbacc2febib84) 2019; 40
Wang (prgbacc2febib79) 2018
Piccialli (prgbacc2febib6) 2021; 66
Di Martino (prgbacc2febib83) 2014; 19
Chen (prgbacc2febib94) 2019
Pacheco (prgbacc2febib88) 2021; 25
Sharifi-Noghabi (prgbacc2febib107) 2019; 35
Mobadersany (prgbacc2febib21) 2018; 115
Ozturk (prgbacc2febib2) 2021; 45
McDonald (prgbacc2febib5) 2015; 22
Aggarwal (prgbacc2febib10) 2021; 4
Katzman (prgbacc2febib17) 2018; 18
Sharma (prgbacc2febib75) 2019; 9
Sun (prgbacc2febib82) 2018; 16
Chen (prgbacc2febib67) 2020; 0062
Yao (prgbacc2febib28) 2017; vol 10434
Jia (prgbacc2febib96) 2009
Baltrusaitis (prgbacc2febib97) 2019; 41
van Sonsbeek (prgbacc2febib90); vol 12445
Chen (prgbacc2febib93)
Xu (prgbacc2febib71) 2022
Ye (prgbacc2febib77) 2019
Yang (prgbacc2febib70) 2021; 33
Kawahara (prgbacc2febib45) 2019; 23
Deng (prgbacc2febib8) 2020; 14
Chen (prgbacc2febib26) 2021
Ghosal (prgbacc2febib65) 2021; 11596
Cui (prgbacc2febib35) 2022; vol Part V
Wang (prgbacc2febib98) 2021; 31
Li (prgbacc2febib108) 2022; 151
Devlin (prgbacc2febib55) 2019; vol 1
Stahlschmidt (prgbacc2febib15) 2022; 23
Jacenkow (prgbacc2febib72) 2022
Li (prgbacc2febib48) 2020; vol 2020
Huang (prgbacc2febib57) 2019
Li (prgbacc2febib31) 2021
Schulz (prgbacc2febib20) 2021; 11
Demner-Fushman (prgbacc2febib81) 2016; 23
Pölsterl (prgbacc2febib52) 2021
Lu (prgbacc2febib14) 2021; 33
Mwangi (prgbacc2febib63) 2015; 12
Yan (prgbacc2febib27) 2021; 21
Holste (prgbacc2febib16) 2021
Beltagy (prgbacc2febib56) 2020
Vaswani (prgbacc2febib100) 2017; vol 30
Tan (prgbacc2febib92)
Yao (prgbacc2febib32) 2020; 65
Cheng (prgbacc2febib106) 2022; 41
Althloothi (prgbacc2febib85) 2014; 47
Yap (prgbacc2febib42) 2018; 27
Yu (prgbacc2febib102) 2020
Li (prgbacc2febib33) 2021
Boehm (prgbacc2febib11) 2021; 22
Silva (prgbacc2febib44) 2020
Lu (prgbacc2febib25) 2021; 5
Gao (prgbacc2febib7) 2020; 32
Date (prgbacc2febib50) 2019; 59
Lu (prgbacc2febib18) 2021; 594
Gurcan (prgbacc2febib19) 2009; 2
Wang (prgbacc2febib112) 2017
Nan (prgbacc2febib104) 2022; 82
Comfere (prgbacc2febib1) 2015; 42
Cao (prgbacc2febib41) 2021; 70
Azizi (prgbacc2febib105)
Cui (prgbacc2febib68) 2021; vol 4
Bakas (prgbacc2febib66) 2018
Huang (prgbacc2febib95) 2020
Lee (prgbacc2febib61) 2021
Zadeh (prgbacc2febib73) 2017
Yoo (prgbacc2febib38) 2019; 7
Doherty (prgbacc2febib3) 2013; 12
Bach (prgbacc2febib62) 2017; 18
(prgbacc2febib22) 2023
Johnson (prgbacc2febib89) 2019; 6
Huang (prgbacc2febib12) 2020; 3
Li (prgbacc2febib58) 2020
Whiting (prgbacc2febib4) 2004; 140
Suk (prgbacc2febib86) 2015; 220
Chauhan (prgbacc2febib53) 2020; vol 12262
Singh (prgbacc2febib101) 2022; 207
Wang (prgbacc2febib80) 2017
(prgbacc2febib23) 2023
Schlemper (prgbacc2febib74) 2019; 53
Perez (prgbacc2febib78) 2018
Carpenter (prgbacc2febib30) 2006; 7
Harold (prgbacc2febib99) 1936; 28
Ma (prgbacc2febib103) 2022
Wang (prgbacc2febib60) 2021; vol 2021
Braman (prgbacc2febib34) 2021
Guan (prgbacc2febib36) 2021; vol
Petersen (prgbacc2febib37) 2010; 74
Du (prgbacc2febib46) 2016; 32
Wang (prgbacc2febib29) 2021; 37
Harold Li (prgbacc2febib91)
Borgli (prgbacc2febib110) 2020; 7
Cheerla (prgbacc2febib47) 2019; 35
Yang (prgbacc2febib109) 2021; 77
Zhang (prgbacc2febib49) 2021; 22
Cai (prgbacc2febib51) 2022
Cassidy (prgbacc2febib111) 2021; 1
Chen (prgbacc2febib64) 2021
Zhou (prgbacc2febib39) 2021; 32
Parisot (prgbacc2febib40) 2018; 48
Xu (prgbacc2febib76) 2019; 14
Ayala Solares (prgbacc2febib9) 2020; 101
Duanmu (prgbacc2febib59) 2020
References_xml – volume: 594
  start-page: 106
  year: 2021
  ident: prgbacc2febib18
  article-title: AI-based pathology predicts origins for cancers of unknown primary
  publication-title: Nature
  doi: 10.1038/s41586-021-03512-4
– volume: 11596
  year: 2021
  ident: prgbacc2febib65
  article-title: G-MIND: an end-to-end multimodal imaging-genetics framework for biomarker identification and disease classification
  publication-title: Proc. SPIE
  doi: 10.1117/12.2581127
– volume: 207
  year: 2022
  ident: prgbacc2febib101
  article-title: A neural architecture search for automated multimodal learning
  publication-title: Expert Syst. Appl.
  doi: 10.1016/j.eswa.2022.118051
– volume: 25
  start-page: 3554
  year: 2021
  ident: prgbacc2febib88
  article-title: An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification
  publication-title: IEEE J. Biomed. Health Inform.
  doi: 10.1109/JBHI.2021.3062002
– volume: 48
  start-page: 117
  year: 2018
  ident: prgbacc2febib40
  article-title: Disease prediction using graph convolutional networks: application to autism spectrum disorder and Alzheimer’s disease
  publication-title: Med. Image Anal.
  doi: 10.1016/j.media.2018.06.001
– volume: 3
  start-page: 2398
  year: 2020
  ident: prgbacc2febib12
  article-title: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines
  publication-title: npj Digit. Med.
  doi: 10.1038/s41746-020-00341-z
– volume: 32
  start-page: 1544
  year: 2016
  ident: prgbacc2febib46
  article-title: Structured sparse canonical correlation analysis for brain imaging genetics: an improved GraphNet method
  publication-title: Bioinformatics
  doi: 10.1093/bioinformatics/btw033
– volume: 74
  start-page: 201
  year: 2010
  ident: prgbacc2febib37
  article-title: Alzheimer’s disease neuroimaging initiative (ADNI): clinical characterization
  publication-title: Neurology
  doi: 10.1212/WNL.0b013e3181cb3e25
– volume: 77
  start-page: 29
  year: 2021
  ident: prgbacc2febib109
  article-title: Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: a mini-review, two showcases and beyond
  publication-title: Inf. Fusion
  doi: 10.1016/j.inffus.2021.07.016
– start-page: 4015
  year: 2021
  ident: prgbacc2febib26
  article-title: Multimodal Co-attention transformer for survival prediction in gigapixel whole slide images
– start-page: 568
  year: 2020
  ident: prgbacc2febib44
  article-title: Pan-cancer prognosis prediction using multimodal deep learning
– volume: 41
  start-page: 1520
  year: 2022
  ident: prgbacc2febib106
  article-title: A fully automated multimodal MRI-based multi-task learning for glioma segmentation and IDH genotyping
  publication-title: IEEE Trans. Med. Imaging
  doi: 10.1109/TMI.2022.3142321
– volume: 8
  start-page: 34
  year: 2021
  ident: prgbacc2febib43
  article-title: A patient-centric dataset of images and metadata for identifying melanomas using clinical context
  publication-title: Sci. Data
  doi: 10.1038/s41597-021-00815-z
– start-page: 1
  year: 2022
  ident: prgbacc2febib51
  article-title: A multimodal transformer to fuse images and metadata for skin disease classification
  publication-title: Vis. Comput.
  doi: 10.1007/s00371-022-02492-4
– year: 2022
  ident: prgbacc2febib103
  article-title: Are multimodal transformers robust to missing modality?
  doi: 10.1109/CVPR52688.2022.01764
– year: 2020
  ident: prgbacc2febib58
  article-title: A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports
  doi: 10.1109/BIBM49941.2020.9313289
– volume: 6
  start-page: 317
  year: 2019
  ident: prgbacc2febib89
  article-title: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
  publication-title: Sci. Data
  doi: 10.1038/s41597-019-0322-0
– volume: 5
  start-page: 555
  year: 2021
  ident: prgbacc2febib25
  article-title: Data-efficient and weakly supervised computational pathology on whole-slide images
  publication-title: Nat. Biomed. Eng.
  doi: 10.1038/s41551-020-00682-w
– volume: 23
  start-page: 304
  year: 2016
  ident: prgbacc2febib81
  article-title: Preparing a collection of radiology examinations for distribution and retrieval
  publication-title: J. Am. Med. Inform. Assoc.
  doi: 10.1093/jamia/ocv080
– volume: 59
  start-page: 39
  year: 2019
  ident: prgbacc2febib50
  article-title: Applications of deep learning and artificial intelligence in Retina
  publication-title: Int. Ophthalmol. Clin.
  doi: 10.1097/IIO.0000000000000246
– start-page: 9049
  year: 2018
  ident: prgbacc2febib79
  article-title: TieNet: text-image embedding network for common thorax disease classification and reporting in chest x-rays
  doi: 10.1109/CVPR.2018.00943
– year: 2020
  ident: prgbacc2febib95
  article-title: Pixel-bert: Aligning image pixels with text by deep multi-modal transformers
– volume: 7
  start-page: 283
  year: 2020
  ident: prgbacc2febib110
  article-title: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy
  publication-title: Sci. Data
  doi: 10.1038/s41597-020-00622-y
– volume: 1
  start-page: 5
  year: 2021
  ident: prgbacc2febib111
  article-title: The DFUC 2020 dataset: analysis towards diabetic foot ulcer detection
  publication-title: Eur. Endocrinol.
  doi: 10.17925/EE.2021.17.1.5
– start-page: 3287
  year: 2021
  ident: prgbacc2febib16
  article-title: End-to-end learning of fused image and non-image features for improved breast cancer classification from MRI
  doi: 10.1109/iccvw54120.2021.00368
– volume: vol
  year: 2021
  ident: prgbacc2febib36
  article-title: Predicting esophageal fistula risks using a multimodal self-attention network
  publication-title: Medical Image Computing and Computer Assisted Intervention—MICCAI
  doi: 10.1007/978-3-030-87240-3
– volume: 16
  start-page: 841
  year: 2018
  ident: prgbacc2febib82
  article-title: A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data
  publication-title: IEEE/ACM Trans. Comput. Biol. Bioinform.
  doi: 10.1109/TCBB.2018.2806438
– volume: 40
  start-page: 1001
  year: 2019
  ident: prgbacc2febib84
  article-title: Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis
  publication-title: Hum. Brain Mapp.
  doi: 10.1002/hbm.24428
– volume: 45
  year: 2021
  ident: prgbacc2febib2
  article-title: Non-hematologic diagnosis of systemic mastocytosis: collaboration of radiology and pathology
  publication-title: Blood Rev.
  doi: 10.1016/j.blre.2020.100693
– volume: 2
  start-page: 147
  year: 2009
  ident: prgbacc2febib19
  article-title: Histopathological image analysis: a review
  publication-title: IEEE Rev. Biomed. Eng.
  doi: 10.1109/RBME.2009.2034865
– volume: 18
  start-page: 1
  year: 2017
  ident: prgbacc2febib62
  article-title: Breaking the curse of dimensionality with convex neural networks
  publication-title: J. Mach. Learn. Res.
– year: 2023
  ident: prgbacc2febib22
– volume: 42
  start-page: 333
  year: 2015
  ident: prgbacc2febib1
  article-title: Dermatopathologists’ concerns and challenges with clinical information in the skin biopsy requisition form: a mixed-methods study
  publication-title: J. Cutan. Pathol.
  doi: 10.1111/cup.12485
– start-page: 667
  year: 2021
  ident: prgbacc2febib34
  article-title: Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data
– volume: 22
  start-page: 114
  year: 2021
  ident: prgbacc2febib11
  article-title: Harnessing multimodal data integration to advance precision oncology
  publication-title: Nat. Rev. Cancer
  doi: 10.1038/s41568-021-00408-3
– volume: vol Part V
  start-page: 626
  year: 2022
  ident: prgbacc2febib35
  article-title: Survival prediction of brain cancer with incomplete radiology, pathology, genomics, and demographic data
– volume: 7
  start-page: 250
  year: 2019
  ident: prgbacc2febib38
  article-title: Deep learning of brain lesion patterns and user-defined clinical and MRI features for predicting conversion to multiple sclerosis from clinically isolated syndrome
  publication-title: Comput. Methods Biomech. Biomed. Eng. Imaging Vis.
  doi: 10.1080/21681163.2017.1356750
– start-page: 15745
  year: 2021
  ident: prgbacc2febib64
  article-title: Exploring simple siamese representation learning
  doi: 10.1109/cvpr46437.2021.01549
– year: 2022
  ident: prgbacc2febib72
  article-title: Indication as prior knowledge for multimodal disease classification in chest radiographs with transformers
  doi: 10.1109/ISBI52829.2022.9761567
– volume: 21
  start-page: 1
  year: 2021
  ident: prgbacc2febib27
  article-title: Richer fusion network for breast cancer classification based on multimodal data
  publication-title: BMC Medical Inform. Decis. Mak.
  doi: 10.1186/s12911-020-01340-6
– volume: 151
  year: 2022
  ident: prgbacc2febib108
  article-title: Human treelike tubular structure segmentation: a comprehensive review and future perspectives
  publication-title: Comput. Biol. Med.
  doi: 10.1016/j.compbiomed.2022.106241
– year: 2017
  ident: prgbacc2febib112
  article-title: ChestX-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases
  doi: 10.1109/CVPR.2017.369
– volume: 9
  start-page: 1
  year: 2019
  ident: prgbacc2febib75
  article-title: DeepInsight: a methodology to transform a non-image data to an image for convolution neural network architecture
  publication-title: Sci. Rep.
  doi: 10.1038/s41598-019-47765-6
– volume: 23
  start-page: 569
  year: 2022
  ident: prgbacc2febib15
  article-title: Multimodal deep learning for biomedical data fusion: a review
  publication-title: Brief. Bioinform.
  doi: 10.1093/bib/bbab569
– year: 2022
  ident: prgbacc2febib71
  article-title: Multimodal learning with transformers: a survey
– ident: prgbacc2febib93
  article-title: UNITER: uNiversal image-TExt representation learning
  doi: 10.1007/978-3-030-58577-8_7
– volume: 47
  start-page: 1800
  year: 2014
  ident: prgbacc2febib85
  article-title: Human activity recognition using multi-features and multiple kernel learning
  publication-title: Pattern Recognit.
  doi: 10.1016/j.patcog.2013.11.032
– volume: 41
  start-page: 423
  year: 2019
  ident: prgbacc2febib97
  article-title: Multimodal machine learning: a survey and taxonomy
  publication-title: IEEE Trans. Pattern Anal. Mach. Intell.
  doi: 10.1109/TPAMI.2018.2798607
– start-page: 10494
  year: 2019
  ident: prgbacc2febib77
  article-title: Cross-modal self-attention network for referring image segmentation
  doi: 10.1109/CVPR.2019.01075
– volume: 70
  year: 2021
  ident: prgbacc2febib41
  article-title: Using DeepGCN to identify the autism spectrum disorder from multi-site resting-state data
  publication-title: Biomed. Signal Process. Control.
  doi: 10.1016/j.bspc.2021.103015
– ident: prgbacc2febib92
  article-title: LXMERT: learning cross-modality encoder representations from transformers
  doi: 10.18653/v1/D19-1514
– year: 2018
  ident: prgbacc2febib66
  article-title: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge
– volume: vol 2020
  start-page: 1384
  year: 2020
  ident: prgbacc2febib48
  article-title: A novel pathological images and genomic data fusion framework for breast cancer survival prediction
  doi: 10.1109/EMBC44109.2020.9176360
– volume: 22
  start-page: 318
  year: 2021
  ident: prgbacc2febib49
  article-title: Accuracy of artificial intelligence-assisted detection of esophageal cancer and neoplasms on endoscopic images: a systematic review and meta-analysis
  publication-title: J. Dig. Dis.
  doi: 10.1111/1751-2980.12992
– year: 2019
  ident: prgbacc2febib57
  article-title: ClinicalBERT: modeling clinical notes and predicting hospital readmission
– ident: prgbacc2febib91
  article-title: Work in progress VISUALBERT: a simple and performant baseline for vision and language
– volume: 22
  start-page: 1191
  year: 2015
  ident: prgbacc2febib5
  article-title: The effects of changes in utilization and technological advancements of cross-sectional imaging onradiologist workload
  publication-title: Acad Radiol.
  doi: 10.1016/j.acra.2015.05.007
– volume: 82
  start-page: 99
  year: 2022
  ident: prgbacc2febib104
  article-title: Data harmonisation for information fusion in digital healthcare: a state-of-the-art systematic review, meta-analysis and future research directions
  publication-title: Inf. Fusion
  doi: 10.1016/j.inffus.2022.01.001
– year: 2021
  ident: prgbacc2febib61
  article-title: A variational information bottleneck approach to multi-omics data integration
– volume: 66
  start-page: 111
  year: 2021
  ident: prgbacc2febib6
  article-title: A survey on deep learning in medicine: why, how and when?
  publication-title: Inf. Fusion
  doi: 10.1016/j.inffus.2020.09.006
– volume: 18
  start-page: 1
  year: 2018
  ident: prgbacc2febib17
  article-title: DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network
  publication-title: BMC Med. Res. Methodol.
  doi: 10.1186/s12874-018-0482-1
– volume: vol 30
  year: 2017
  ident: prgbacc2febib100
  article-title: Attention is all you need
– volume: 220
  start-page: 841
  year: 2015
  ident: prgbacc2febib86
  article-title: Latent feature representation with stacked auto-encoder for AD/MCI diagnosis
  publication-title: Brain Struct. Funct.
  doi: 10.1007/s00429-013-0687-3
– start-page: 319
  year: 2019
  ident: prgbacc2febib54
  article-title: ScispaCy: fast and robust models for biomedical natural language processing
  doi: 10.18653/v1/w19-5034
– volume: 19
  start-page: 659
  year: 2014
  ident: prgbacc2febib83
  article-title: The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism
  publication-title: Mol. Psychiatry
  doi: 10.1038/mp.2013.78
– volume: 412
  start-page: 197
  year: 2020
  ident: prgbacc2febib24
  article-title: Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data
  publication-title: Neurocomputing
  doi: 10.1016/j.neucom.2020.05.087
– volume: 53
  start-page: 197
  year: 2019
  ident: prgbacc2febib74
  article-title: Attention gated networks: learning to leverage salient regions in medical images
  publication-title: Med. Image Anal.
  doi: 10.1016/j.media.2019.01.012
– volume: 11
  start-page: 1
  year: 2021
  ident: prgbacc2febib20
  article-title: Multimodal deep learning for prognosis prediction in renal cancer
  publication-title: Front. Oncol.
  doi: 10.3389/fonc.2021.788740
– start-page: 529
  year: 2021
  ident: prgbacc2febib31
  article-title: Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information
  doi: 10.1007/978-3-030-87237-3
– start-page: 248
  year: 2009
  ident: prgbacc2febib96
  article-title: ImageNet: a large-scale hierarchical image database
  doi: 10.1109/cvprw.2009.5206848
– volume: 12
  start-page: 381
  year: 2013
  ident: prgbacc2febib3
  article-title: Midbrain and hindbrain malformations: advances in clinical diagnosis, imaging, and genetics
  publication-title: Lancet Neurol.
  doi: 10.1016/S1474-4422(13)70024-3
– volume: 33
  start-page: 563
  year: 2021
  ident: prgbacc2febib14
  article-title: Integrating pathomics with radiomics and genomics for cancer prognosis: a brief review
  publication-title: Chin. J. Cancer Res.
  doi: 10.21147/j.issn.1000-9604.2021.05.03
– volume: vol 10434
  start-page: 406
  year: 2017
  ident: prgbacc2febib28
  article-title: Deep correlational learning for survival prediction from multi-modality data
  doi: 10.1007/978-3-319-66185-8_46
– volume: 33
  start-page: 696
  year: 2021
  ident: prgbacc2febib70
  article-title: Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport
  publication-title: IEEE Trans. Knowl. Data Eng.
  doi: 10.1109/TKDE.2019.2932666
– volume: 28
  start-page: 321
  year: 1936
  ident: prgbacc2febib99
  article-title: Relations between two sets of variates
  publication-title: Biometrika
  doi: 10.1093/biomet/28.3-4.321
– volume: 140
  start-page: 189
  year: 2004
  ident: prgbacc2febib4
  article-title: Sources of variation and bias in studies of diagnostic accuracy: a systematic review
  publication-title: Ann. Intern. Med.
  doi: 10.7326/0003-4819-140-3-200402030-00010
– start-page: 3942
  year: 2018
  ident: prgbacc2febib78
  article-title: FiLM: visual reasoning with a general conditioning layer
– volume: vol 12445
  start-page: 106
  ident: prgbacc2febib90
  article-title: Towards automated diagnosis with attentive multi-modal learning using electronic health records and chest x-rays
  doi: 10.1007/978-3-030-60946-7_11
– volume: 14
  start-page: 15
  year: 2019
  ident: prgbacc2febib76
  article-title: Development and validation of a risk prediction model for radiotherapy-related esophageal fistula in esophageal cancer
  publication-title: Radiat. Oncol.
  doi: 10.1186/s13014-019-1385-y
– start-page: 1
  year: 2022
  ident: prgbacc2febib69
  article-title: M2F: a multi-modal and multi-task fusion network for glioma diagnosis and prognosis
  doi: 10.1007/978-3-031-18814-5_1
– start-page: 1103
  year: 2017
  ident: prgbacc2febib73
  article-title: Tensor fusion network for multimodal sentiment analysis
  doi: 10.18653/v1/d17-1115
– start-page: 3743
  year: 2020
  ident: prgbacc2febib102
  article-title: Deep multimodal neural architecture search
  doi: 10.1145/3394171.3413977
– volume: 37
  start-page: 1
  year: 2021
  ident: prgbacc2febib29
  article-title: GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction
  publication-title: Bioinformatics
  doi: 10.1093/bioinformatics/btab185
– volume: vol 4
  start-page: 657
  year: 2021
  ident: prgbacc2febib68
  doi: 10.1007/978-3-030-87240-3
– volume: 10
  year: 2020
  ident: prgbacc2febib87
  article-title: Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection
  publication-title: Sci. Rep.
  doi: 10.1038/s41598-020-78888-w
– start-page: pp 2097
  year: 2017
  ident: prgbacc2febib80
  article-title: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases
– start-page: 3615
  year: 2020
  ident: prgbacc2febib56
  article-title: SCIBERT: a pretrained language model for scientific text
  doi: 10.18653/v1/d19-1371
– volume: vol 2021
  start-page: 1169
  year: 2021
  ident: prgbacc2febib60
  article-title: Modeling uncertainty in multi-modal fusion for lung cancer survival analysis
  doi: 10.1109/ISBI48211.2021.9433823
– volume: 12
  start-page: 229
  year: 2015
  ident: prgbacc2febib63
  article-title: A review of feature reduction techniques in neuroimaging
  publication-title: Neuroinform
  doi: 10.1007/s12021-013-9204-3.A
– volume: 32
  start-page: 1
  year: 2021
  ident: prgbacc2febib39
  article-title: Cohesive multi-modality feature learning and fusion for COVID-19 patient severity prediction
  publication-title: IEEE Trans. Circuits Syst. Video Technol.
  doi: 10.1109/TCSVT.2021.3063952
– volume: 65
  year: 2020
  ident: prgbacc2febib32
  article-title: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks
  publication-title: Med. Image Anal.
  doi: 10.1016/j.media.2020.101789
– start-page: 14313
  year: 2021
  ident: prgbacc2febib33
  article-title: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning
  doi: 10.1109/CVPR46437.2021.01409
– volume: 14
  start-page: 470
  year: 2020
  ident: prgbacc2febib8
  article-title: Deep learning in digital pathology image analysis: a survey
  publication-title: Front. Med.
  doi: 10.1007/s11684-020-0782-9
– volume: 160
  start-page: 80
  year: 2021
  ident: prgbacc2febib13
  article-title: Integration of deep learning-based image analysis and genomic data in cancer pathology: a systematic review
  publication-title: Eur. J. Cancer
  doi: 10.1016/j.ejca.2021.10.007
– volume: 35
  start-page: i501
  year: 2019
  ident: prgbacc2febib107
  article-title: MOLI: multi-omics late integration with deep neural networks for drug response prediction
  publication-title: Bioinformatics
  doi: 10.1093/bioinformatics/btz318
– year: 2019
  ident: prgbacc2febib94
  article-title: Uniter: Learning universal image-text representations
– volume: 31
  start-page: 8816
  year: 2021
  ident: prgbacc2febib98
  article-title: Correction to: evaluation of a convolutional neural network for ovarian tumor differentiation based on magnetic resonance imaging (31 4960–71 10.1007/s00330-020-07266-x)
  publication-title: Eur. Radiol.
  doi: 10.1007/s00330-021-07854-5
– year: 2021
  ident: prgbacc2febib52
  doi: 10.1007/978-3-030-87240-3_66
– volume: 27
  start-page: 1261
  year: 2018
  ident: prgbacc2febib42
  article-title: Multimodal skin lesion classification using deep learning
  publication-title: Exp. Dermatol.
  doi: 10.1111/exd.13777
– volume: 7
  start-page: R100
  year: 2006
  ident: prgbacc2febib30
  article-title: CellProfiler: image analysis software for identifying and quantifying cell phenotypes
  publication-title: Genome Biol.
  doi: 10.1186/gb-2006-7-10-r100
– volume: 0062
  start-page: 1
  year: 2020
  ident: prgbacc2febib67
  article-title: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis
  publication-title: IEEE Trans. Med. Imaging
  doi: 10.1109/tmi.2020.3021387
– volume: vol 12262
  start-page: 529
  year: 2020
  ident: prgbacc2febib53
  article-title: Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment
  doi: 10.1007/978-3-030-59713-9_51
– ident: prgbacc2febib105
  article-title: Big self-supervised models advance medical image classification
  doi: 10.1109/ICCV48922.2021.00346
– volume: 32
  start-page: 829
  year: 2020
  ident: prgbacc2febib7
  article-title: A survey on deep learning for multimodal data fusion
  publication-title: Neural Comput.
  doi: 10.1162/neco_a_01273
– year: 2023
  ident: prgbacc2febib23
– volume: 23
  start-page: 538
  year: 2019
  ident: prgbacc2febib45
  article-title: Seven-point checklist and skin lesion classification using multitask multimodal neural nets
  publication-title: IEEE J. Biomed Health Inform.
  doi: 10.1109/JBHI.2018.2824327
– volume: 35
  start-page: i446
  year: 2019
  ident: prgbacc2febib47
  article-title: Deep learning with multimodal representation for pancancer prognosis prediction
  publication-title: Bioinformatics
  doi: 10.1093/bioinformatics/btz342
– volume: 4
  start-page: 65
  year: 2021
  ident: prgbacc2febib10
  article-title: Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
  publication-title: npj Digit. Med.
  doi: 10.1038/s41746-021-00438-z
– volume: 101
  year: 2020
  ident: prgbacc2febib9
  article-title: Deep learning for electronic health records: a comparative review of multiple deep neural architectures
  publication-title: J. Biomed. Inf.
  doi: 10.1016/j.jbi.2019.103337
– volume: 115
  start-page: E2970
  year: 2018
  ident: prgbacc2febib21
  article-title: Predicting cancer outcomes from histology and genomics using convolutional networks
  publication-title: Proc. Natl Acad. Sci. USA
  doi: 10.1073/pnas.1717139115
– volume: vol 1
  start-page: 4171
  year: 2019
  ident: prgbacc2febib55
  article-title: BERT: pre-training of deep bidirectional transformers for language understanding
– year: 2020
  ident: prgbacc2febib59
  article-title: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molecular and demographic data
  doi: 10.1007/978-3-030-59713-9_24
SSID ssj0002511920
Score 2.5565927
Snippet The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet...
SourceID proquest
pubmed
crossref
iop
SourceType Aggregation Database
Index Database
Enrichment Source
Publisher
StartPage 22001
SubjectTerms diagnosis and prognosis
fusion method
medical data
multimodal learning
Title Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review
URI https://iopscience.iop.org/article/10.1088/2516-1091/acc2fe
https://www.ncbi.nlm.nih.gov/pubmed/37360402
https://www.proquest.com/docview/2830213657
Volume 5
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIOP
  databaseName: IOP Science Platform
  customDbUrl:
  eissn: 2516-1091
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0002511920
  issn: 2516-1091
  databaseCode: IOP
  dateStart: 20190101
  isFulltext: true
  titleUrlDefault: https://iopscience.iop.org/
  providerName: IOP Publishing
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources (ISSN International Center)
  customDbUrl:
  eissn: 2516-1091
  dateEnd: 99991231
  omitProxy: true
  ssIdentifier: ssj0002511920
  issn: 2516-1091
  databaseCode: M~E
  dateStart: 20190101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1Lb9QwELbacuFCi3gtpchIcODg3cTOw25PCFpVSKUcqOgByYqdMVpBk6i7e-HAb2fGzq5UBBVCkZw4GSsTx4-x55sZxl4apVvnpBPKm1YUDRhhTFEIXbRVJsG5EDdzzj5UpxfF-8vycosdbWxh-mEc-qd4mRwFpyocAXF6hjNyJcif5azxXgbYZneUxnUFWe-df9xssJDsbGQ2qib_VPDGVLSNr_u7lBlnm5Nd9mXNZwKZfJuulm7qf_zmwvE_P2SP3RulUP4mkd5nW9A9YPYdwMAjwvCqb_FxWNFWGu8Dn1_hsMObruVd34mUI2wpn3d8VPHgOYL25otIR7ivmDvkDU_2MQ_Zxcnxp7enYoy_IHyRZ0tB8amKklR3Ofi2JlfzHrxzyHvZGDwq0L6SmfQqBMjyYFpyZ2aqBnBhA6AesR3kCp4wXkmnoZQU6loWdRV0cDWogHd8XvtWT9hs_SusH52TU4yM7zYqybW2VFmkLs9tqqwJe70pMSTHHLfQvsJ_YMfeubiFjt-gG66_OltaackcOcvt0IYJe7FuIhY7ImlXmg761cJGT2oEGqwn7HFqOxvGVK2wKjP59B8Z2Wd3Kax9Qgg9YzvL6xUcoPCzdM9jI8f07Ocxpufq8y-4UwBs
linkProvider IOP Publishing
linkToPdf http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1Lb9QwELZokRCXtojXtjyMBAcO3k2cl80NtaxaHqUHKvVmYnuMVm2TqLt76a-vx_auVAQVEsohcTJWJn5OPJ-_IeStLITVmmtWGGlZ2YJkUpYlE6WtMw5au7CY8-24PjwtP59VZynOadgL0w9p6B_7y0gUHIswAeLExM_INUM-y0lrDHcwGazbIPeromowdsPR95P1Igvaz5JnyT35p8y3pqMN_8q_W5phxpluk58rXSPQ5Hy8XOixuf6NxvE_PmaHbCVrlH6M4o_IPegeE3UAMNCANLzsrX_slrikRntHZ5d--KFtZ2nXdyymEGNKZx1Nrh5_DuC92TzIIf4rpD7QlsZ9Mk_I6fTTj_1DluIwMFPm2YJhnKqyQhdeDsY2SDlvwGjt9a9a6Y8ahKl5xk3hHGS5kxZpzWTdgv_BASiekk2vFTwntOZaQMUx5DUvm9oJpxsonL9j8sZYMSKTVXUok0jKMVbGhQrOciEUFhi6zXMVC2xE3q9zDJGg4w7Zd74eVOql8zvk6C254eqXVpXiCrclZ7nydTQib1bNRPkOiV6WtoN-OVeBUQ3Bg82IPIvtZ61Y0RS-KDO--4-KvCYPTg6m6uvR8Zc98hAj3UfQ0AuyubhawktvDy30q9DmbwAblAOr
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Deep+multimodal+fusion+of+image+and+non-image+data+in+disease+diagnosis+and+prognosis%3A+a+review&rft.jtitle=Progress+in+biomedical+engineering+%28Bristol%29&rft.au=Cui%2C+Can&rft.au=Yang%2C+Haichun&rft.au=Wang%2C+Yaohong&rft.au=Zhao%2C+Shilin&rft.date=2023-04-01&rft.issn=2516-1091&rft.eissn=2516-1091&rft.volume=5&rft.issue=2&rft_id=info:doi/10.1088%2F2516-1091%2Facc2fe&rft.externalDBID=NO_FULL_TEXT
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2516-1091&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2516-1091&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2516-1091&client=summon