Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Abstract The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do…

• NS
  National Science Foundation

  Awards: 1452485, CAREER 1452485, CAREER
• UD
  U.S. Department of Veterans Affairs

  Awards: I01BX004366, I01CX002171
• RS
  Radiological Society of North America
• LM
  Leona M. and Harry B. Helmsley Charitable Trust

  Award: G-1903-03793