A visual–omics foundation model to bridge histopathology with spatial transcriptomics

Artificial intelligence has revolutionized computational biology. Recent developments in omics technologies, including single-cell RNA sequencing and spatial transcriptomics, provide detailed genomic data alongside tissue histology. However, current computational models focus on either omics or image analysis, lacking their integration. To address this, we developed OmiCLIP, a visual-omics foundation model linking hematoxylin and eosin images and transcriptomics using tissue patches from Visium data. We transformed transcriptomic data into 'sentences' by concatenating top-expressed gene symbols from each patch. We curated a dataset of 2.2 million paired tissue images and transcriptomic data across 32 organs to…

• CP
  Cancer Prevention and Research Institute of Texas

  Award: RR220017
• NH
  National Heart, Lung, and Blood Institute

  Award: R01HL169204-01A1
• NC
  National Cancer Institute

  Award: R01 CA284315
• NI
  National Institute of General Medical Sciences

  Award: R35GM150460
• NI
  National Institute of Neurological Disorders and Stroke

  Award: K22 NS112678