In-context learning enables multimodal large language models to classify cancer pathology images

Ferber, Dyke; Wölflein, Georg; Wiest, Isabella C.; Ligero, Marta; Sainath, Srividhya; Laleh, Narmin Ghaffari; Nahhas, Omar S. M. El; Müller‐Franzes, Gustav; Jäger, Dirk; Truhn, Daniel; Kather, Jakob Nikolas

doi:10.1038/s41467-024-51465-9

articleNature CommunicationsNov 21, 2024GOLD OA

In-context learning enables multimodal large language models to classify cancer pathology images

DFDyke Ferber GWGeorg Wölflein ICIsabella C. Wiest MLMarta Ligero SSSrividhya Sainath

Heidelberg University · University Hospital Heidelberg · +6 more institutions

PubMed

Indexed incrossrefdatacitedoajpubmed

Abstract

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in…

Citation impact

122

total citations

FWCI: 38.23
Percentile: 100%
References: 24

Citations per year

Authors

11

Topics & keywords

Topics

Keywords

Computer science
Artificial intelligence
Machine learning
Deep learning
Subtyping
Context (archaeology)
Image processing
Image (mathematics)

UN Sustainable Development Goals

Quality Education

No related works found for this paper.