Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4

Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes, and designing a two-step training…

• NN
  National Natural Science Foundation of China

  Award: 62272327
• KA
  King Abdullah University of Science and Technology

  Awards: REI/1/5202-01-01, REI/1/0018-01-01, REI/1/5234-01-01, FCC/1/1976-44-01, RGC/3/4816-01-01, REI/1/4940-01-01