Interactive computer-aided diagnosis on medical image using large language models

Computer-aided diagnosis (CAD) has advanced medical image analysis, while large language models (LLMs) have shown potential in clinical applications. However, LLMs struggle to interpret medical images, which are critical for decision-making. Here we show a strategy integrating LLMs with CAD networks. The framework uses LLMs’ medical knowledge and reasoning to enhance CAD network outputs, such as diagnosis, lesion segmentation, and report generation, by summarizing information in natural language. The generated reports are of higher quality and can improve the performance of vision-based CAD models. In chest X-rays, an LLM using ChatGPT improved diagnosis performance by 16.42 percentage points compared to…

• NN
  National Natural Science Foundation of China

  Awards: 62131015, 82394432, STI 2030, U23A20295
• SA
  Science and Technology Commission of Shanghai Municipality

  Award: 21010502600