Evaluating large language models on medical evidence summarization

Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical…

• NS
  National Science Foundation

  Awards: P30CA013696, 2145640, 2303038, 2019844
• NI
  National Institutes of Health

  Awards: P30CA013696, Bridge2AI, 4R00LM013001
• NC
  National Cancer Institute

  Award: P30CA013696
• UN
  U.S. National Library of Medicine

  Awards: 4R00LM013001, 5R01LM009886