articleNature ChemistryMay 20, 2025HYBRID OA

A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

AMA.H. MirzaNANawaf AlamparaSKSreekanth KunchapuMRMartiño Ríos-GarcíaBEBenedict Emoekabu

Helmholtz Institute Jena · Friedrich Schiller University Jena · +9 more institutions

PubMed
Indexed incrossrefpubmed

Abstract

Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However,…

Citation impact

52
total citations
FWCI
21.09
Percentile
100%
References
68
Citations per year

Authors

35

Topics & keywords

Topics
Keywords
  • Benchmarking
  • Harm
  • Process (computing)
  • Value (mathematics)
  • Chemistry
  • Management science
  • Computer science
  • Cognitive science
UN Sustainable Development Goals
  • Quality Education
No related works found for this paper.

Funding