A critical assessment of using ChatGPT for extracting structured data from clinical notes

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT's capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral "prompt engineering" process, leveraging OpenAI's API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k)…

• CP
  Cancer Prevention and Research Institute of Texas

  Award: RP230330
• DO
  Division of Intramural Research, National Institute of Allergy and Infectious Diseases

  Award: U01AI169298
• NI
  National Institutes of Health

  Awards: RP230330, U01AI169298, R01GM140012, R01DE030656, R01GM141519, P50CA70907, U01CA249245, RP180805, R35GM136375
• NC
  National Cancer Institute

  Awards: U01CA249245, P50CA70907
• NI
  National Institute of General Medical Sciences

  Awards: R01GM140012, R35GM136375, R01GM141519
• NI
  National Institute of Dental and Craniofacial Research

  Award: R01DE030656