Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers

Large language models such as ChatGPT can produce increasingly realistic text, with unknown information on the accuracy and integrity of using these models in scientific writing. We gathered fifth research abstracts from five high-impact factor medical journals and asked ChatGPT to generate research abstracts based on their titles and journals. Most generated abstracts were detected using an AI output detector, 'GPT-2 Output Detector', with % 'fake' scores (higher meaning more likely to be generated) of median [interquartile range] of 99.98% 'fake' [12.73%, 99.98%] compared with median 0.02% [IQR 0.02%, 0.09%] for the original abstracts. The AUROC of the AI output detector was 0.94. Generated abstracts scored…

• BW
  Burroughs Wellcome Fund
• CC
  Conquer Cancer Foundation

  Award: 2022YIA-6675470300
• BC
  Breast Cancer Research Foundation
• FA
  Fanconi Anemia Research Fund
• AS
  American Society of Clinical Oncology
• NI
  National Institutes of Health

  Awards: U01TR003528, U01-CA243075, R01LM013337, K12CA139160
• SU
  Stand Up To Cancer
• NH
  National Heart, Lung, and Blood Institute

  Awards: R01LM013337, U01TR003528
• NC
  National Cancer Institute

  Awards: U01-CA243075, K12CA139160
• NI
  National Institute of Dental and Craniofacial Research

  Award: R56-DE030958
• NC
  National Center for Advancing Translational Sciences

  Award: U01TR003528