Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages

The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making.

To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox. Design, Setting, and Participants: A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot. Intervention: Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model. Main Outcomes and Measures: The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey.