EP1062 - ChatGPT Responses to Commonly Asked Colon and Rectal Cancer Questions from a Patient Perspective

ePoster Presenter(s)

• 

  Alyssa Habermann, MD (she/her/hers)

  General Surgery Resident
  Virginia Commonwealth University
  Virginia Commonwealth University
  Richmond, VA, United States

  Disclosure information not submitted.

  View ePoster

Author(s)

• TG

  Tania Gupta, n/a

  Medical Student
  Virginia Commonwealth University School of Medicine
  Richmond, Virginia, United States

  Disclosure information not submitted.
• SC

  Stephen Cohen, MD, MBA

  Section Chief, General Surgery Richmond VAMC
  VCU Medical Center
  Richmond, Virginia, United States

  Disclosure information not submitted.
• SS

  Syed A. Shah, MD FACS

  Asst. Prof. Surgery
  Central Virginia VA Healthcare System/ Virginia Commonwealth University
  Richmond, Virginia, United States

  Disclosure information not submitted.
• AK

  Adam Khader, MD, PhD, FACS, FSSO

  Associate Professor of Surgery
  Central Virginia VA Health Care System - Richmond, VCU School of Medicine
  Richmond, Virginia, United States

  Disclosure information not submitted.
• KF

  Kathryn Fong, MD, FSVS, FACS

  Surgeon
  Central Virginia VA Health Care System / Virginia Commonwealth School of Medicine
  Richmond, Virginia, United States

  Disclosure information not submitted.

Senior Author(s)

• MA

  Michael Amendola, MD MEHP

  Professor of Surgery
  VCU/VA Health Care System
  Richmond, Virginia, United States

  Disclosure information not submitted.

Purpose/Background: Patients are increasingly turning to large language models (LLMs) such as ChatGPT for medical guidance, prompting concerns around medical competence. This study assesses the proficiency of ChatGPT 4.o in answering patient questions regarding the surgical management of colon and rectal cancer.

Methods/Interventions: ChatGPT 4.o (accessed 12/16/2024) was prompted with 10 patient-based questions regarding the surgical management of colon and rectal cancer. A five-surgeon panel (one boarded general surgeon, two boarded colorectal surgeons, and two boarded surgical oncologists) evaluated each response for completeness, accuracy, and effective communication (5-point Likert scale assessment). ANOVA was used to compare difference between response categories and unpaired t-test was used to compare scores between colon and rectal cancers.

Results/Outcomes: The average scores of all colon and rectal questions were 13.02 ± 0.57 and 12.94 ± 0.55, respectively out of a total score of 15. The highest scoring colon question related to postoperative length of stay (13.80 ± 1.30) and the lowest scoring colon question pertained to operative time (12.20 ± 1.92). The highest scoring rectal questions discussed bowel continence and return to normal activities (13.60 ± 1.34 and 13.60 ± 1.52) and the lowest scoring rectal question explained surgical procedures for removing rectal cancer (11.80 ± 2.28). Average scores (out of 5) for completeness, accuracy, and communication were 4.56 ± 0.69, 4.35 ± 0.73, and 4.16 ± 0.71, respectively (p < 0.001). There was no significant difference between evaluator scores for colon and rectal questions (p>0.05).

Conclusion/Discussion: ChatGPT 4.o is capable of answering commonly asked patient questions regarding colon and rectal cancer with reasonable completeness, accuracy, and effective communication with statistically significant variations between these categories.

Figure 1
ChatGPT prompts. ChatGPT 4.0 was asked the following questions (from a patient perspective) regarding colon and rectal cancer with a 300-word limit. The same questions were asked for colon and rectal cancer.
Table 1
Descriptive summary of evaluator's ratings of each question. Question stems here are abbreviated; full question stems can be found in Figure 1. Each question had a maximum possible score of 15.