EP1001 - Comparative Evaluation of ChatGPT-5 and OpenEvidence in Complex Clinical Decision-Making for Rectal Adenocarcinoma

Monday, May 11, 2026

3:40 PM - 3:45 PM ET

ePoster Presenter(s)

Ryan Meyer, MD (he/him/his)

General Surgery Resident
Los Robles Regional Medical Center
Thousand Oaks, CA, United States

Disclosure information not submitted.

Author(s)

AM

Austen Mauch, MD

General Surgery Resident
Los Robles Hospital
Thousand Oaks, California, United States

Disclosure information not submitted.
TW

Tyler Wilson, MD

General Surgery Resident
HCA Los Robles Regional Medical Center
THOUSAND OAKS, California, United States

Disclosure information not submitted.
RG

Roya Gheissari, MD

Surgical Resident
HCA Los Robles
Thousand Oaks, California, United States

Disclosure information not submitted.
TM

Tadevos Makaryan, MD

General Surgery resident
Los Robles Medical Ceneter
Thousand oaks, California, United States

Disclosure information not submitted.
SP

Shivam Pandya, MD

Resident Physician
Los Robles Regional Medical Center Department of Surgery
Thousand Oaks, California, United States

Disclosure information not submitted.

Senior Author(s)

MF

Manabu Fujita, M.D.

Surgical Oncologist
Los Robles Medical Center
Thousand Oaks, California, United States

Disclosure information not submitted.

Purpose/Background: Artificial intelligence (AI) is increasingly used for clinical decision making. Large language models (LLMs) such as ChatGPT-5 (OpenAI, San Francisco, CA) and OpenEvidence (Boston, MA) are regularly used by physicians. ChatGPT models have progressed significantly since their inception, continuing to set the standard for LLMs. OpenEvidence is a newer platform that directly links responses to verifiable primary sources. Their comparative accuracy in rectal cancer clinical decision-making has not been evaluated.

Methods/Interventions: The most recent National Comprehensive Cancer Network® (NCCN) Guidelines for Rectal Cancer were reviewed. Three complex clinical questions were generated per decision-making page, yielding 138 unique scenarios that were then classified into 3 groups: workup, treatment, and surveillance. ChatGPT-5 and OpenEvidence were queried with identical prompts. Up to 3 clarifying questions were allowed in cases of incomplete or ambiguous responses. Responses were scored independently by two physicians on a 5-point Likert scale: 5 = completely correct; 4 = correct with clarification; 3 = correct but incomplete; 2 = partially incorrect; 1 = absolutely incorrect. Correctness was defined as ≥3; Accuracy as ≥4. Proportions were compared with Fisher's exact test, and score distributions with Mann-Whitney U test.

Results/Outcomes: Of 138 questions, ChatGPT-5 achieved Correct responses in 137 (98.6%) and Accurate responses in 117 (84.2%), compared to 129 (92.8%) and 113 (81.3%) for OpenEvidence, respectively. Overall Accuracy favored ChatGPT-5 (p = 0.034), while Correctness showed no difference (p = 1.0 ). Subgroup analysis revealed: Workup/Diagnosis (N=37): ChatGPT-5 higher mean score (4.84 vs 4.38, p = 0.013); ChatGPT-5 Correctness significantly higher (100% vs 89%, p < 0.001). No significant difference in accuracy. Treatment (N=77): OpenEvidence higher mean score (4.47 vs 4.43, p=0.77). Correctness was equal (97% vs 96%, p=1.0). Accuracy was significantly higher in OpenEvidence (83.1% vs 77.9%, p = 0.006). Surveillance (N=21): ChatGPT-5 higher mean score (4.57 vs. 4.24, p=0.43). Correctness (100% vs 90%, p < 0.001), but no difference in Accuracy (86% vs. 75%, p=1.0).

Both platforms achieved >90% Correctness across all domains, creating a ceiling effect that limited statistical differentiation. However, ordinal Likert analysis and Accuracy thresholds revealed GPT-5 advantages in workup/surveillance and OpenEvidence strengths in treatment.

Conclusion/Discussion: Both ChatGPT-5 and OpenEvidence demonstrate high concordance with NCCN rectal cancer guidelines. ChatGPT-5 performed better in workup and surveillance accuracy, while OpenEvidence performs better in treatment recommendations. These results highlight the efficacy of guideline-based platforms to enhance evidence-based rectal cancer care.