EP530 - Can AI Think Like a Surgeon? Evaluating Large Language Models in Colorectal Decision-Making.

Purpose/Background: Large language models (LLMs) are increasingly being applied in clinical settings, yet their diagnostic and treatment accuracy for surgical conditions remain uncertain. This study primarily aimed to evaluate the diagnostic and treatment accuracy of LLMs for common colorectal conditions encountered in the emergency room setting with secondary aims to assess their ability to determine procedure type and urgency, and to examine whether access to imaging improved performance or confidence.

Methods/Interventions: De-identified history and physical data from patients who underwent colorectal evaluation in the emergency department over a one-year period were extracted from the electronic medical record. Each case was entered into four LLMs (ChatGPT-o3, ChatGPT-4o, Claude Sonnet 3.7, and Gemini 2.5 Pro ) using a custom LLM-experiment runner tool built in Python that utilized our standardized prompt with two variations (with and without imaging). Following our standardized prompt, the tool automatically processed these cases with model parallelism, a temperature of 0.1 for reproducibility, automatic retry logic of 3 attempts per API call and built-in response parsing to standardize the outputs. The prompt specifically instructed the models to provide a primary diagnosis, treatment plan, surgical recommendation, and a self-reported confidence rating for each response. Three independent reviewers conducted blinded chart reviews to verify the actual care provided, and model outputs were scored for diagnostic, treatment, procedural, and urgency accuracy relative to the patient's real-world management. A qualitative thematic analysis was performed to identify recurring patterns in diagnostic reasoning and sources of error.

Results/Outcomes: Among the 80 patients, 52% presented with colitis or complications of inflammatory bowel disease, 15% with perianal disease, and the remainder with bowel obstruction, leak, or colon perforation; 35% required operative intervention. ChatGPT-4o demonstrated the highest diagnostic accuracy (80%), treatment accuracy (80%) and achieved the greatest overall combined diagnostic and treatment accuracy (70%) without imaging whereas ChatGPT-o3 performed best with imaging with a diagnostic accuracy of 90% (Figure 1). Across all models, mean diagnostic accuracy improved significantly with imaging (74% vs. 87%, p< 0.05). ChatGPT-o3 most accurately identified the appropriate surgical procedure (96%) while Gemini 2.5 Pro achieved the highest accuracy in classifying surgical urgency (63%). Across all models, accuracy of treatment recommendations and surgical procedural selection did not improve with imaging.

Qualitative analysis revealed that diagnostic errors most occurred due to failure to identify disease extent (perforation, abscess, sepsis), and misattribution of etiology (e.g., ischemic vs diverticular colitis). Treatment errors were most often due to incomplete management (missing antibiotics, bowel rest, or drainage), incorrect surgical timing or procedure, and under-treatment relative to disease severity.

Conclusion/Discussion: LLMs showed moderate-to-high accuracy in colorectal emergency management, particularly with imaging. Although ChatGPT-4o performed most consistently, these models should be applied with caution, as they often miss the subtle contextual and clinical nuances that guide surgical decision-making. Used responsibly, LLMs may complement, but not replace, expert clinical judgment.

Diagnostic, Treatment, Procedural Accuracy, and Confidence of Large Language Models With and Without Imaging