EP302 - ChatGPT vs Surgical Residents: Evaluating Knowledge of Benign Anorectal Disease Using Clinical Practice Guidelines

Purpose/Background: Large Language Models (LLMs) such as ChatGPT have rapidly emerged as accessible sources of medical knowledge, prompting increasing interest in their reliability within surgical disciplines. Although these models show promise for medical decision-making, their accuracy in subspecialized areas of surgery has not been rigorously evaluated. The purpose of this study is to assess ChatGPT’s knowledge of benign anorectal disease using assessments based on published clinical practice guidelines by the American Society of Colon and Rectal Surgeons (ASCRS). ChatGPT’s performance will be compared to that of general surgery residents to establish a benchmark for its potential role in clinical support.

Methods/Interventions: Five separate 20-question multiple-choice examinations were developed using the ASCRS clinical practice guidelines for benign anorectal diseases, including anal fissures, rectal prolapse, pilonidal disease, hemorrhoids, and the complex of anorectal abscess/fistula-in-ano/rectovaginal fistula. ChatGPT-5 was then prompted to complete each examination and its responses were recorded. Five categorical general residents were asked to complete the same set of examinations. For each participant and ChatGPT, the number of correct and incorrect responses were documented. Individual test scores were combined into a single individual score out of 100 questions.

Results/Outcomes: A Mann–Whitney U test was conducted to evaluate differences in test performance between general surgery residents (N = 5) and the artificial intelligence model (ChatGPT-5, N = 1). The analysis compared scores across five guideline-based assessments—anal fissures, rectal prolapse, pilonidal disease, hemorrhoids, and anorectal abscess/fistula-in-ano/rectovaginal fistula—as well as overall performance. No statistically significant differences were found between resident and ChatGPT scores across any of the categories: anal fissures (U = 0.000, p = 0.114), rectal prolapse (U = 0.000, p = 0.143), pilonidal disease (U = 0.000, p = 0.137), hemorrhoids (U = 0.000, p = 0.137), anorectal abscess/fistula-in-ano/rectovaginal fistula (U = 1.000, p = 0.343), or overall score (U = 0.000, p = 0.143). Therefore, the null hypothesis—that there is no statistically significant difference between general surgery residents and ChatGPT in guideline-based knowledge of benign anorectal disease—was retained.

Conclusion/Discussion: ChatGPT demonstrated a high level of accuracy in answering guideline-based questions on benign anorectal disease, achieving scores comparable to those of general surgery residents. Although ChatGPT’s overall performance exceeded the resident mean, statistical analysis revealed no significant difference between groups. These findings suggest that large language models may possess a knowledge base consistent with that of surgical trainees when evaluated against established clinical practice guidelines. However, while ChatGPT performed well on structured assessments, its ability to integrate clinical context, exercise judgment, and adapt to patient-specific factors remains unproven. Further investigation is warranted to assess its reliability and potential role as a reference tool in surgical decision-making.