EP151 - Can a Publicly Available LLM Identify Pelvic Anatomy in Colorectal Operative Images?

Purpose/Background: Large language models (LLMs) could accelerate educational labeling of operative images, but their anatomic accuracy is unclear. General-purpose LLMs are now widely available and easily accessible for use by surgical learners. We evaluated an LLM's ability to identify pelvic structures relevant to colorectal surgery.

Methods/Interventions: Eighteen intraoperative images from publicly available videos (Youtube) were annotated for presence and location or absence of seven targets: pelvic brim, colon or rectum, urinary bladder, uterus, ovaries, iliac vessels, and ureters. ChatGPT 5.o thinking was prompted twice in temporary chat on the same images: (1) Unlabeled (no overlays) and (2) Pre-labeled with five transparent overlays that were manually added (pelvic brim, colon or rectum, bladder, uterus, ovaries), with iliacs and ureters unlabeled. Pre-labeled images were uploaded along with unlabeled images for reference. For each structure and scenario, we recorded true positive (TP), true negative (TN), false positive (FP), and false negative (FN), and calculated sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and F1. Primary outcomes were F1 and accuracy for identifying pelvic structures, primarily iliac vessels and ureters.

Results/Outcomes: Across 126 structure-decisions per unlabeled and pre-labeled scenarios (7 targets per image, 18 images), unlabeled images achieved 38.9% accuracy (TP 31, TN 13, FP 10, FN 59), F1 0.47, sensitivity 0.34, and specificity 0.56. With pre-labels, accuracy increased to 43.7% (TP 41, TN 9, FP 14, FN 52) with higher F1 0.55 and sensitivity 0.44, but lower specificity 0.39, indicating more false positives. In structures, performance improved most for colon and rectum (accuracy 72% to 94%, F1 0.83 to 0.97) and the pelvic brim (25% to 47%, F1 0.40 to 0.64), showed no change for uterus (62% to 62%, F1 0.73 to 0.75), and declined for ovaries (44% to 31%, F1 0.53 to 0.27). Despite assistance, LLM was persistently unable to identify iliac vessels (12% to 6%, F1 0.22 to 0.12), and ureters (0% to 0%).

Conclusion/Discussion: General-purpose LLM provided unacceptably low rates of identifying pelvic structures, except the most recognizable structures. Poor performance could not be salvaged by adding manual annotations, as the LLM would still mislabel correctly labeled structures and was persistently unable to deduce the location of ureters and iliac vessels. LLMs are not ready to supplement the visual aspect of learning that is important to surgical residents. Larger studies including visual language models (VLMs) trained for surgical settings would be helpful to validate these findings.

Surgical view of pelvic brim, labeled by AI
Example of AI labeling of seven structures (right) compared to original image (left).