EP861 - Compared to Clinicians, How Well Can Artificial Intelligence Recognize Anorectal Pathology?

Purpose/Background: With the increased use of artificial intelligence (AI) for medical advice, we wanted to understand the accuracy and quality of the image interpretation functions of these platforms.

Methods/Interventions: Seven common benign anorectal diseases were evaluated: prolapsed internal hemorrhoid, thrombosed external hemorrhoid, perianal abscess, anal fissure, anal fistula, condyloma acuminata and rectal prolapse. Images were uploaded to two AI platforms (ChatGPT 5.0 and Claude 4.5 Sonnet) for image interpretation. These findings were compared with the findings in Grucela et al. (2010), who evaluated the accuracy of physicians, both attendings and residents, and students of various levels of experience in identifying the same pictures of common benign anorectal pathology. These findings were also compared to previous versions of the platforms (ChatGPT 4.0 and Claude 3.5 Sonnet). The most up to date AI platforms were also asked to interpret five images of anal cancer.

Results/Outcomes: ChatGPT 5.0 was able to identify 6 out of 7 benign anorectal pathology correctly: prolapsed internal hemorrhoid, thrombosed external hemorrhoid, perianal abscess, anal fistula, condyloma acuminata and rectal prolapse. Claude 4.5 Sonnet was able to identify 3 correctly: prolapsed internal hemorrhoid, thrombosed external hemorrhoid, and condyloma acuminata correctly. Neither ChatGPT nor Claude were able to identify any of the anal cancer images, interpreting them as open perineal wounds or surgical wounds. ChatGPT required prompting with follow-up responses/questions in order for it to provide an image interpretation. It also had the most variable answers, whereas Claude provided one to two diagnoses. ChatGPT provided more information than what was originally asked, such as recommendations to see a specialist (i.e., surgeon, colorectal surgeon, dermatologist, etc.). Compared to its previous version, Claude 4.5 Sonnet did worse and was unable to identify anal fissure and anal fistula when it previously had.

Conclusion/Discussion: ChatGPT and Claude were able to identify condyloma acuminata and rectal prolapse correctly, which were the top diagnoses physicians and students identified correctly in Grucela et al. (2010). Claude 3.5 Sonnet performed better than ChatGPT 4.0, identifying 4 out of 7 anorectal pathologies correctly. Updated version ChatGPT 5.0 outperformed both Claude 3.5 Sonnet and Claude 4.5 Sonnet. Diagnosis of malignancy is complex and involves important clinical context, making isolated image interpretation challenging. As AI continues to develop and improve its accuracy, it can potentially serve as a useful diagnostic adjuncts in identifying common benign anorectal pathologies and direct patients to appropriate specialists for professional evaluation.

Results