EP152 - Diagnostic Accuracy, Management Recommendations, and Patient Education on Ostomy Care Using ChatGPT: An Inter-Reviewer Reliability Study

Purpose/Background: Ostomy-related complications are common, and patients often struggle to troubleshoot on their own until they can receive nuanced management from specialized ostomy nurses and colorectal surgeons. Large Language Models (LLMs) such as ChatGPT have been proposed as decision-support tools for diagnosis, treatment guidance, and patient education. However, their reliability and clinical applicability of these outputs remain uncertain. The aim of this study was to evaluate the quality and consistency of ChatGPT-generated recommendations for ostomy-related complications.

Methods/Interventions: Ten ostomy vignettes including various stomal related skin issues, prolapse and hernia were evaluated using ChatGPT prompts requesting diagnosis, treatment recommendations, and patient education material. Two colorectal surgeons and two ostomy nurses independently rated ChatGPT outputs for appropriateness and comprehensiveness (1-5 Likert scale) and indicated willingness-to-use in practice (Yes/No). Inter-rater reliability was measured using quadratic-weighted Cohen's κ for pairwise comparisons, Fleiss' κ for multi-rater agreement, and bootstrap 95% confidence intervals (CI). Mean scores across reviewer groups were calculated.

Results/Outcomes: Across all cases, the mean diagnostic appropriateness score was 4.0 (SD 1.2) and diagnostic comprehensiveness 4.1 (SD 1.1). Treatment recommendations averaged 3.9 for appropriateness and 3.9 for comprehensiveness, while patient education content averaged 4.1 and 4.2, respectively. Willingness-to-use was endorsed in 80% of cases by surgeons compared with only 60% by nurses. Agreement among reviewers was low: among surgeons, κ ranged from 0.04 (95% CI −0.04 to 0.40) for diagnostic appropriateness to 0.44 (95% CI −0.19 to 0.78) for treatment comprehensiveness, while other domains showed negative or negligible agreement. Nurses demonstrated similarly limited concordance, with κ values between −0.17 and 0.33 and no agreement for willingness-to-use (κ=0.00, 95% CI −0.27 to 0.00). When all four raters were analyzed together, Fleiss' κ values across diagnostic, treatment, and education domains remained low ( < 0.20). The ChatGPT-generated patient education pamphlet was rated at a mean clarity of 3.8 and comprehensiveness of 4.0, with readability corresponding to an eighth-to-ninth grade level using the Flesch Kincaid score.

Conclusion/Discussion: ChatGPT demonstrated moderate overall performance in generating diagnostic, management, and patient education content for ostomy-related complications, with higher acceptance by surgeons than nurses. Although average scores suggest potential usefulness, inconsistent ratings and the presence of unsafe elements emphasize the need for further fine-tuning with expert oversight. Although these findings support a possible role for ChatGPT as an adjunct in patient education and early decision making for ostomy related issues, model training and high quality data repository is needed to improve its potential use as an adjunct to clinician expertise.