EP583 - Reduced Bias and Inflation in the 2025 Standardized Letter of Recommendation for Colorectal Fellowship Application

Purpose/Background: Letters of recommendation (LOR) play an integral role in selecting colon and rectal surgery fellowship applicants to interview and subsequently rank. In a previous study of the 2024 application cycle, there were substantial issues with score inflation, gender bias, and racial bias. The Association of Program Directors for Colon and Rectal Surgery modified the standardized LORs to address these issues. In this study, we aimed to analyze whether these changes were successful in mitigating bias and score inflation.

Methods/Interventions: All standardized letters of recommendation for the 2025 applicant cycle at a single institution were included and analyzed retrospectively. Letter writers ranked applicants across 9 domains including: technical skills, clinical knowledge, clinical judgement, interpersonal skills with patients, interpersonal skills with colleagues, situational awareness, receptivity to feedback, problem solving, and integrity. Rankings were based on descriptions which we then converted to a Likert scale. We compared category assignment for each domain by gender, self-reported race/ethnicity, type of medical degree, location of medical training, and gender of letter writers using Kruskal Wallis tests. We additionally compared questions that were similar between the 2024 and the 2025 applicant cycle to ascertain if score inflation was reduced.

Results/Outcomes: In total, 463 standardized LORs were analyzed for the 2025 applicant cycle and were compared with 356 letters from 2024 cycle. There was substantially less score inflation in 2025 in 3 of the 4 domains that were directly comparable. Gender bias in 2025 was essentially eliminated with the exception that women scored better men in interpersonal skill (mean 1.3土0.5 vs 1.2土0.5, p=0.03) and situational awareness (mean 1.5土0.6 vs 1.4土0.6, p=0.04). Racial bias was also largely eliminated except for receptivity to feedback, where black applicants had worse scores compared to other self-identified racial groups. Male and female letter writers in general rated applicants similarly across all domains, apart from clinical judgement, where female letter writers rated applicants worse than male letter writers. Female letter writers rated female applicants significantly better than male applicants in the domain of situation awareness, receptivity to feedback, and integrity. Male letter writers rated female and male applicants similarly across all the domains.

Conclusion/Discussion: Changes in the 2025 standardized letter of recommendation for colorectal fellowship applicants resulted in less score inflation and reduced gender and racial bias compared with 2024. This suggests that ongoing efforts to improve standardized letters of recommendation can be successful. Continued interval improvement is needed as substantial score inflation persists.