The AAMC is regularly exploring ways to improve the residency application process through initiating its own research, supporting research led by other stakeholders in the academic medicine community, and making improvements to the ERAS® program. Not all research yields changes to the application process, but each project has the potential to inform the AAMC — and its partners — in creating future innovations.
- Active AAMC research and innovations
- Past AAMC research and innovations
- Resources from Academic Medicine
- Collaborations with specialties
- ERAS data supporting residency research initiatives
Active AAMC research and innovations
-
USMLE Step 1 Exam and COMLEX-USA LEVEL 1 Pass-Fail Transition Collaborative Research to Begin: In February 2020, the National Board of Medical Examiners (NBME) announced that it would only report pass/fail outcomes for USMLE Step 1 exams taken on or after January 26, 2022, with the National Board of Osteopathic Medical Examiners (NBOME) following with a similar announcement for COMLEX-USA LEVEL 1 exam scores. This in part led the AAMC, NBME, National Resident Matching Program® (NRMP®), American Association of Colleges of Osteopathic Medicine (AACOM), and NBOME to collaborate on a research program that evaluates the impact of this change both during and after the transition to residency. The multi-year research agenda will establish a baseline before pass/fail outcomes are reported for all applicants and explore changes over time. Research results will provide the community with information about current and planned future practices for students and advisors and identify any need for training and guidance regarding best practices for appropriate use, fairness, and validity in selection.
-
Supplemental ERAS application: For the ERAS 2023 cycle, residency programs within 16 specialties may request their applicants to complete the supplemental ERAS application, in addition to the MyERAS application. In its second year of use, the supplemental application is designed to help students share more about themselves and assist program directors in finding applicants that fit their programs’ setting and mission.
During the ERAS 2022 season, the association collaborated with three specialties for this project’s operational pilot year. The current analysis of the supplemental ERAS application for year 1 is available on the Data and Reports page.
The supplemental ERAS application requests that applicants provide responses to geographic preferences (by region and by urban or rural setting); their most meaningful experiences and other impactful life events, if applicable; and program signals.
-
Application caps: Using archival data, the AAMC has been modeling the implications of different levels of application caps on the probability of entering training in general, and by applicant type, gender, and race/ethnicity. The goal of this work is to identify any unintended consequences of application caps and to use empirical evidence to inform changes to the residency application process. In 2022, this research has been extended to a broader sample of specialties, and work will continue to refine preliminary models. Despite certain limitations regarding model fit and available data, preliminary findings show that it is possible to reduce the number of applications submitted without impacting estimated entry rates, although implementing caps may disadvantage DO and IMG applicants. Further details on these preliminary findings and potential impact for differing gender and race/ethnicity groups were shared at the Learn Serve Lead meeting in 2022.
-
Preference signaling: In 2020, the AAMC started a partnership with otolaryngology to evaluate their preference signaling program. The association’s team of researchers with expertise in selection are serving in a consulting role to help design the research program, analyze data, and report out results. After reporting and sharing initial findings, detailed follow-up analyses from the 2020 administration linking signals to interview and Match outcomes were published in JAMA Network Open, with the results of this work helping to inform future adoption of preference signaling in the residency application process.
-
Program characteristics and Match® data: Since 2018, the AAMC and eight other organizations involved in the transition to residency have offered a free tool called Residency Explorer™. The Residency Explorer tool was created to empower applicants as they consider where to apply for residency. It allows applicants to compare themselves with previously matched applicants and residents entering a program and to explore many program characteristics.
-
Virtual interviews and anti-bias training: In 2020, in response to the pandemic, the AAMC’s team of selection experts developed training materials and best practices to help applicants and program directors navigate virtual interviews. These free resources included strategies to help applicants prepare for virtual interviews and guidance for program directors about setting up and implementing a fair virtual interview process. These resources continue to help students and program directors.
-
Data to help medical schools improve their curricula: The AAMC Resident Readiness Survey, a new process to help program directors provide feedback to medical schools about the performance of their graduates for continuous quality improvement of the curricula is now in Year 3 and fully operational. The survey’s data summaries for Year 1 and for Year 2 are available. For the 2022-2023 academic year, the AAMC surveyed program directors from December 2022 to March 2023. Reports will be available in late spring 2023. can be updated with the Y3 information.”
-
Developing Foundational Competencies for Undergraduate Medical Education: The Association of American Medical Colleges (AAMC), the American Association of Colleges of Osteopathic Medicine (AACOM), and the Accreditation Council for Graduate Medical Education (ACGME) are co-sponsoring an initiative to create a common set of foundational competencies for use in undergraduate medical education programs in the United States. This initiative aligns with recommendations outlined in the Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC) and is part of a comprehensive effort by all three organizations to improve the transition to residency. Throughout 2022, the project team has collected input from hundreds of academic medicine professions through a series of discussion forums, webinars and meetings. Please contribute your input to help shape the future of undergraduate medical education.”
Throughout 2022, the project team has collected input from hundreds of academic medicine professions through a series of discussion forums, webinars and meetings. Please contribute your input to help shape the future of undergraduate medical education.
Past AAMC research and innovations
The following projects have been discontinued or are no longer active.
-
Data to help students reduce number of applications: From 2017-2021, the AAMC provided annual data to help students anchor their initial thinking about the number of residency programs they should apply to. There is a point of diminishing returns (which varied by specialty, USMLE Step 1 score, and applicant type) where submitting one more application did not necessarily increase the student’s likelihood of entering a residency program. The association also provided data showing overall entrance rates for each specialty, the portion of applicants who enter training in another specialty, and entrance rates for applicants who fail their first attempt of the USMLE Step 1 exam.
-
Video or virtual interviewing: The AAMC completed a multiyear Standardized Video Interview pilot project to explore how to create a uniform video interview process for residency applicants and programs. The project informed future work supporting unbiased virtual interview experiences.
Recent AAMC studies on the Core EPAs
-
Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students’ Self-Assessed Skills by Specialty: This article describes results of an analysis of Association of American Medical Colleges (AAMC) Graduation Questionnaire (GQ) data for 12,308 medical students who graduated from U.S. MD-degree granting medical schools in the 2018-2019 academic year. Students self-assessed their skills in each of the thirteen activities described by the AAMC as Core Entrustable Professional Activities (EPAs) for entering residency. Students’ self-assessed skills varied across EPAs and by specialty, findings that can inform collaborative efforts by undergraduate and graduate medical educators to ensure graduates’ acquisition of the skills expected of them at the start of residency.
-
In September 2022, the AAMC published a monograph on the 10-school Core Entrustable Professional Activities (EPAs) For Entering Residency Pilot. This monograph includes detailed description of the activities of the 7-year pilot (2014-2021) and the major outcomes as well as an annotated bibliography of the peer-reviewed publications of the entire pilot team to date. The monograph is available (at no cost) for download in the AAMC Store.
-
An article entitled “Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020” was published in JAMA Network Open in September 2022. This article describes progress in data collection and entrustment decision-making at four participating schools in the Core EPAs pilot. The authors cautioned, however, that substantial gaps remained for a subset of Core EPAs.
Resources from Academic Medicine
The Undergraduate to Graduate Medical Education Transition as a Systems Problem: A Root Cause Analysis
This article describes the work of the Undergraduate Medical Education to Graduate Medical Education Review Committee to apply a quality improvement approach and systems thinking to explore the underlying causes of dysfunction in the UME-GME transition.
Reimagining the Transition to Residency: A Trainee Call to Accelerated Action
The authors reflect on their experience as the trainee representatives on the Undergraduate Medical Education to Graduate Medical Education Review Committee. They call for the medical education community—and the Coalition member organizations in particular—to accelerate fully implementing the UGRC recommendations that were released in August 2021.
Blue Skies with Clouds: Envisioning the Future Ideal State and Identifying Ongoing Tensions in the UME-GME Transition
The authors describe how the Undergraduate Medical Education to Graduate Medical Education Review Committee envisioned an ideal state for the transition to residency, analyzed 3 persistent key tensions in the transition, and devised strategies to balance the upsides and downsides inherent to these tensions.
A Simulation of the Effect of Interview Caps on the National Resident Matching Program Match in Otolaryngology
The authors created a simulation otolaryngology Match to analyze the effects of interview caps. The simulation included 120 residency programs, 360 positions, and 570 applicants (each assigned a competitiveness weighted value).
Interview Without Harm: Reimagining Medical Training’s Financially and Environmentally Costly Interview Practices
Travel for medical school, residency, and fellowship interview generates carbon emissions that damage the environment and population health. The authors call for creative, structural reforms that reduce the need for such travel while increasing equity.
"Program Directors’ Assessment of Postgraduate Year 1 Residents’ Readiness for Graduate Medical Education: Lessons and Insights From the Association of American Medical Colleges Inaugural 2020–2021 Resident Readiness Survey"
A newly published abstract describes the AAMC’s Resident Readiness Survey (RRS) and includes evidence about the feasibility of this standardized process for program directors to provide feedback to medical schools about their graduates.
"Responding to Recommended Changes to the 2020–2021 Residency Recruitment Process From a Diversity, Equity, and Inclusion Perspective"
This article examines recommended changes to the 2020–2021 residency recruitment process from a diversity, equity, and inclusion perspective, highlighting new opportunities created by these recommendations and detailing challenges that programs must carefully navigate to ensure equity for all candidates.
“The Otolaryngology Residency Program Preference Signaling Experience”
As part of the 2021 Match®, otolaryngology applicants could participate in a preference signaling process — signaling up to five programs of particular interest at the time of application submission. In this report, the authors describe that preference signaling process, and they report that signaling markedly increased applicants’ ability to obtain interview offers from programs of particular interest and that this effect was present across the spectrum of applicant competitiveness.
A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace
In this single-school study, conducted at a school participating in the Core EPAs pilot, the authors report on the results of their study comparing modified versions of the Ottawa and Chen scales on workplace-based assessment forms.
"Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study"
In this article, the authors report on the results of the first round of theoretical entrustment decision-making at four of the participating pilot schools.
"Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters"
In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates’ readiness for residency. The purpose of this study is to investigate the validity evidence of two proposed workplace-based entrustment scales.
“Development and Validation of a Machine Learning-Based Decision Support Tool for Residency Applicant Screening and Review”
In this article, a multidisciplinary team discusses the development and validation of a machine-learning based decision support tool for residency applicant screening and review as a way to address the growing number of applications that residency programs receive.
“Using Machine Learning in Residency Applicant Screening”
Complementing the article above, this episode of the “Academic Medicine Podcast” describes the development of a decision support tool that incorporates machine learning and the use of that tool in residency applicant screening. It also looks at the residency application process and potential ways that artificial or augmented intelligence might mitigate current challenges.
Collaborations with specialties
Some examples of how the AAMC has supported or developed connections with other researchers and leadership organizations within the academic medicine community include:
-
Exploring working with the Association of Professors of Gynecology and Obstetrics on modeling and evaluating the early result acceptance program as part of the AAMC’s efforts to improve the overall residency application process and contribute to evidence-informed innovations.
-
Continuing the Careers in Medicine program’s annual Virtual Specialty Forum providing medical students exposure with specialty representatives at 21 participating specialties.
-
Regularly maintaining connections with specialty organizations (such as OB-GYN and otolaryngology) to maintain awareness of the innovations they are researching.
-
Facilitating discussions between the undergraduate medical education and graduate medical education (GME) communities (student affairs and program directors).
ERAS data supporting transition to residency research initiatives
In addition, ERAS data are used to support a variety of internal and external research initiatives, including studies promoting diversity and equity in graduate medical education.
-
The AAMC continues to work with national organizations and agencies, including the National Institutes of Health, which has leveraged ERAS data by conducting specialty-specific analyses. These analyses examined ERAS applicant trends for specific specialties to determine applicant characteristics that are correlated with successful matriculation. Studies thus far have included specialties of urology, pediatric surgery, colon and rectal surgery, thoracic surgery, and complex general surgical oncology.
-
ERAS data are also used to identify inequities in the medical education continuum. For example, Dowin Boatright, MD, MBA, MHS, analyzed ERAS data to determine racial disparities in Alpha Omega Alpha honor society selection.
-
The AAMC makes data available that display historical and current ERAS data for applicants, GME programs, and researchers. These tables, which present data by applicant type, medical school type, sex, race/ethnicity, and other identifiers are used to understand applicant trends by specialty as well as the total applicant population as a whole. For example, research examined data in surgery residency and fellowship programs.
-
Other AAMC data, such as GME Track® and Faculty Roster information, are used by academic medicine researchers to examine diversity issues in residency. For example, recent articles examined the effect of surgery faculty diversity on general surgery resident attrition and women in leadership and their influence on gender diversity in plastic surgery programs.