Each year, some 90,000 aspiring doctors take the MCAT® exam, and scores of admissions teams comb through the results as part of their search for talented future physicians. That means the test must do its job well.
So, in 2015 the AAMC redesigned the exam, adding new topics and skills that applicants need to tackle medical school and 21st century medicine: biochemistry, psychology, sociology, and scientific reasoning.
Since then, a team of researchers has been exploring whether the updated test accurately predicts success in medical school, how well underrepresented groups fare on it, and much more. Now, for the first time, they’ve published their results in four Academic Medicine papers.
“It’s essential for admissions teams to know that MCAT scores predict success in medical school. But the research shows more than that the exam is doing its job.”
Cynthia Searcy, PhD
“After extensive study, the MCAT Validity Committee has several major insights based on examinees who have finished their first year of medical school,” says Cynthia Searcy, PhD, AAMC senior director of MCAT research and development. “It’s essential for admissions teams to know that MCAT scores predict success in medical school. But the research shows more than that the exam is doing its job. Importantly, the papers also show how schools can use the MCAT exam to support their broader goal of creating the talented, diverse physician workforce that our nation needs.”
Below are some answers to key questions about the test, insights from the experts who spent years studying it, and what’s going to happen next.
Do MCAT scores predict success in medical school?
The researchers report finding a strong relationship between MCAT scores and students’ success in the first year of medical school.
Probing the exam’s validity was no simple matter. The team studied nearly 8,000 students across the country — and did a deeper dive at 16 medical schools to explore performance in courses ranging from medical ethics to molecular biology.
“The researchers found that, on average, students with higher MCAT scores perform better in their first-year courses,” Searcy explains. “In addition, they found these results at a range of schools that vary in many ways, including their academic missions, applicant pools, teaching practices, and approaches to grading. That’s good news.”
What’s more, numbers crunched since the studies’ publication indicate that the test predicts how well students perform in second-year courses as well as on Step 1 of the United States Medical Licensing Examination, major milestones in moving toward on-time graduation from medical school.
The researchers also wanted to know whether MCAT scores added value beyond what undergraduate GPAs offer in predicting medical school success. The results showed that both measures predicted well — but that using the two metrics together worked much better.
“When you look at sources of information, sometimes one alone may do the job, but that’s not the case here,” says Joshua Hanson, MD, MPH, one of the validity paper’s authors and associate dean for student affairs at the University of Texas Health San Antonio Joe R. and Teresa Lozano Long School of Medicine. “Admissions committees can be reassured that it makes sense to continue using both the MCAT and GPA.”
Is the MCAT exam unfair to certain groups?
When it comes to people of all different races, ethnicities, socioeconomic backgrounds, and genders, the new exam predicts medical school success equally well, the research shows.
“For given MCAT scores, students from different groups are as successful in their medical school courses as students from other groups with the same score,” says Hanson. So, for example, a white woman with a 505 could be expected to have the same chance of academic success as a black man with a 505. “We didn’t even see small statistical differences that might raise concerns,” he adds.
And the test’s ability to predict success was solid in terms of another significant measure: students advancing to year 2 of medical school. For example, based on their MCAT scores, 95% of medical students from groups that are underrepresented in medicine (URM) would be expected to move up — and 94% achieved that milestone.
“For given MCAT scores, students from different groups are as successful in their medical school courses as students from other groups with the same score. We didn’t even see small statistical differences that might raise concerns.”
Joshua Hanson, MD, MPH
University of Texas San Antonio Long School of Medicine
But, as with the old exam, research showed that examinees from underrepresented and lower socioeconomic status groups on average scored lower on the new exam than other test-takers, patterns also found on other academic admissions tests, undergraduate GPAs, and elsewhere throughout the educational spectrum.
If the MCAT exam is fair, why do different groups score differently on it?
Often, URM students battle major obstacles to academic achievement, including structural racism, lack of access to mentors, and poor educational resources, researchers note. In addition, inequities start as early as daycare and continue through college, say the authors of a paper that outlines proposals for tackling such systemic woes.
In an attempt to address some of these concerns, the MCAT designers took such steps as poring over course offerings at numerous colleges, including underresourced and minority serving institutions, to make sure that students had opportunities to learn the content tested on the new exam.
In addition, many medical schools work hard to put scores into context, creating a holistic picture that includes personal traits and lived experiences, notes Ngozi F. Anachebe, MD, PharmD, associate dean of admissions and student affairs at Morehouse School of Medicine and a co-author of one of the papers. In fact, acceptance rates among various groups are more similar than one might predict given the differences in their scores and GPAs.
For her part, Anachebe also encourages medical schools to provide services and environments that help URM students flourish. “We create a collaborative atmosphere at our school so students don’t feel uncomfortable asking questions or getting help. It’s all for one and one for all. Everybody, from housekeeping staff to faculty members, is really invested in seeing the students we accept succeed.”
What does the test say about students who don’t have the highest scores?
Researchers uncovered some intriguing information when they looked at test-takers who scored in the middle third of the MCAT scoring scale.
For one, people with scores in that middle range (495-504) did comparatively well in medical school. Consider some numbers: 95% of students with middle-third scores moved from year 1 to year 2 of medical school on time compared with 98% of those with upper-range scores (505-528).
And, the researchers found, schools hoping to increase diversity are well served to look in that middle third. Tucked away in that group were more students from low-income, rural, and URM backgrounds.
That’s vital information given that, for example, black and Hispanic students are more likely to go on to care for patients who live in medically underserved communities. In addition, Anachebe notes, diverse students provide important perspectives. “Some medical school students have never been around poor people,” she explains. “To have classmates who can discuss what it’s like to be poor, how hard it may be just to pay for transportation to a doctor’s appointment, can be eye-opening.”
What’s more, she says, less-resourced students with average academic credentials sometimes are spurred by passion. “That fire in the belly is so important,” she says. “It can inspire students to dig deep to study hard and learn the material so they can go back and uplift their communities.”
What was surprising about how students prepare for the test?
At the time of the new exam, the AAMC, together with the Robert Wood Johnson Foundation and the Khan Academy, released more than 1,000 free MCAT study videos, and it provides many free and low-cost materials, from flashcards to practice questions. In addition, the AAMC did extensive outreach to ensure that all students, especially those at underresourced schools, knew about ways to prepare for the new test.
“Fantastic physicians come from everywhere — including very modest backgrounds. So it’s critically important to understand how to reach those individuals to ensure they understand the test and know how to prepare for it.”
Joshua Hanson, MD, MPH
University of Texas San Antonio Long School of Medicine
But the researchers discovered a troubling fact: Test-takers from lower socio-economic backgrounds and those who attend less-resourced colleges used many free and low-cost prep materials at lower rates than their more-resourced peers.
“Fantastic physicians come from everywhere — including very modest backgrounds,” says Hanson. “So it’s critically important to understand how to reach those individuals to ensure they understand the test and know how to prepare for it.”
So, among the MCAT Validity Committee’s next steps is studying why less-resourced test-takers aren’t using free prep materials more. To achieve this and other goals, the group has begun qualitative and quantitative studies exploring how test-takers prepare to enable the AAMC to develop new materials and expand outreach accordingly.
Of course, committee members will also continue to monitor how well MCAT scores predict success throughout medical school. “What we have so far are intriguing and reassuring findings,” says Hanson, “but there is much more work to do.”
“We need to understand MCAT scores — and we need to understand them in a broader context — so we can build a physician workforce that takes good care of our patients,” he adds. “That’s the goal. All of this is about taking good care of patients.”