AAMC Reporter: March 2006

MCAT Scores Predict Medical School Success

Medical College Admission Test (MCAT®) scores can be used to predict how well students will do during their clinical rotations in the third year of medical school, as well as during the basic science curricula of the first two years, according to a recent AAMC study. This result was somewhat unexpected to faculty and admissions personnel who formerly believed that the predictive validity of the MCAT was limited to the basic sciences.

In fact, MCAT scores were better predictors of the scores later achieved on the United States Medical Licensing Examination (USMLE) Step tests than were undergraduate grade-point averages (GPAs). Grades in medical school, however, were best predicted by taking both MCAT scores and undergraduate GPAs into consideration, according to the report.

"This does give support for continuing to use the MCAT for selecting who is going to do well in the basic science classes and beyond," said Ellen Julian, Ph.D., associate vice president of the AAMC Section for Applicant Assessment Services and author of the report "Validity of the Medical College Admission Test for Predicting Medical School Performance," published last fall in Academic Medicine. 

The study followed two cohorts of a total of 4,076 students from 14 different medical schools selected to represent diverse geographical regions, school sizes and types, and different curricula. Investigators collected data about the study participants from the time they started medical school, in 1992 and 1993, through their first year of residency. Academic success or lack of success was measured in three categories: grade-point averages for years one and two (which were analyzed separately, and then results were combined because they differed very little); grade-point averages for year-three clerkships; and academic difficulty and distinction. For the purposes of this study, academic difficulty included only those problems that could be directly attributed to academic performance, not to personal issues. Year-four data were not included because there was insufficient variation in grades.

"We are trying to help schools identify the applicants who will be successful," Julian explained. "Using just undergraduate GPAs and MCAT scores can reduce the uncertainty by half."

Writing Sample Predicts Clinical Competence

Although this particular study did not differentiate among the four MCAT sections—verbal reasoning, physical sciences, biological sciences, and writing sample—to determine which ones have the most predictive validity, there is other research evidence that high scores on the writing sample indicate a high level of future clinical competence. This finding was based on an analysis of data for 1,776 medical students attending Jefferson Medical College between 1992 and 1999. These data were collected as part of the Jefferson Longitudinal Study of Medical Education being conducted by the medical college, published in the October 2000 supplement to Academic Medicine.

Mohammadreza Hojat, Ph.D., is a research professor in the department of psychiatry and human behavior, and director of the Jefferson Longitudinal Study in the Center for Research in Medical Education and Health Care at Jefferson Medical College, Thomas Jefferson University. Summarizing the study's results, Hojat said that the writing sample was a good predictor of performance in the third year of medical school but not of performance in the first two years of basic science.

"It seemed that the writing sample measured different aspects of ability than science aptitude," Hojat said. "It measured ability that is mostly related to clinical performance, not to performance on objective examinations of medical knowledge."

This result may be important to medical school admissions committees depending on their schools' unique criteria, Hojat said.

"If the criterion measure is performance in the first two years of medical school, then the science sections of the MCAT should be given significant consideration," Hojat said. "If the criterion measure is later clinical performance, then the verbal skill and writing sample should be given as much attention as the other sub-scales."

Successful performance on a test, of course, is much easier to measure than successful performance as a physician, says Lewis H. Nelson, III, M.D., associate dean for Student Services and Admissions at Wake Forest University School of Medicine. Qualities such as empathy, common sense, dedication, focus, and the ability to work as a team are important components of performance both during medical school and later as a practicing physician, according to Nelson.

"The Medical College Admission Test is a component of evaluating applicants for medical school," Nelson said. "I do not see that it is the sole component."

Unfortunately, personal qualities are not easily measured items on standardized tests. During the face-to-face interview, admissions committees may be able to determine whether a student has some of these attributes. The problem is that a medical school may not be able to interview the thousands of people who apply to the institution. At Wake Forest University School of Medicine, for example, only 500 applicants out of an expected 6,600 will be invited to come in for an interview. Out of that group, 108 will be admitted.

"All medical schools have this tyranny of numbers," Nelson said.

The verbal reasoning and writing sample scores can be helpful tools for assessing which applicants may have particular strengths in these important non-science areas. The AAMC, said Julian, is currently researching whether it is possible to build measures of these personal qualities that can be administered in a standardized manner to large numbers of applicants, without being unduly susceptible to coaching.

Using MCAT's Predictors

While MCAT scores clearly were predictive of academic success in the AAMC study, some schools used the MCAT's predictive validity to identify those students who could benefit from extra help rather than those students who were already at the high end of the performance curve. Julian explained that a rural state school that was represented in the study used students' MCAT scores to identify those students who would need extra help to succeed to succeed in medical school.

"The school worked a lot with those students even before they enrolled and was able to fill the holes in their academic preparation so that they performed as well as people who came in with much higher MCAT scores," Julian said. "Ironically, this makes it appear as if the MCAT has no predictive power at such schools, but they are actually finding it very useful."

The fact that MCAT scores had different levels of predictive validity among medical schools reflects the unique missions of these schools, Julian said.

Julian also noted that this research showed that some schools focus on the success of their best-prepared students, while others achieved a high degree of success in nurturing their students who started with lower MCAT scores.

The results of the study clearly show the predictive validity of MCAT scores. Even these strong results leave half of the variation in performance unexplained. "I think it will give admissions committees added confidence in what they're already doing, which is using MCAT scores in conjunction with other variables that address that half of the variance that the MCAT doesn't predict," said Julian. She added that the findings may ease the lingering concern that test scores have little bearing on identifying students who have the potential to succeed in the clinical years.

In another related study, Hojat and colleagues looked at the predictive validity of the MCAT for students who have taken the exam more than once. In these situations, they found the highest predictive validity when they averaged scores from all the exams, instead of using just the highest or the lowest score. "The average is the best predictor of performance for those who repeat the MCAT," Hojat explained.  

One area for future study that has not yet received much attention, according to Hojat, is whether the predictive validity of MCAT scores may vary among different undergraduate institutions. In a much earlier study published in the Journal of Medical Education in 1987, Hojat and colleagues examined data for almost 3,000 students entering Jefferson Medical College from 10 diverse undergraduate institutions. What they found were significant variations between undergraduate schools in the predictive validity of MCAT scores.

Together, these studies indicate that the MCAT is helpful in a variety of ways and can be used differently by medical schools with diverse missions.

—By Anne Blank, Special to the Reporter

Contact Us    © 1995-2009 AAMC    Terms and Conditions    Privacy Statement