AAMC Reporter: January 2006

MSPE: Will It Overcome Challenges?

By Scott Harris

Three years after it began, a nationwide effort by the AAMC to improve the evaluation of applicants for medical residencies faces persistent challenges involving communication and timing among medical schools and teaching hospitals.

The situation has raised questions about how schools and hospitals are responding to the latest AAMC guidelines for preparing the Medical School Performance Evaluation (MSPE), a document first distributed in 2002 to replace the traditional "dean's letter" of evaluation with a more analytical tool.

Residency-program directors had sought an alternative to the decades-old use of the dean's letter because they wanted greater objectivity and standardization in the evaluation process. They said the letter often seemed to provide little more than personal recommendations or predictions of students' future performance. Instead, the directors sought an approach that would offer greater consistency, objectivity, and standardization in evaluations across medical schools. The goal also was to encourage more collaboration among schools and hospitals engaged in various aspects of the application process.

But the resulting MSPE, which medical schools have been sending to residency-program directors on November 1, has run into some complications. Despite continuing interest in the document, a recent AAMC survey of about 1,000 hospital residency programs found that about one-third of 237 respondents were not reading all their candidates' MSPEs in their entirety. Many directors said they read only a summary section, which includes an assessment of applicants' "comparative performance" in relation to their medical school peers.

'Unique Characteristics'

Other sections of the AAMC's six-part guide to preparing the MSPE provide information about applicants' "unique characteristics," such as leadership and research abilities; academic history, including whether students have been required to take remedial work or have been the subject of "adverse action(s)" by their institutions; and "academic performance and professional attributes" in preclinical/basic science coursework, as well as in core clinical and elective rotations. The AAMC guide also includes five appendices showing comparative performance data for the students in graphic form.

AAMC survey data indicate that only about half of all medical school deans believe that the MSPE format has resulted in "more straightforward reporting of 'negative' information about students." Yet experts on the residency application process continue to see great merit in the MSPE.

Robert Sabalis, associate vice president for student programs in the AAMC's Division of Medical School Affairs, said the MSPE presented opportunities for more objective assessments than did the dean's letter. Student-affairs officials who wrote the letters were more apt to minimize students' imperfections, he said, and thus would give residency-program directors less than full and accurate portraits of applicants. Sabalis said the MSPE had already resulted in "some very positive" changes.

"It gets the student-affairs officials thinking as representatives of society and of the medical profession, and not just as student advocates," Sabalis said. "So the document is a more honest representation."

"The MSPE has helped to overcome the prior resistance in medical schools to evaluating professionalism, and student-affairs deans now agree that evaluating professionalism is not only possible but also necessary," Sabalis said. "So today, if you have grades of all A's but you consistently mishandle patients as a medical student, you may be required to complete some remedial activity. This will be reported and, depending on the situation, the student's graduation or competitiveness as a residency applicant may be brought into question."

Leaders in medical education agree that issues relating to communication and timing, as well as a relatively high turnover rate among residency directors, may help explain why the MSPE has not been more uniformly successful.

"The concern is that residency-program directors only read the [MSPE's] bottom line," said Gail Morrison, M.D., vice dean for education at the University of Pennsylvania School of Medicine and head of the AAMC's MSPE advisory committee. "If they're only doing that, the feeling is, why are we doing this?"

'Sticking Point'

Sabalis said a sticking point between student-affairs deans and residency directors has been the MSPE's November 1 distribution date. Residency directors want to receive the document earlier so they can begin interviewing candidates, Sabalis said. But student-affairs officials say they cannot fully or accurately evaluate medical students for residency programs until the students have completed all core clinical training, which is usually in early fall.

"We're limited by the academic calendar, which differs regionally," Sabalis said.

Lanny Garth Close, M.D., Columbia University College of Physicians and Surgeons

Lanny Garth Close, M.D., director of the residency program of the department of otolaryngology/head and neck surgery at Columbia University College of Physicians and Surgeons, called the MSPE a useful resource, but he expressed concerns about timing.

"I'm more than willing to look at the MSPE," said Close, who is also a professor of otolaryngology. "I like the format. I think the student-affairs deans have worked hard to resolve a lot of the issues that were in the dean's letter, where it sometimes seemed as if every student was going to win the Nobel Prize."

But, Close continued: "All our major decisions regarding residency candidates have to be made prior to the MSPE coming out. The early bird gets the worm. This year we interviewed 42 candidates for three spots. Plus, in New York and many other places, the weather gets bad later in the year. So we're always going to have to do our interviews early. I understand [the student-affairs officials'] point of view in terms of waiting for the grades, but it's just not the reality. Waiting is just not going to happen."

Close added that he sometimes uses the MSPE retroactively as a "tiebreaker" between two otherwise equally qualified candidates.

Student affairs experts have said that educating their medical school colleagues and residency-program directors — particularly those who are new to their positions — about the MSPE could go a long way toward making them comfortable with the MSPE format, thus increasing their use of the document.

Peter J. Katsufrakis, M.D., M.B.A., Keck School of Medicine, University of Southern California

"There's a lot of turnover in personnel, especially with residency-program directors," said Peter J. Katsufrakis, M.D., M.B.A., associate dean for student affairs at the Keck School of Medicine of the University of Southern California and the immediate past national chair of AAMC's Group on Student Affairs (GSA). "A well-intentioned program director may... undermine the MSPE process without even knowing it."

Morrison, who, in addition to being vice dean of Penn's medical school, is its academic programs director and a professor of medicine, said the MSPE process could benefit from "a little more marketing and a little more communication."

The survey's results showed that "you can't just assume the word will get out," she said.

"It's one thing to change it, but if the people receiving the [MSPE] aren't quite sure how to use it or even what it is, you've got another kind of problem," Morrison said.

Morrison said "marketing" of the MSPE has been more thorough and more successful among student-affairs officials than among residency-program directors. But she suggested that a greater effort was needed to encourage residency directors to use the MSPE more effectively — or, in some cases, to get them to use it at all.

"We want to make sure there is uniformity in how medical school officials get these forms out to residencyprogram directors, so that two different students from two different schools are evaluated equally," Morrison said. "We can't mandate that medical schools use this. We can only make recommendations. People take liberties — how much, we're not sure."

The MSPE advisory committee is scheduled to meet next in February to discuss future research and the direction of the MSPE. The document also is expected to be discussed in April at a national meeting of the AAMC's Group on Student Affairs.

"This really is a situation where a 'win-win' is possible," Katsufrakis said. "A good student who ends up in a bad fit at a residency program due to imperfect information exchange is an undesirable thing. What has to happen is that there is honesty in what's being reported and faith in the process."

"There's a dynamic tension around this issue," Katsufrakis said. "But it's really just a question of trust and good faith between all the parties involved."

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement