Rating the Rankings: Medical Education Weighs in on the U.S. News Guide to Grad Schools

By Suria Santana

Some might argue that rankings have become just another fact of life. Nowadays, they can be seen in countless forms, virtually everywhere one looks: college and professional sports rankings, America's most "livable cities" rankings, bestseller books rankings, music "billboard" rankings, to name just a few examples. The public has become accustomed to the simplicity and apparent practicality of this kind of nicely packaged, easily digestible information.

But for some, the problem starts when the attempt to rank or define an institution, a city, or an artwork ends up unfairly depicting a set of very complex facts that cannot be easily quantified and compared.

This is an opinion held by some members of the academic medicine community when it comes to the U.S. News & World Report (USN&WR) annual ranking of this country's medical schools. Representatives from a number of institutions believe that these rankings unfairly portray a community made up of distinct institutions that have their particular set of strengths and unique qualities.

A 2001 Academic Medicine article critiquing the USN&WR rankings argued that they fall short on both methodological and conceptual grounds. The authors concluded that the USN&WR rankings are "ill conceived," "unscientific," and "conducted poorly." In addition, they "ignore medical school accreditation; judge medical school quality from a narrow, elitist perspective; and do not consider social and professional outcomes in program quality calculations."

Since 1987, medical schools have been ranked yearly by USN&WR. The magazine also ranks other graduate school programs such as law, business, and liberal arts schools, as well as colleges and hospitals.

Critics of the USN&WR medical school ranking system say that, unlike undergraduate programs, medical schools have very similar educational quality, mostly due to the strict accreditation requirements of the Liaison Committee on Medical Education (LCME). "Our medical schools are idiosyncratic in that they all have their own strengths and unique history," explains Robert Sabalis, Ph.D., vice president of the AAMC Section for Student Affairs and Programs. "But because the LCME has such strict scrutiny of them, it is not the same as looking at the full range of colleges in this country. Who really does it help to rank 126 institutions that are already in the top 2 percent of academic institutions?"

USN&WR conducts two different medical schools rankings, according to the magazine's director of media relations, Richard Folkers. "We rank the top schools in research and, separately, the ones in primary care. Because research and primary care training produce two different types of doctors, we look separately at the schools that have those different focuses. The methodology varies slightly between [how we conduct the surveys and rank the schools] in research and in primary care."

The magazine conducts surveys of medical school deans, senior faculty, admissions officials, and residency directors, asking them to rate their schools and the quality of other institutions' programs on a scale of one to five, according to Folkers. This "quality assessment" - or reputation ranking - constitutes 40 percent of the final score.

The medical school representatives are then asked to rate the institutions in the areas of research and primary care separately. "Thirty percent of the score in research schools looks at research activity, and that is measured as the total amount of NIH research grants awarded to the medical school and its affiliated hospitals," Folkers explains. "In primary care, that same 30 percent looks at the percentage of outgoing M.D.'s entering primary care residencies in the fields of family practice, pediatrics, and internal medicine."

Other factors that go into the total score include how selective medical schools are when it comes to factors like an entrant's average MCAT score and GPA, as well as the proportion of applicants admitted and the ratio of full-time faculty to students.

'Apples and oranges'

In answering the question of whether USN&WR rankings fairly portray the quality and standings of this country's medical schools, Arthur Levine, M.D., senior vice chancellor for health sciences and dean of the University of Pittsburgh School of Medicine, opines: "That depends on who's selling apples and who's buying oranges."

According to Dr. Levine, the amount of funding an institution receives from the NIH is the only equitable way of ranking medical schools, since it is currently the only objective benchmark in a nationally competitive peer review context. "That puts an absolute bottom line under the research trait of a medical school," he says. "Then, one has to make the segue from a medical school's research trait to the notion that, without exception, an institution that is strong in research is going to be strong in recruiting its faculty, and a strong faculty will recruit a strong student body, and a strong faculty and student body will lead to strong education, and strong education and research means strong patient care."

One of the problems with the USN&WR rankings, in Dr. Levine's opinion, is its use of institutions' average MCAT scores and GPAs. These measures are heavily influenced by a school's mission - which might include serving the needs of a particular state or attracting minority students - and they may result in a student body that is somewhat less academically credentialed.

Answering the criticism regarding schools' decision to accept students with slightly lower MCAT and GPA averages and their subsequent "penalization" in the USN&WR ranking, Folkers argues that GPAs and MCAT scores are just two of many factors the magazine takes into consideration.

Relevant to applicants?

"Our mission is to help students make an extremely important and extremely expensive life decision," Folkers continues. "We are always trying very hard to remind ourselves that we are providing a service to prospective students and that this is not a horse race. Unfortunately, some of the horse race aspects get too emphasized. We try to remind ourselves and students that there are a lot of factors that go into finding the right school, and students should really read across the lines and look at everything that goes into that ranking in order to find out what is the best school for them. Looking at incoming GPAs and MCATs is a good way to find out how you stack up against the student body."

But do students find these rankings useful? Lauren Oshman, M.D., MPH, American Medical Student Association (AMSA) national president, thinks the rankings are too simplistic to be helpful, since several other factors play a role in influencing an applicant's choice of medical school. "Some of the most important ones for today's medical students are tuition, because medical students' debt average more than $100,000 at graduation," Dr. Oshman says. "Other issues include the curriculum at a medical school, diversity in the student body, support for medical students, and the overall quality of medical education. The rankings don't delve into those critical issues to any depth."

In addition, "for the average pre-medical student, I don't think it's very helpful to know that Harvard is a very high quality medical school," Dr. Oshman quips. "For many premedical students, applying to school is not a choice between Harvard, Yale, and Columbia. It's a choice between your state school and some very expensive private schools that are perhaps less well known. So these rankings don't help them make the difficult decision to learn about every medical school in the country."

Another set of information that AAMC's Dr. Sabalis does not think is beneficial for medical school applicants is the component of the U.S. News ranking system that relates to research funding. "Most first- to fourth-year medical students aren't that interested in knowing how much research is going on at a medical school, because they are not going to participate in it by and large, unless they are Ph.D. students as well," he says. "One could make the point that excellent researchers are also excellent teachers, and some people do, but the number of research dollars that an institution receives has no direct relationship to the quality of the four-year undergraduate medical education program."

Dr. Oshman thinks the reputation - or "quality assessment" - rankings are particularly problematic. "Knowing the reputation of a medical school as seen by faculty members and deans is not the same as knowing whether medical students are satisfied with their education. AMSA tries to provide that information from [the perspective of] medical students to pre- medical students across the country, so applicants can get an insider look at the medical school versus the opinion of someone who is in the administration. There's often a disconnect in what they say," she observes.

Through AMSA's network of medical schools' chapters, premedical students are able to talk to medical students and discuss the pros and cons of certain institutions.

Trina Denton, a medical school applicant, says that the rankings were a little helpful but not particularly influential in her decision. "Personally, I looked at the rankings a couple of times, but there were other things that influenced my application to medical school," says Denton. "These things vary by student, but I was personally looking at location and the programs the school provided."

She thinks that the rankings' influence may vary by student, however.

Why rank?

"We started doing this service because nobody else was," Folkers says. "We came to feel that this is a hugely expensive and important life decision, whether we're talking about four years of college, or graduate school and beyond. There's simply a vacuum of impartial information."

Another service to be provided by USN&WR this year is the equivalent of a medical school guidebook. "This will be a comprehensive guide to every school with key information that prospective students might need," Folkers says.

The AAMC produces a yearly guide to medical school, the Medical School Admission Requirements (MSAR).

Despite the barrage of criticism the magazine receives on its rankings on both methodological and conceptual grounds, Folkers thinks there will always be demand for the kind of information his publication provides. "I think that most people, when they look at it in balance, do think that we provide a good service," says Folkers.

Antonio Gotto, M.D., Ph.D., D.Phil., provost for medical affairs and dean of Cornell University Weill Medical College, says that even though the information in the ranking is not statistically significant, there likely will continue to be demand for it. "We can get along fine without the rankings. But the American public likes rankings, and this is something USN&WR knows, so I'm sure they will continue."

Contact Us    © 1995-2008 AAMC    Terms and Conditions    Privacy Statement