When the Centers for Medicare and Medicaid Services (CMS) released the Overall Star Ratings on its Hospital Compare website in late July, many in the academic medicine community objected, asserting that the ratings did not provide an accurate view of quality at major teaching hospitals and could potentially mislead patients. At this time, CMS plans to update the star ratings every four months. The agency will include new measures and remove retired measures from the methodology on a continual basis. New ratings are expected in mid-October.
CMS used as few as nine and a maximum of 64 quality measures—including readmission rates, safety, mortality, and patient experience—in assigning a one- to five-star rating to more than 4,000 hospitals nationwide. Critics have said the ratings raise red flags in part because they are not applied equally, with some hospitals reporting a handful of metrics and others reporting on more than 60 metrics. For example, two hospitals that report the same measures could have vastly different ratings based on the institution’s size or location. And CMS rejected recommendations from the academic medicine and hospital communities that metrics be risk adjusted for differences in sociodemographic status.
Originally scheduled for distribution in April, the ratings were delayed by three months after Congress and other stakeholders, including the AAMC, called on CMS to address flaws in the methodology before the release. Critics of the ratings have pointed out that teaching hospitals treat patients with complex conditions who often cannot be treated at community hospitals. In addition, many smaller hospitals were unable to provide enough data to report on all the metrics, putting teaching hospitals that reported in more categories at a disadvantage.
“For those hospitals that are sitting in an environment or community where they have a significant number of disadvantaged patients—poor, living alone, having a disproportionate amount of mental illness—they can’t control readmission rates the same way as a hospital in a high socioeconomic area can.”
Elizabeth Mort, MD, MPH, Massachusetts General Hospital
“The reality is that hospitals cannot be rated like movies,” said Darrell G. Kirch, MD, AAMC president and CEO. “The AAMC and the nation’s major teaching hospitals strongly support providing patients with transparent and meaningful information. However, the new star ratings offer an overly simplistic picture about hospital quality that could end up driving patients away from the very hospitals where they could receive the best care.”
According to a CMS analysis, more than two-thirds of teaching hospitals received three or fewer stars, and only one teaching hospital received five stars. Smaller hospitals offer fewer services and often lack the resources for the advanced care required by patients with complex conditions or multiple illnesses. Based on AAMC data from February 2016, members of the AAMC Council of Teaching Hospitals represent 71 percent of all Level 1 trauma centers and 88 percent of all comprehensive cancer centers. These institutions also provide complex services that are not available elsewhere. The CMS methodology does not account for this complexity. And, an AAMC analysis found that hospitals reporting fewer quality metrics were more likely to receive higher star ratings and that nearly half of the five-star ratings went to hospitals that reported on 60 percent or fewer of the metrics.
“If you’re an 88-year-old Medicare patient who needs a knee replacement, but you have congestive heart failure or hypertension, … what you want as a patient is a hospital that will take good care of you in the event something goes south,” said Elizabeth Mort, MD, MPH, senior vice president for quality and safety and chief quality officer at Massachusetts General Hospital in Boston.
Oversimplified ratings discount sociodemographic status
Michelle Schreiber, MD, senior vice president and chief quality officer at Henry Ford Health System in Detroit described several concerns with the methodology. She noted, for example, that there are metrics in the ratings that are weighed equally yet measure very different aspects of care. She said that although patient satisfaction is “very important,” mortality is a more significant metric.
The ratings are also oversimplified, Schreiber said, noting that in some cases the difference between one-star and three-star hospitals is less than one percentage point. In addition, by not risk adjusting, the ratings did not account for patients who might not have family support, proper nutrition, or transportation to get to follow-up appointments. “Without those in place, the risk of ending up back in the hospital is far greater, leading to higher readmission rates,” she said.
In fact, a study published in JAMA Internal Medicine in 2015 confirmed that sociodemographic factors, such as low income or a lack of family support, were linked to higher readmission rates.
“For those hospitals that are sitting in an environment or community where they have a significant number of disadvantaged patients—poor, living alone, having a disproportionate amount of mental illness—they can’t control readmission rates the same way as a hospital in a high socioeconomic area can,” Mort said.
The methodology can be “potentially damaging” for patients and their families who might choose a hospital because of a higher star rating rather than on its clinical services, said Steve Lipstein, president and CEO of the BJC HealthCare and former AAMC chair. The 14 hospitals in the St. Louis–based health system, which includes teaching institutions and smaller, rural hospitals, received ratings from two to four stars. BJC hospitals receiving higher ratings were more likely to be in affluent areas, Lipstein reported.
“Instead of being helpful to patients and their families in selecting a hospital, it turns out that at the local level [the ratings] are very misleading,” he said.
Schreiber added that while she “questions the methodology CMS used,” she is glad the star ratings have “become a catalyst for such a robust discussion” on hospital quality. “We are committed to developing meaningful and useful measures to help assess quality.”
In making decisions about where to go for treatment, Lipstein said patients should look at local circumstances, talk with people they trust, and research which hospitals have the services appropriate for their needs. But to rely on ratings that were “never intended to compare one hospital with another is going to result in patients not getting meaningful information and will be potentially damaging,” he said.