Related topics

What Exactly Makes a Hospital the ‘Best’?

July 18, 1995

``Make sure your fax machine is turned on and full of paper when you leave your office Wednesday night,″ read an advisory to more than 125 major medical centers last week.

Early Thursday, fax machines around the country whirred into action as U.S. News and World Report notified the winners of its annual rankings of the best hospitals. That same morning, about 100 reporters and editors received bulging press packets touting the winners.

U.S. News’s six-year-old consumer guide to hospitals has become the hottest marketing tool in the health-care industry, a public-relations gold mine for the winning medical centers. The magazine wins, too, because the ``Best Hospitals″ issue generally sells slightly better than the average U.S. News and leads to a booming business selling reprints to winning hospitals.

But the rankings _ which this year include 128 hospitals in 16 specialties _ are also coming under fire. Officials at two major hospitals, University Hospitals of Cleveland and Hospital of the University of Pennsylvania, say the list is little more than a popularity contest because the methodology is flawed.

``It’s a sexy thing to do,″ says David Shulkin, chief quality officer for the University of Pennsylvania Hospital System. ``But in 1995, the data needed to come up with a consumer’s guide to health care aren’t readily available.″

Sour grapes? Not exactly. While neither hospital made the magazine’s current ``Honor Roll″ of the nation’s top 18 hospitals, University Hospitals is ranked among the top 40 institutions _ the top 4 percent of hospitals evaluated _ in nine specialties, up from four last year. Pennsylvania is in the top 40 of 13 specialties rated this year.

Orry Jacobs, executive vice president at University Hospitals, acknowledges that his interest in the magazine’s methodology was spurred by the failure of his institution’s orthopedics service to make the list. But he adds, ``The more I’ve looked at it, the more it appears to me to be a sham.″

U.S. News calls the list the only objective nationwide ranking of hospitals but readily acknowledges its limitations. ``There is no commonly accepted yardstick for measuring the quality of hospitals,″ says Avery Comarow, assistant managing editor at the publication and editor of the rankings. ``There wasn’t when we got into this six years ago, and there isn’t now. We’ve moved to an approach that has gotten more and more credible.″

The rankings are based on three equally weighted elements: reputation among the nation’s doctors as determined by an annual randomized survey; death-rate data compiled by the federal government; and a group of nine measures, such as ratio of nurses to beds and availability of certain high-technology services, that the magazine says are related to quality. The ranking system is based on a model developed by the National Opinion Research Center at the University of Chicago.

The rankings have occasionally included some surprising anomalies:

In 1993, Massachusetts General Hospital, consistently one of the highest rated institutions overall, placed 16th in the country in otolaryngology, a specialty dealing with ailments of the ear, nose and throat. The hospital doesn’t offer such a service.

Memorial Sloan-Kettering Cancer Center, for three consecutive years the nation’s top cancer hospital, placed 34th for orthopedic surgery. The institution has a small, highly regarded orthopedic program for cancer patients, but except in unusual cases it doesn’t provide care to noncancer patients.

The Cleveland Clinic finished eighth in gynecology, up from 11th last year, even though it just resumed delivering babies two months ago after a 20-year hiatus. Moreover, though the clinic offers renowned gynecological cancer and in vitro fertilization services, it doesn’t have a neonatal intensive care unit, an important attribute of top gynecological centers according to the magazine’s own definition.

Critics also question the magazine’s reliance on mortality rates gathered by the Health Care Financing Administration as its only measure of what happens to patients in hospitals. The agency still collects the rates but stopped publishing them in 1992 because of concern about their reliability as a way to rank hospitals.

Mr. Jacobs and Dr. Shulkin also argue that several of the nine quality indicators, such as the ratio of medical procedures to beds, don’t necessarily prove quality of care. And they maintain that physician opinion surveys are likely to produce results that confirm already held perceptions and aren’t based on firsthand knowledge.

Mr. Comarow acknowledges that despite an elaborate screening process, more than one hospital has made the list in a specialty it didn’t offer. But he says that the rankings are intended to point readers to hospitals with the ``highest and best levels of care,″ so it isn’t surprising that centers make the lists without offering routine services in a specialty. ``The Cleveland Clinic isn’t there because of its baby-delivering capability,″ he says.

He also says that the magazine has refined its use of mortality data to judge hospital quality.

Meantime, he adds, ``Our target audience isn’t hospitals; it’s our readers,″ and the ratings shouldn’t be looked at as a horse race.

But hospitals crave the recognition that comes from a top ranking. ``It has helped our reputation nationally,″ says John D. Clough, chairman of health affairs at the Cleveland Clinic, a perennial ``best hospital″ winner that this year nosed out the Mayo Clinic as the leading heart center. ``We cite it every chance we get.″

Duke University Medical Center, Durham, N.C., which is regularly among the nation’s top five hospitals, recently began using the rankings in ads in the Southeast. ``From a scientific perspective, we take these ratings with a grain of salt,″ a spokeswoman says. ``But from a marketing perspective, let’s be frank, it’s nice to be recognized.″

Even the University of Pennsylvania and University Hospitals in Cleveland use the rankings in marketing campaigns. ``In Pennsylvania, New Jersey and Delaware, we’re by far No. 1,″ says William N. Kelley, chief executive of the University of Pennsylvania Health System. ``We use that in our advertising.″

Mr. Jacobs says University Hospitals features only its Rainbow Babies and Children’s Hospital in ads. Since reputation is the only factor used to rate pediatric services, he says, the hospital’s ranking reflects only its popularity among doctors, not the methodology he attacks.

Update hourly