What happens when patients find out how good their doctors really are? (2004)

http://newyorker.com/archive/2004/12/06/041206fa_fact?currentPage=all

by danso • 12 years ago

184 53 12 years ago

conorh• 12 years ago

I hope that more and more data becomes available around healthcare and outcomes!

Medicare recently produced this data http://www.nytimes.com/interactive/2014/04/09/health/medicar... and the center that my wife works at used it to to produce this graph http://blog.parathyroid.com/parathyroid-surgery-medicare/. They are quite a bit better at this particular operation than anywhere else in the US, not surprising given how many more they do, however it is often difficult for patients to know or understand that - who should they trust? More data please! Interestingly this operation is one that Atul Gawande (who wrote this article in 2004) specializes in too.

angersock• 12 years ago

If you're interested in referral patterns:

http://omni.docgraph.org/

arn• 12 years ago

Great article. Might be hard to get through, so I'll put out this quote which I enjoyed and compelled me to read the rest of it.

"Matthews had started a cystic-fibrosis treatment program as a young pulmonary specialist at Babies and Children’s Hospital, in Cleveland, in 1957, and within a few years was claiming to have an annual mortality rate that was less than two per cent. To anyone treating CF at the time, it was a preposterous assertion. National mortality rates for the disease were estimated to be higher than twenty per cent a year, and the average patient died by the age of three. Yet here was Matthews saying that he and his colleagues could stop the disease from doing serious harm for years. “How long [our patients] will live remains to be seen, but I expect most of them to come to my funeral,” he told one conference of physicians.

In 1964, the Cystic Fibrosis Foundation gave a University of Minnesota pediatrician named Warren Warwick a budget of ten thousand dollars to collect reports on every patient treated at the thirty-one CF centers in the United States that year—data that would test Matthews’s claim. Several months later, he had the results: the median estimated age at death for patients in Matthews’s center was twenty-one years, seven times the age of patients treated elsewhere. He had not had a single death among patients younger than six in at least five years."

yesiamyourdad• 12 years ago

What grabbed me:

"In this short speech was the core of Warwick’s world view. He believed that excellence came from seeing, on a daily basis, the difference between being 99.5-per-cent successful and being 99.95-per-cent successful."

hsitz• 12 years ago

I took notice of that passage also. But the main reason the 99.5 to 99.95 percentages is meaningful is that they were measures of a daily risk. So it quickly compounds into a meaningful percentages; the 99.95% rate meant an 83% chance patient would stay well this year, the 99.5% rate meant only a 16% chance of staying well for this year.

Many of the medical percentages that we see quoted in media are derived using a lifetime risk basis, _not_ a daily one. On a lifetime basis the difference between 99.95% and 99.5% _is_ negligible.

I wonder how many people reading this article are aware of the difference.

bdcravens3• 12 years ago

I have Cystic Fibrosis. I don't think it's fair to assume that the bell curve associated with treatment results is fair to attribute to the quality of care. Factors such as climate and pollution can significantly contribute to health, as can socio-economic and average distance patients travel, as well as access to insurance.

For instance, I'm from North Texas; I went to college in Oklahoma. Back then (1996) Texas had a type of insurance for children with CF (it actually extended to age 21) unavailable in Oklahoma. I ended up losing care, going off of my meds, and by the time I was back in Texas and had the income to support my needs (I came from a very poor family) I had lost 30 pounds (down to 98lbs) and had lost 25% of my lung capacity. Was this attributable to the clinic in Dallas? Of course not.

(FWIW, I'm up to 155lbs these days and my lung capacity is probably in the upper 5-10% of all CF patients)

rcthompson2• 12 years ago

Of course there are other effects that go into that bell curve (I'd be more worried about doctor-to-patient ratios and time per patient), but I think the article does an excellent job of highlighting the differences in care.

bdcravens3• 12 years ago

Very true. It's important to note that the majority of a (well maintained) CF patient's health is the result of the the breathing treatments and enzyme treatments; I could probably not see a doctor for years if I had my meds. (Probably true of many chronic conditions I'm sure, but speaking to my experience only)

rcthompson2• 12 years ago

What stuck out for me was the difference in attitude between the two example centers. It's "67% is still pretty good, so keep it up" vs "90% is a big decrease from 105%; this is a failure and we need to make adjustments." In articular, the former type of attitude is certainly the norm from my experience (with medical care in general, I have no experience with CF).

timbre• 12 years ago

This idea applies to more than medicine. A boss once told me, speaking about a piece of work I thought was pretty good, "If it could be better, it's wrong." It stuck with me, and I've found that everyone I meet who is exceptional at some skill has this attitude.

DannoHung• 12 years ago

Yeah, that's sorta stuck with me. I know a lot of people who say that the perfect is the enemy of the good.

Well, buddy, that just means that if you have a good solution, it's stopping you from getting a perfect solution.

pbhjpbhj• 12 years ago

The counter-idiom is "Excellence is good enough.". I have used it as a mantra to counter the paralysing need not to act on things unless perfection can be achieved.

grifpete• 12 years ago

Q.What do you call the guy who passes out last in his class on his med finals? A. Doctor

jtheory• 12 years ago

Q. What do you call the guy who scored just a teensy bit lower than that?

A. Mister.

IOW, this joke/insight is only valuable if the standard for passing is set too low.

If it's set correctly, then everyone who passed has shown they are prepared to be a competent doctor.

shabble• 12 years ago

Of course, in the UK/Commonwealth you might offend a surgeon if you call him "Doctor"[1]

[1] http://www.bmj.com/content/321/7276/1589

herdrick• 12 years ago

If competence is, as the article suggests, normally distributed, then there's no natural level at which to draw the cutoff line. But anyway, iirc, studies show no correlation between outcomes and class rank (nor selectiveness of medical school), at least for surgeons.

trustfundbaby• 12 years ago

Ostensibly an article about medicine but really more about how to achieve true excellence at what you do ... definitely a long read but well worth the time

jsudhams2• 12 years ago

The same question came to my mind and I asked a doctor "Sir, what happens to the students in MBBS who just pass taking a border mark?" The doctor said you should still be ok because passing score itself is like 60% + lot of practice involved. I did not bother to check the pass mark required for the course but now-a-days I go to doctor who have at least 10+ years of experience.

lmm• 12 years ago

now-a-days I go to doctor who have at least 10+ years of experience.

That's not very Kantian. How are less-experienced doctors supposed to get experience?

jsudhams2• 12 years ago

With some else :) Keeping the joke aside. I would want them to assist senior doctor for some time and personally i always use second opinion. BTW, if you do not you can get anti-biotic off the shelf and most of the doctors will give you in first visit.

altrus2• 12 years ago

Significantly, this article neglects to consider how sample size (and demographics) might affect the measurement of hospital treatment quality.

tl;dr: If you don't adjust for the sample size, what may appear to be the best, or worse, hospital, may look like that only because of how they have (un)successfully treated a single patient, and thereby yield an unrealistic estimate of patient quality (ie; 100% cure rate of a single patient, which may not be reflective of the actual caliber of doctors).

Further reading, which is strongly recommended if you want to learn more;

http://nsmn1.uh.edu/dgraur/niv/TheMostDangerousEquation.pdf‎

danso• 12 years ago

Sorry...but I don't get what you're postulating...that the doctors and hospitals whose reputations are at stake...nevermind the lives of their patients who are affected by policy decisions...would be totally ignorant of that basic statistical caveat? That it's possible that the top-of-the-line hospital just happened to cure a couple of really lucky patients and everyone who has lauded that hospital has never heard of regression analysis?

I guess it's possible...but it's probably more likely that the New Yorker is not trying to be a reference on statistical methods. In any case, the statistical caveat you mention is arguably addressed in this catchall-paragraph that briefly describes the problem of quality-of-care statistics:

In recent years, there have been numerous efforts to measure how various hospitals and doctors perform. No one has found the task easy. One difficulty has been figuring out what to measure. For six years, from 1986 to 1992, the federal government released an annual report that came to be known as the Death List, which ranked all the hospitals in the country by their death rate for elderly and disabled patients on Medicare. The spread was alarmingly wide, and the Death List made headlines the first year it came out. But the rankings proved to be almost useless. Death among the elderly or disabled mostly has to do with how old or sick they are to begin with, and the statisticians could never quite work out how to apportion blame between nature and doctors. Volatility in the numbers was one sign of the trouble. Hospitals’ rankings varied widely from one year to the next based on a handful of random deaths. It was unclear what kind of changes would improve their performance (other than sending their sickest patients to other hospitals). Pretty soon the public simply ignored the rankings.

Even with younger patients, death rates are a poor metric for how doctors do. After all, very few young patients die, and when they do it’s rarely a surprise; most already have metastatic cancer or horrendous injuries or the like. What one really wants to know is how we perform in typical circumstances. After I’ve done an appendectomy, how long does it take for my patients to fully recover? After I’ve taken out a thyroid cancer, how often do my patients have serious avoidable complications? How do my results compare with those of other surgeons?

(the author himself is a surgeon and has written a lot about the problems of reliably measuring quality of care performance)

altrus2• 12 years ago

To be clear, I'm not at all suggesting that doctors do this out of malice - only that effective statistical analysis is really, really, hard, and it's not immediately clear that surgeons (regardless of their international reputation) would have the background required to properly analyze these studies in a coherent framework, especially when it's not immediately relevant to a patients outcome in an operating theatre.

I'm not suggesting that they're ignorant of basic statistical facts, but I am definitely suggesting that they're not immediately aware of the subtle assumptions implicit in many of the statistical models they use.

For example, given the large numbers we're talking about, not only is it possible that the "number 1" hospital in any particular field is there because of statistical fluke, it's actually likely that this is the case.

The (mis)use of statistics certainly isn't limited to medicine, but it is one of the places where its misinterpretation has the biggest impact.

https://www.sciencenews.org/article/odds-are-its-wrong

aidenn0• 12 years ago

Doesn't the central limit theorum imply that most sane distributions of quality of care (including all doctors are equally good) will result in a bell curve?

There are tests you can do to tell if the variance between care centers exceed that you would expect from chance, but the mere observation of a bell curve seems completely uninteresting.

hessenwolf• 12 years ago

Say each doctor's patients had a common constant death rate for that doctor, not a crazy assumption, and those constant death rates across doctors followed a gamma distribution (not hard, because I could fit my ass with a gamma), then the distribution of each doctor's patients' life expectancies would be exponential and the distribution of life expectancies of the whole group would be Pareto.

Exponential is completely positively skewed and thin-tailed, and Pareto is roughly the same shape but very fat-tailed.

droopyEyelids• 12 years ago

Good article, but I feel Atul Gawande mischaracterized "evidence-based practice" in a way that does our whole planet a disservice when he poo-poohed it to lionize Warwick.

Evidence based practice isn't really new, it's been a thing since '92. And while it is about using studies to make treatment decisions, it's not about forbidding doctors from thinking for themselves.

But what really got my goat was the fact that evidence-based medicine is the new thing, and it's an essential step in the evolution of medicine, and more people need to understand it. Evidence based medicine is about applying statistics to determine when treatments cause more harm than good. With modern medicine's advances in imaging techniques, we're hitting all sorts of new and dangerous problems where the benign abnormalities we all accumulate over our lives are treated at great expense, inconvenience and loss of health.

ars• 12 years ago

I don't think he was "poo-poohing" it exactly. If everyone only follows evidence based medicine there will never be any improvement because there will never be any changes.

So someone has to be the person who ignores it, who tries new things. And that someone, in this field, is Warwick.

The evidence part comes in when that person also tracks their results! i.e. makes it a study, and Warsick certainly seems to do so.

6cxs2hd62• 12 years ago

Yes I got the impression his attitude about evidence-based medicine is similar to that saying about accounting -- that "accounting is the art of looking in the rear view mirror and dragging your ass into the future".

i.e. Accounting isn't worthless. It's necessary, but not sufficient.

PaulJulius• 12 years ago

Patients with CF at Fairview got the same things that patients everywhere did—some nebulized treatments to loosen secretions and unclog passageways (a kind of mist tent in a mouth pipe), antibiotics, and a good thumping on their chests every day. Yet, somehow, everything he did was different.

I found this quote especially interesting. I've often found this to be true, that you look over at someone who's much better, and on one hand they're just doing the same things you're doing, but somehow they're just on a different level. It's very difficult to distinguish exactly how someone achieves better results, that is, until it's been explained. Here, we see this clearly in how the level of lung function is perceived. Both centers provide the same treatment, but at the Cincinnati center lung levels around 70% are "okay", but at Minnesota, any drop is unacceptable, even if the patient is already at above-normal levels.

It sort of reminds me of the "Don't deal with it, fix it" article that was also posted recently. You'll be coasting around at a certain stable level of engagement until one day you see something you've never considered before and all of a sudden you see a whole new world of possibilities.

graeham2• 12 years ago

Medicine is like any other profession - it shouldn't be surprising that some doctors are better than others, like some lawyers/engineers/teachers are better than others.

There are a lot of challenges with healthcare metrics. Medical problems are more complex, arising from more factors than practically any other problem. Most other sciences can be resolved by theory or relatively convergent empirical data. Medicine is empirical, but the scatter is huge.

Data is not tracked well. When it does exist, it is in formats that are not well conducive to analysis and sharing. This seems to be slowly (and expensively) improving. Pressure for privacy adds to inertia for innovations that could use medical data for improved outcomes.

With the title of the article, I was hoping there would be some more discussion about implications of some 'rating system' for doctors. It would probably superficially be a good idea, but such a system would obviously result in some level of gaming. Does a doctor in a rated system only take easy cases to keep a good track record? Are hard cases given more weighting? Who/ how is such a weighting decided? Such a system could offer a results-based compensation scheme for doctors, which is lacking from most publicly funded systems.

derekp7• 12 years ago

What about the equiv. of code reviews? For surgeons, take videos of each surgery, and have them rated by an impartial review panel. I can't find the link now, but I recently saw a segment posted on Youtube that compared a surgeon with good technique, to one with poor technique, and it correlated with the overall outcome of the patients.

FLUX-YOU• 12 years ago

I remember this, but I also can't find it. Part of it involved maintaining proper view of the field through laparoscopes, minimizing wasted movement, and avoiding unnecessary tissue damage. That last one has the most direct effect on outcomes, while the former two can lead to the third.

The lower ranked surgeons would have the scope jostling around and the laparoscopic instruments would contact nearby tissues, while the higher ranked surgeons just seemed to have more elegance in his skill; a laser-precision focus to borrow from this thread and an intention and purpose for every movement.

I had a similar impression watching that video on surgery technique to watching high level Starcraft player replays versus some afternoon ladder replays by mid-level players. When placed side-by-side, the differences were fairly obvious even if you've never done the act yourself.

yodsanklai2• 12 years ago

"Medicine is like any other profession - it shouldn't be surprising that some doctors are better than others"

Surely, some doctors are better than others, but (for the sake of discussion :)) I don't think every professions or activities are equivalent with respect to 'rating'. Take the extreme example of walking. All human beings walk equally well. This is unlike playing music or solving math problems for instance.

I'm certainly not saying that medicine is as easy as walking, but maybe for a wide range of health problems, there are well known standard procedures or solutions that all doctors are competent enough to follow. And in doubt, they usually refer their patient to a specialist, who in turn can send them to a more competent doctor.

robotresearcher2• 12 years ago

"All human beings walk equally well."

This is obviously not true.

CapitalistCartr• 12 years ago

You have nothing at stake with the walking of others. I have friends who are massage therapists. They notice the walking quality of others and what it means closely. When I see a doctor I personally have a lot at stake, which is why I do my homework.

Domenic_S• 12 years ago

What you call the person who graduated dead last in med school?

"Doctor".

SeanDav• 12 years ago

This reminds me of a true story I once read about the best car salesman in the country at the time. Single handedly he was selling more than some not-so-small entire dealerships. The difference was his incredible focus and approach to even the tiniest detail that could help a sale and willingness to try new things.

One thing that stands out in my memory about this was the fact the he made sure he always had 3 (!) pens for the customer to sign the sales agreements. Just in case 1 pen stopped working and the backup pen malfunctioned. He brought this sort of laser focus to absolutely every aspect of the business.

How many salesmen would have a spare pen, already in place, let alone 2 spares. How many would even have thought about it and what the right number might be?

Focus, attention to detail and continuous innovation.

Terr_• 12 years ago

Unfortunately, there's also a cargo-cult aspect to this kind of thing, when managers start saying: "Everyone is now required to have three pens at their station at all times."

I'm in a company where the suit-wearing side says "we need innovation" to the tech side, but they don't actually want a discussion about what they see as opportunities or directions. They just want to invoke it by rote ritual, or order it as if we kept some on shelves.

Terretta• 12 years ago

Money quote, applicable to hacking in general:

“We are used to thinking that a doctor’s ability depends mainly on science and skill. The lesson from Minneapolis is that these may be the easiest parts of care. Even doctors with great knowledge and technical skill can have mediocre results; more nebulous factors like aggressiveness and consistency and ingenuity can matter enormously... What the best may have, above all, is a capacity to learn and adapt—and to do so faster than everyone else.”

rsanek• 12 years ago

Really good takeaway near the end of the article: "What the best may have, above all, is a capacity to learn and adapt—and to do so faster than everyone else."

What happens when patients find out how good their doctors really are? (2004)

Related Stories