We're so good at medical studies that most of them are wrong

http://arstechnica.com/science/news/2010/03/were-so-good-at-medical-studies-that-most-of-them-are-wrong.ars?utm_source=rss&utm_medium=rss&utm_campaign=rss

by alexandros • 16 years ago

46 45 16 years ago

arstechnica.com

nkassis• 16 years ago

Could it also be because Statistics is hard and that a lot of researchers don't have a good grasp of it?

marshallp3• 16 years ago

Especially these bio/medical types, rigorous thinking isn't a requirement in their qualification process.

applicative2• 16 years ago

Isn't statistics what you resort to when there isn't scope for rigorous thinking?

Eliezer• 16 years ago

There are plenty of methods (cough report Bayesian likelihood ratios cough and publish your raw data cough) that are simple enough for even average scientists to use. They would rather use much more complicated statistics, though, because then they get to publish more papers with "significant" results.

If you can handle calculus, which most scientists take, then you can handle likelihood ratios, believe me.

But the incentives are terrible, which is quite a different thing from supposing that the average PhD is too dumb to learn good statistics if the incentives were strong.

maurycy2• 16 years ago

Does it apply to other sciences? (I think it does but I don't have any access to papers confirming it)

pw0ncakes2• 16 years ago

Most likely, not nearly as much. In medicine, tests are expensive and ethically complicated, because you're experimenting on living beings whose inner workings we barely understand. So the ability to make firm conclusions, like you can do in the sciences, is not there.

Gravity is proved. Evolution is rock solid. We still don't know if coffee is good or bad for us.

maurycy2• 16 years ago

I don't mean physics. I mean human sciences with statistics heavily applied (eg, economy or sociology).

a-priori3• 16 years ago

Gravity is proved. Evolution is rock solid. We still don't know if coffee is good or bad for us.

Gravity is not proven. Falling is easily observed, and general relativity is a pretty good theory for it, but the jury is still out on how exactly gravity works. Evolution is also easily observed (e.g., drug resistant pathogens); its basis in natural selection and genetics is the part that is rock solid.

As for coffee, it's some combination of good and bad. Which part dominates in which situations is up for debate.

(Apologies for being pedantic -- caffeine high just set in.)

roundsquare2• 16 years ago

Gravity is proved. Evolution is rock solid.

Sure, thats old physics/biology. You can't compare that to state of the art medicine. You need to look into modern cosmology, string theory, quantum mechanics and make the comparison. Unfortunately, I'm not in a position to make such a comparison.

That aside, the incentives are different. Medicine --> $$$. String theory doesn't.

mattchew2• 16 years ago

Some time back on--I think--Overcoming Bias, I read that less than half of medical studies are verified when independent verification is attempted.

Does anyone have this reference? I can't find it again.

nazgulnarsil• 16 years ago

http://www.overcomingbias.com/2009/07/meds-to-cut.html

http://clinicalevidence.bmj.com/ceweb/about/knowledge.jsp

mattchew2• 16 years ago

Thanks NN. I don't think this is quite the same thing as what I was thinking of, but useful anyhow.

JoeAltmaier2• 16 years ago

Three kids, same pediatrician. First kid: "Drink lots of whole milk". 2nd kid: "drink only skim milk; kids are too fat". 3rd kid: "drink 2% milk in moderation".

teilo• 16 years ago

Oh, I could tell you stories. OBGYNs who didn't know the answer to a question, and spent all of 10 minutes Googling it (I'm not kidding) and coming back with an "authoritative" recommendation. But really, there is a big difference between researchers and practitioners.

You do point out a very important phenomenon, however. A good many doctors don't know a thing about proper statistical analysis, or bayesian reasoning. They implicitly trust whatever research falls into their laps, even if they don't fully understand it, and they begin to treat based upon their faulty understanding.

a-priori3• 16 years ago

Patient-care medicine is not a science. That is, doctors do not use the scientific method when dealing with patients. It's certainly informed by science regarding best practices and techniques, but diagnoses and treatments ultimately comes down to a doctor's intuition about a given situation.

bpyne• 16 years ago

A lot comes down to past experience as well. For good doctors, it's a learning process that starts in residency and never finishes.

dkarl3• 16 years ago

I have a friend who's a doctor. A few years ago I read a few books about nutrition for laypeople, and when I had questions, I tried asking him. He couldn't tell me much. Virtually everything I asked him about, I knew more about it than he did, just from reading a few books. He obviously knew more about enzymes, metabolic pathways, and cardiac arteries than I did, but when it came to practical questions like how much protein I need to consume or how well calcium is absorbed from broccoli and collard greens, he had learned literally nothing in medical school. The closest he came to studying nutrition was when he took a special elective unit on diabetes in rural Hispanic populations.

The way he sees it, nutrition is a separate specialty done by people who study less than doctors and get paid less than doctors, and it isn't taught in medical school, so he isn't embarrassed at all by his ignorance. He regularly gives advice to people who have risk factors like high blood pressure or high cholesterol, but the advice he gives them (eat in moderation, eat more vegetables and less junk food, exercise a little) is pretty generic, and none of them follow it anyway.

foldr2• 16 years ago

>He regularly gives advice to people who have risk factors like high blood pressure or high cholesterol, but the advice he gives them (eat in moderation, eat more vegetables and less junk food, exercise a little) is pretty generic, and none of them follow it anyway.

Doesn't that kind of justify his ignorance, though? Doctors don't have time to be experts on everything (or on anything, really), and it seems unlikely that an ability to hold informed discourse on the nutritional value of broccoli would have anything more than a marginal effect on outcomes for his patients.

ericd• 16 years ago

Doctors are typically educated in how to fix problems, not so much in prevent them. This is unfortunate, since most people think they're experts in everything including nutrition, and treat them as such, but they're really not.

Maybe if doctors were paid to keep their patient base healthy rather than just fix them when they're sick, they'd look into more of that.

philwelch2• 16 years ago

Makes perfect sense if the first kid is skinny, the second kid is fat, and the third kid is medium-sized.

JoeAltmaier2• 16 years ago

Three identical skinny-as-a-rail Altmaier boys.

giardini• 16 years ago

"Why Most Published Research Findings Are False" (2005) by John P. A. Ioannidis

http://www.plosmedicine.org/article/info:doi/10.1371/journal...

carbocation• 16 years ago

The first portion of the article entirely focuses on multiple testing. If you don't perform alpha spending or Bonferroni-type correction despite looking multiple times at your data, you will - of course! - find spurious associations. This is simply bad science and/or lazy refereeing on the part of the journals.

In human genetics we have Mark Daly, David Altshuler, and Eric Lander to thank for doing the thoughtful theoretical work beforehand to identify the alpha for identifying genomewide significance for any study: 5E-8. To my knowledge, GWAS results are much less commonly found to be invalid in a later study.

The message is this: everyone knows at least one method of correction for multiple testing, and virtually everyone knows that they should be doing this. No journal should publish papers that merely show nominal, instead of corrected, P values.

medgeek• 16 years ago

Most medical studies have a small sample size because it is difficult and often prohibitively expensive to do studies with large numbers of subjects. Small sample size increases the likelihood of spurious results.

The other problem is that mainstream journalism does an abyssmal job of reporting science. How many times have your read a newspaper article or seen a segment on the evening news that tells you that X is good for your health or X is bad for your health as if it was an absolute truth, a law of nature on par with V=IR or F=ma? Then, out of curiosity, you look up the actual journal article published by the scientists and find their claims to be considerably more modest.

a-priori3• 16 years ago

From the end of the article: "... such as compelling researchers to share both data sets and the code for statistical models."

This would be a welcome change. It seems very strange to me that it is not the accepted practice to make available any source code used in the analysis. It's an essential part of their methodology, and any bugs in that code could produce hard-to-detect flaws in their results (dropped data, rounding errors, etc). Just look at the work jgrahamc did on climate change analysis: http://news.ycombinator.com/item?id=1128782.

applicative2• 16 years ago

It is stunning in general that it can count as 'science' and a piece of 'scientific research' though the code employed is held secret. In fact, the use of proprietary software in science is at least as dubious as the use of .doc format for government documents.

gchpaco• 16 years ago

One of the problems I've run into in my current work is that statistics deals with problems a lot like the drunk man looking for his keys under the lamp. As far as I can tell there is one man in the entire world who works on stable distributions (like normal distributions, but with tunable skew and heavy-tailedness parameters), which have been quite useful for me. If you have or believe you have data that has subtle dependencies among the random variables, you could use a copula but multivariate Archimedian copulas are hard to compute with (far as we've been able to tell, at least), copula fitting is a research problem, and copula choice is black magic.

yaroslavvb• 16 years ago

A big problem is that there's often no objective way to pick the "right" statistical test. This gives experimenter the freedom to choose the statistical procedure that favors positive conclusion. Here's a classical example when the statistician has the freedom in choosing between binomial and negative binomial test

Two experimenters contrast treatments A and B. They both have A preferred to B in first 5 patients, and B preferred in the 6th. First experimenter planned to run 6 experiments and count the number of successes, so they get P-value 0.11 for the hypothesis that A is better. The second experimenter planned to run comparisons until B is preferred, up to 6, and got P-value of 0.03. (see Appendix A of http://www.annals.org/content/130/12/995.full.pdf+html)

A realistic example of this issue coming up http://www.jstor.org/pss/2336980

Another example is choosing between one-tailed and two-tailed t-test. When you ask for the probability of effect being as extreme as observed x under null hypothesis, should you ask for probability of effect>x or |effect|>|x|?

Eliezer Yudkowsky goes into some discussion on this http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

The most illustrative example of subjectivity of hypothesis testing is probably the issue of testing strings for randomness. There are many tests for testing whether a particular string of bits is generated by a Bernoulli process, with not one having a legitimate claim to being "the right one"

One way to remove the bias is three-way triple-blind testing, ie to measure effects of existing, new, and placebo treatments, have statistician analyze datasets while blinded to their true labels.

xiaoma• 16 years ago

I'm not a fan of the common sense check. A lot of science is, or at least was, completely contrary to common sense.

However, reproducibility is absolutely vital. If future researchers can't replicate the results of a study, then statistical flukes are going to be a real problem. If studies can be replicated, then statistical error can be handled very efficiently. Each time a given piece of research is reproduced (even if it's only reproduced to serve as a set-up control for a future study), will exponentially reduce the odds of a statistical fluke surviving.

We're so good at medical studies that most of them are wrong

Related Stories