Why most published scientific research is probably false [video]

http://www.economist.com/blogs/graphicdetail/2013/10/daily-chart-2

by xijuan • 13 years ago

62 53 13 years ago

Probably false? As in you have a better chance claiming the negation of a scientific paper's conclusion than the actual conclusion? I doubt it.

bigchewy• 13 years ago

yes, probably false. This is not news, e.g. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ came out in 2005.

of course, research "proving" that research tends to be false can get us into a little circular game...

a_bonobo2• 13 years ago

There's a follow-up paper to the one you linked that claims that much less published research findings are false:

We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with respect to journal submissions (0.5% more FP per 100 submissions, P=0.12).

[1] http://biostatistics.oxfordjournals.org/content/early/2013/0...

The circle has begun :)

mjn• 13 years ago

In certain fields, that's not implausible. Many results are along the lines of "compound X is effective against disease Y", where the negation, "compound X is not effective against disease Y", is a reasonable baseline assumption, because most compounds are not effective against most diseases.

Results where the prior odds could plausibly be considered 50/50 are another story. Scientific research that doesn't take the form "rule out the null hypothesis with p values" is also a different matter (lots of CS and physics, among other fields, has a more complex mingling of theory and epistemology than experiment/nullhypothesis/pvalue/repeat).

jamesaguilar2• 13 years ago

That's a good point. I didn't consider the fact that possibly many discoveries have a very low prior expectation of truthood.

zzleeper• 13 years ago

I wouldn't be so sure. Not talking about mathematics or CS, but in many social sciences (and also medicine and so on) the paper is just stating that A->B but i) there can be a lot more things going on that explain whatever correlation you are finding (from reverse causality to bad experimental setup), and *more importantly ii) in order to get published, you need to present a somewhat interesting or controversial statement.

If you state something obvious and most likely true, good luck getting to Nature/Science. If you state something unusual, then that will sell (where selling is getting citations) so you will see it published

XorNot• 13 years ago

Asimov wrote an excellent piece on this - The Relativity of Wrong: http://chem.tufts.edu/answersinscience/relativityofwrong.htm

The Earth isn't flat, but it certainly approximates being flat for small measurements of its surface (such as those early civilizations would've been able to make).

pallandt2• 13 years ago

A good opportunity for mentioning Benford's Law: http://en.wikipedia.org/wiki/Benford%27s_law#Scientific_frau...

kineticfocus• 13 years ago

Maybe an opportunity for mentioning Simpson's Paradox: http://en.wikipedia.org/wiki/Simpson%27s_paradox

pallandt2• 13 years ago

Good point!

officialjunk2• 13 years ago

so... non-scientific articles about scientific research are true? probably not.

chattoraj• 13 years ago

See the original paper: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

officialjunk2• 13 years ago

i see. but then again, wouldn't their findings imply their own published results are likely false?

chasing• 13 years ago

Look. Clearly science makes progress that's "true" in the sense that it becomes useful and can be used as functional models of the way things work.

This video's using a very reductionist kind of statistics to point out that, yes, an individual piece of research making a claim might have a good chance of being wrong. Which is why science doesn't say, "Oh, well Larry just proved that Saturn orbits Uranus so let's just never think about that again and instead move on to proving that the Sun is fully powered by excess heat radiating off of Elon Musk's brain." Science is a process that works in aggregate, using a large volume of research and scientists checking one another as a way to smooth over this very imperfect process. Over time. Science checks itself. That's the whole point. That's why it reaches some pretty damned good conclusions about the way things work.

So.

I don't know what the point of this video is. Science is wrong? Scientists are stupid? The Economist is smart? I should believe the Republicans when they say the Earth couldn't possibly be warming because that one time it snowed in a part of Texas where it never really snows all that often?

pygy_3• 13 years ago

Science checks itself.

Not that much, actually. There's no point in reproducing the work of others to advance your own scientific career. Unless your research directly directly depends on them, it is counterproductive.

epistasis5• 13 years ago

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding.

In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

gabriel344• 13 years ago

Actually there are loads of scientific papers only reviewing other papers (usually a set of them), trying to infer conclusions and criticizing them on methodology and rigidity. Scientific careers advance both in quality and in quantity of papers published. Not everyone gets featured on Science or Nature.

dhaivatpandya• 13 years ago

Agreed. Try getting funding for verifying a paper's results. If you can't publish results, it'll be a hard time convincing anyone that your work is worthwhile.

yetanotherphd• 13 years ago

The problem with their reasoning is that it relies on a very high prior that the hypothesis is false.

In fact an explicit analysis of the prior over the hypothesis and the power of the test, would be roughly equivalent to the informal discussion that goes along with the statistical results.

The main issues in my opinion are that the number and nature of studies that produce null results if unknown, and that there is a bias in the literature towards positive results. While this bias incentivizes researchers to use powerful tests, it comes at a big cost.

leot• 13 years ago

The conclusions of this video depend on an idealized view (and thus a poor model) of research and science. In fact, there are many different kinds of results (associated with different levels of confidence and which almost all require a nuanced interpretation in order to be properly understood) and many different kinds of researchers. The best results, across many fields, are rarely if ever single papers with a single experiment with p<0.05. The good ones have multiple mutually-confirming experiments with much smaller p-values. And often for the very best results, p-value-style analyses are redundant: what would be the p-value associated with the line that Hubel and Wiesel claim was triggering the firing of their cat's retinal ganglion cell [https://www.youtube.com/watch?v=IOHayh06LJ4]? Does it even matter?

[Edit: Parenthesis in first sentence, for clarity]

001sky2• 13 years ago

Two game-theoretic strategies need to be mitigated/bred out out of Academia:

(1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc).

(2) The "lick the cookie" problem, where nobody will verify your results because there its considered degrading (professionally) to 'not be first' at the table, as the author of origin. [a]

These both in combination lead to something of a "tradgedy of the commons" where the basic core of the discipline erodes in presitige/utility, as the individual contributors seek to maximize their personal productivity from the public good (the repuation of groundbreaking science).

[a] This is the childhood strategy of making anything you touch first unatractive to all those who follow.

edits: for clarity.

xherberta• 13 years ago

Interesting, and nicely stated! I wonder what it would look like if funding and publishing could be made to incentivize really valuable pursuits, rather than rewarding trendiness, novelty, and shock value.

snowwrestler2• 13 years ago

Neither of these are actually problems.

If (1), your results are obscure or lack applicability, then they are not useful anyway and so it doesn't matter if they are ever confirmed. A surprisingly large amount of scientific research fits into this category.

While (2) few researchers set out solely to verify another's result, if the result is noteworthy, it will be incorporated as a component of future research and therefore tested indirectly. This other comment (not mine) explains how:

https://news.ycombinator.com/item?id=6662124

eli_gottlieb2• 13 years ago

(1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc).

This is simply a problem of explaining findings to other researchers. Every single researcher with a genuine finding on his hands will, at some point, seem like his work is obscure and pointless compared to "all those other guys" who are building on established research for immediate application.

And then there's the issue that your failure to understand someone's research doesn't mean it isn't research. Sometimes what a field needs is just a few more smart people willing to work at understanding what the hell's going on.

epistasis5• 13 years ago

The vast majority of scientific papers are not single experiments with one p-vaule, but rather a handful experiments to a dozen or more experiments, only some of which may be reduced to a p-value. And in most biological research, at least two lines of evidence are required before a reviewer will accept a claim (e.g. "OK, you may have found something, now verify it with a PCR.").

So this entire setup is just kind of crap, and not representative of scientific research.

In addition, this simple point, which is quite interesting, and necessary to keep in mind when interpreting multiple p-values, is widely acknowledged in the field, which is why False Discovery Rate methods started to be used as far back as the 90s. This initial point was first published as a "The sky is falling, what are all you idiot medical researchers doing?!" type of paper by Ioannidis, which is a great way to make a name for oneself. However, even his own interpretation did not hold up well, and he has stopped pushing the point. Summarizing an extensive comment on Metafilter [1]

Why Most Published Research Findings Are False: Problems in the Analysis The article published in PLoS Medicine by Ioannidis makes the dramatic claim in the title that “most published research claims are false,” and has received extensive attention as a result. The article does provide a useful reminder that the probability of hypotheses depends on much more than just the p-value, a point that has been made in the medical literature for at least four decades, and in the statistical literature for decades previous. This topic has renewed importance with the advent of the massive multiple testing often seen in genomics studies.Unfortunately, while we agree that there are more false claims than many would suspect—based both on poor study design, misinterpretation of p-values, and perhaps analytic manipulation—the mathematical argument in the PLoS Medicine paper underlying the “proof” of the title's claim has a degree of circularity. As we show in detail in a separately published paper, Dr. Ioannidis utilizes a mathematical model that severely diminishes the evidential value of studies—even meta-analyses—such that none can produce more than modest evidence against the null hypothesis, and most are far weaker. This is why, in the offered “proof,” the only study types that achieve a posterior probability of 50% or more (large RCTs [randomized controlled trials] and meta-analysis of RCTs) are those to which a prior probability of 50% or more are assigned. So the model employed cannot be considered a proof that most published claims are untrue, but is rather a claim that no study or combination of studies can ever provide convincing evidence.
ASSESSING THE UNRELIABILITY OF THE MEDICAL LITERATURE: A RESPONSE TO "WHY MOST PUBLISHED RESEARCH FINDINGS ARE FALSE" A recent article in this journal (Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124) argued that more than half of published research findings in the medical literature are false. In this commentary, we examine the structure of that argument, and show that it has three basic components: 1) An assumption that the prior probability of most hypotheses explored in medical research is below 50%. 2) Dichotomization of P-values at the 0.05 level and introduction of a “bias” factor (produced by significance-seeking), the combination of which severely weakens the evidence provided by every design. 3) Use of Bayes theorem to show that, in the face of weak evidence, hypotheses with low prior probabilities cannot have posterior probabilities over 50%. Thus, the claim is based on a priori assumptions that most tested hypotheses are likely to be false, and then the inferential model used makes it impossible for evidence from any study to overcome this handicap. We focus largely on step (2), explaining how the combination of dichotomization and “bias” dilutes experimental evidence, and showing how this dilution leads inevitably to the stated conclusion. We also demonstrate a fallacy in another important component of the argument –that papers in “hot” fields are more likely to produce false findings. We agree with the paper’s conclusions and recommendations that many medical research findings are less definitive than readers suspect, that P-values are widely misinterpreted, that bias of various forms is widespread, that multiple approaches are needed to prevent the literature from being systematically biased and the need for more data on the prevalence of false claims. But calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that “most research findings are false for most research designs and for most fields” must be considered as yet unproven.

[1] http://www.metafilter.com/133102/There-is-no-cost-to-getting...

Fomite2• 13 years ago

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure.

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

capnrefsmmat4• 13 years ago

It's true that effect sizes are often more important, but it's also true that they're also often incorrect. See e.g.

Ioannidis, J. P. A. (2008). Why Most Discovered True Associations Are Inflated. Epidemiology, 19(5), 640–648. doi:10.1097/EDE.0b013e31818131e7

Most studies are underpowered and are incapable of detecting the true effect. Only if they get lucky and observe an abnormally large effect will they obtain a statistically significant result, so the published results tend to be significant overestiates.

For another good example, see

Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: statistical challenges in estimating small effects. American Scientist, 97, 310–316.

http://www.stat.columbia.edu/~gelman/research/unpublished/po...

eli_gottlieb2• 13 years ago

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

Because in science, not believing things is the default state. If you say, "most published findings are false", you're really saying, "most of the time we have to accept the null hypothesis", which is what we all not-so-secretly believe regarding everything, all the time, in any case.

tokenadult• 13 years ago

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

This is a fair question. I think the reasons the Ioannidis paper was persuasive are that

1) Ioannidis replicated earlier results about the lack of replication of most research reports,

and

2) Ioannidis "showed the work" for how possible, and indeed likely, it is for an effect size that permits a false-positive finding to be published, under reasonable assumptions about the prevalence of false-positive findings and publishing practices. Most scientists were vaguely aware of lack of replication years before anyone heard of Ioannidis, but not many scientists were fully aware of how readily a false-positive finding can be published.

lvs• 13 years ago

This is extremely misleading and feeds an anti-intellectual notion that scientists are just lying to everybody.

First, it perpetuates a common claim of those who don't practice any sort of science: that the output of scientific studies is an enumeration of true/false claims determined with statistical inference logic. (Science media/blogs really aren't helping on this one.)

Second, the math is just wrong: the space of hypotheses is infinite, so it's impossible to say what fraction of these are true.

gabriel344• 13 years ago

Isn't the space of already enumerated and studied hypotheses finite?

capnrefsmmat4• 13 years ago

I don't think the technicalities of the size of the hypothesis space or its uncountability matter. We do studies over large numbers of hypotheses, of which we know a priori only a few are likely to be true. Genome-wide association studies, for example, will study thousands of genes knowing that only a few will have true correlations.

In any case it is an entirely reasonable assumption that of all tested scientific hypotheses, a small fraction are true. We often do not expect them to be true. (A clinical trial may test thirty or more hypotheses, such as "this medication causes diarrhea," and may expect most of them to be false. But they have to check just in case.)

gabriel344• 13 years ago

Here is the link to the source, much friendlier to HN folk that rather read than watch: http://www.economist.com/news/briefing/21588057-scientists-t...

EDIT: To add on to this, the source to the video does make an interesting observation about statistical methods being employed by scientists who doesn't know their pitfalls.

Other thing I take from a more careful reading (as opposed to viewing a two minute highly superficial video) is that the article makes the assumption that all hypothesis will be subject to only one study, if we have three studies denying a certain hypothesis and one confirming it it's pretty easy to catch the false positive on a literature review article (routinely done by people entering academia)

anigbrowl• 13 years ago

This is just an adjunct to this: https://news.ycombinator.com/item?id=6566915 (article and HN discussion).

timr• 13 years ago

For me, the really remarkable thing about this graphic is that it doesn't even support the headline: the number of false positives is a minority of total positives in the given example: 45 / 125 = 36%

gabriel344• 13 years ago

IMO what is damaging to society is that there is no PR to science, so the press takes things that are not yet fully understood or verified by the academic community and publishes it as confirmed (studies say, confirm, etc.)

Groundbreaking results give much more press than their rebuttal, for example, see neutrinos faster than light or that arsenic consuming bacteria both of which were later dismissed in academic circles, but did not enjoyed the same treatment from the media.

vixen99• 13 years ago

I can't imagine what process you have in mind to put right what you see as a problem. Who is the arbiter of the press coverage? You are assuming unrealistic habits on the part of the passive observer of the results of research. Life isn't spent checking out media coverage or providing it with checks and balances unless there's a major issue involved which impinges on people's lives. The solution to your perceived difficulty is to check the facts for yourself from original sources and not expect some arbitrary secondary source to keep you up to date because that is something they are never going to do.

leoc• 13 years ago

That sounds very much like the UK's Science Media Centre: http://www.sciencemediacentre.org/about-us/ . How good a thing the SMC has been is another question.

snowwrestler2• 13 years ago

It is a mistake of reasoning to take a meta-analysis of medical research and expand its conclusions to the rest of science.

Medical research has a number of peculiarities among the sciences, including the complexity of its subject (perhaps the highest of any discipline), the emotional reaction to the subject, the speed at which people try to turn scientific findings into products or advice, and the concommittal eagerness to trust epidemiological results without a known physical mechanism.

It's also a huge mistake to get your science news and opinion from The Economist--a magazine with a great reputation that has nothing to do with its coverage of science

bnegreve• 13 years ago

The claim is:

    "most published scientific research is probably false"

and the evidence for that claim is:

    The number of false negative "might easily be 4 in 10, or in some fields 8 in 10"

(quoted from the video)

This is rather weak.

And maybe more importantly: this assumes that researchers test random hypothesis uniformly drawn in the space of all possible hypothesis, which clearly isn't the case.

Anyway, this can't be really serious.

Gravityloss• 13 years ago

It's just basic Bayesian reasoning. Most of the positive HIV test results are false positives. Even when the P value is less than 0.01 or so.

Why most published scientific research is probably false [video]

Related Stories