Science Exchange Reproducibility Initiative

http://reproducibilityinitiative.org

by abbottry • 14 years ago

113 35 14 years ago

reproducibilityinitiative.org

abbottry2• 14 years ago

Some supporting information:

- Nature: http://www.nature.com/news/independent-labs-to-verify-high-p...

- Slate: http://www.slate.com/articles/health_and_science/science/201...

- Reuters: http://www.reuters.com/article/2012/08/14/idUS117855+14-Aug-...

soperj• 14 years ago

Proofs are for math, not science.

lobo_tuerto• 14 years ago

Maybe the title should say "probe it". Just saying.

diego• 14 years ago

Agree, the title is poorly editorialized. Proofs and validations are very different things. Validation is about being confident that a model can make useful predictions.

aakil• 14 years ago

Plasmyd does this via crowdsourcing, it lets users comment on papers so that researchers can point out anomalies or post their inability to reproduce the same results. It also gives the author a chance to explain their work.

bmahmood8• 14 years ago

Bilal from Science Exchange here. Definitely agree that Plasmyd is a valuable service. We feel this Initiative operates in complement, providing a mechanism for independent validation.

pcrh5• 14 years ago

It is frustratingly difficult to get biomedical scientists to openly discuss the work in their field. PLoS tried to have discussion sections to papers, and Nature has tried various approaches as well. Neither have worked. I looked-up my favorite topic (a popular one) on Plasmyd, and found 20 papers, none of which had a single comment associated with it.

It's honestly a vast waste of intellectual potential, I wish I knew the solution!

fbomb• 14 years ago

"Validations are conducted blind, on a fee-for-service basis."

Seems like a potential conflict of interest there.

archgoon• 14 years ago

How? If they don't know what the answer is supposed to be, it's not clear how they can fake data. This can further be improved by universities contributing to a 'noise' fund, where requested experiments are expected to _not_ turn anything up.

Also, the main purpose of this is not to catch corruption, just prevent mostly honest researchers from fooling themselves (and then others), and increasing the base line for research.

bmahmood8• 14 years ago

Bilal from Science Exchange here. The validations are blind in the sense that we match an investigator who has submitted a study to a provider who can validate it. The investigator doesn't have a choice of who validates their study though.

The fee-for-service actually helps to guard against biased results in two ways. Firstly, providers who validate studies are paid on completion, regardless of a reproducible or irreproducible outcome. Secondly, operating on a fee-for-service basis allows providers to operate outside academic incentives for publications, ensuring no incentive for biased-positive results.

pcrh5• 14 years ago

I see it as something like the National Institute of Standards and Technology (http://www.nist.gov/index.html).

I don't think most academic labs will pay for this. However, I can see it as being something that VC investors might like to see the next time someone presents with a wonder drug.

Xcelerate• 14 years ago

This is great! Too often scientific papers are published that are never actually checked by anyone else. A key part of the scientific method is reproducibility though.

graeme• 14 years ago

I was very surprised when I first learned that results could be published without first having been reproduced. I thought at the least they should have been published with a separate label.

This is a worthy initiative.

tokenadult• 14 years ago

I was very surprised when I first learned that results could be published without first having been reproduced.

A lot of psychology journals have been burned by that over the years,

http://chronicle.com/blogs/percolator/is-psychology-about-to...

http://simplystatistics.org/post/21326470429/replication-psy...

http://scholarlykitchen.sspnet.org/2012/04/26/reproducibilit...

and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science,

http://norvig.com/experiment-design.html

and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline.

(Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin,

http://www.apa.org/pubs/journals/bul/index.aspx

which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for publication in most of the better journals. Perhaps that guideline will eventually be made explicit for all of the better journals.)

_delirium• 14 years ago

Traditionally, the public literature is how scientists communicated, which is what made republication possible in the first place. Lab A, perhaps in Germany, works on a research project, and gets some results. They write up their methodology and results, and communicate them to other scientists by publishing the write-up in a journal. The journal will have the write-up peer-reviewed before publishing it, but that doesn't mean verifying it's correct, only that it's a contribution to the scientific literature worth communicating to its readership. For example, it would make sure the paper has done appropriate data analysis, has cited relevant existing work, is not outright kookery, etc.

Now other scientists, anywhere in the world, will read this and decide what they think of it, which may include attempting to replicate it. If they find different results, or have a qualm with an interpretation or methodology, they can write a response, which will also (after peer-review) be communicated by the journal to its readership, either as another paper, or if shorter, as a letter. Thus the back-and-forth of claim/replication/critique happens, all in the open air of the journal's pages, with any lab anywhere in the world able to follow the conversation and jump in if they wish.

To me that's more transparent than doing all this in some kind of closed pre-publication process. Perhaps in the modern era, the open communication doesn't necessarily have to happen in journals anymore: perhaps it should happen on arXiv, and journals should become more conservative in what they publish. But I would like it to still be public.

podperson• 14 years ago

The title irks me. Instead of "prove it", how about "let us reproduce your results" or "show us the data". Proof in this context is a mistaken concept.

semenko• 14 years ago

There are very few journals that do this -- I can't imagine this ever taking off.

Reproducing experiments seems like a costly (and mostly thankless) effort that few PIs would ever take up.

The only journal I know of that completely reproduces results is Organic Syntheses (http://www.orgsyn.org/), which reproduces every reaction before publication and has a Procedure Checklist for authors: http://www.orgsyn.org/AuthorChecklist.pdf

bmahmood8• 14 years ago

Hi, this is Bilal from Science Exchange. Appreciate your concern, but that is why we feel Science Exchange is uniquely positioned to make a difference. We have developed a network of 1000 core facilities and CROs on our platform, who operate on a fee-for-service basis to conduct experimental services. The Reproducibility Initiative will leverage this network to outsource specific experiments within a study to these facilities.

pcrh5• 14 years ago

It is quite common for experiments to be replicated, usually as preliminary to a further advance.

If the findings claimed are significant enough, other labs will often attempt to replicate a few of the key experiments, and if they don't do so quickly, abandon that direction of investigation, and not report their negative findings.

abbottry2• 14 years ago

Title changed -- geeeeeze #cowersinthecorner

kevinalexbrown• 14 years ago

This is excellent. I have a few questions:

You need to provide the background of your study, the types of experiments undertaken, the materials and methods, and initial results of your study.

Do the technicians reproducing the results get to see the initial results? It seems like it might be more accurate if they didn't. Lots of parameters can be fudged and adjusted, a la Millikan's Oil Drop, when the results don't quite match. I imagine this might be exacerbated with the necessity of researcher-validator communication.

How are conflicts resolved? If my results are not validated, someone made a mistake - me or the validator. If both parties stand by their mutually incompatible results, where does it go from there? I can imagine a lot of researchers I know feeling annoyed that someone whose expertise they cannot verify (due to anonymity) won't "do my experiment correctly".

I imagine that in time there might be specific requirements or explicit funding allocations for such reproduction on grant applications, which would really allow it to take off. As it stands, I imagine a lot of PI's would just ask "hmm, I can spend money that might risk my already high-impact paper, or I can keep the money and not be considered wrong."

Still, this is a great first step toward facilitating a central tenet of the scientific method. Congratulations.

bmahmood8• 14 years ago

This is Bilal from Science Exchange, and we greatly appreciate your support for our initiative!

We will provide the methodology of the original study to those reproducing the results, while the results we believe will be helpful to check against.

For conflicts, it's true we can't force an investigator to publish or note the lack of reproduced outcomes. We do hope they will through the PLOS Collection, for transparency. We do feel though it can provide a valuable check for 'failing fast', for those investigators who want robust results.

In this initial stage, we also agree that funding will be difficult. That's why we hope to focus on small biotechs and research labs that are interested in commercializing their research, and need to show robustness of results for licensing opportunities.

Hopefully then, it can serve as a proof-of-principle for funding agencies to provide a requirement or increase support for reproducibility.

pcrh5• 14 years ago

I think this will be a very valuable service for small biotechs. It would improve the value of their patents and increase chances of getting further funding.

As an experienced scientist myself, I can say there is plenty of scope for misunderstanding and misinterpretation, no matter how careful the original authors were. So I only hope that there will be some (maybe blinded) mechanism for communication between the original researchers and those replicating the published findings, in the event of the "usual" complications.

Also, who would do this replication? Would some of it be outsourced to academic labs with the requisite experience? Advanced findings often depend on advanced techniques. Outsourcing could be problematic, politically, since Big Shot No.1 might not be too interested in shooting down his pal, Alpha-MD-PhD.

FrojoS• 14 years ago

Bilal, I just want to thank you whole hearty for your, and your teams, effort! I'm a Phd student in engineering, where as far as I can tell, it is even worse. I love academia, but the fact that we don't even attempt to live up to the standards we supposedly agreed on makes me sad.

Especially in computer focused areas like CS or Statistics, it would be straight forward to submit more or less completely self-reproducible papers. Alas, it is seemingly impossible to find an adviser who will allow you to publish all your source code and primary data, let alone spend the additional time to get them into a publishable form.

frisco2• 14 years ago

It's very expensive (prohibitively so) to reproduce any substantial study. It also feels like there's a lot of potential downside and very little reward, since there's a presumption of correctness in published papers today. Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly.

Where does the money for this come from? Are you expecting that labs will write this cost into their grants? Have you seen interest from grant agencies to then actually pay for this?

bmahmood8• 14 years ago

Hi, this is Bilal from Science Exchange. We are in discussions with funding agencies and foundations to possibly gather increased financial support. We feel this initial launch will provide a proof-of-principle to gather support.

Currently though, we hope investigators will pay for the validation themselves. We believe it can provide a valuable service to those in small biotechs or those who want to commercialize their research, to improve the robustness of outcomes for potential licensing opportunities.

gwern• 14 years ago

> Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly.

That kind of illustrates the usefulness of replication! If the effect is so finicky that it cannot even be replicated in another lab, in what sense is that result interesting, useful, - or even real?

frisco2• 14 years ago

I think you overestimate most biology labs. It's really easy to screw up a protocol in subtle ways. Often it took the original lab weeks of debugging to get it to work (yes, to get it to work, not to produce a false positive).

Things get more reliable as time goes on and methods become better understood, but the critical steps in most experiments are usually nontrivial to replicate in the beginning. Especially by CROs, which are notorious black boxes, incredibly expensive, and often just plain old unreliable.

joe_the_user• 14 years ago

I don't see this approach really able to solve the underlying problems that it references from

http://www.nytimes.com/2012/04/17/science/rise-in-scientific... and other articles.

"Each year, every laboratory produces a new crop of Ph.D.’s, who must compete for a small number of jobs, and the competition is getting fiercer. In 1973, more than half of biologists had a tenure-track job within six years of getting a Ph.D. By 2006 the figure was down to 15 percent."

I would claim that science requires some basic integrity in its practitioners and if an institution is treating it's members as so many throw-away resources, it is hard to expect those members to move ahead with great idealism. The model of every desperate competitor watching every other competitor seems to be the replacement for model of science as a high ideal. I don't see it working out well.

bmahmood8• 14 years ago

It's true the issue of reproducibility in research is a complex one, with many underlying factors. Culture, misaligned incentives, and lack of resources play strong roles.

We believe however our Initiative can help in laying an initial framework for how one can possibly address aspects of the problem. We've tried to build incentives (ease of outsourcing, rewarding publications) that factor into this issue, and can assist in improving outcomes. But we definitely agree that a holistic solution will require further changes to the academic research infrastructure.

jayzee• 14 years ago

The part I like about the execution is where you have CROs and Core Facilities, and not scientists themselves validate results. Apparently Milikan when he measured the electric charge was off (smaller than actual) in the measurements... but researchers who checked, worried perhaps about their academic reputation (speculating here) slowly adjusted this number upwards over several years till it asymptotically approached the truth. Having a 3rd-party, that is less impacted by academic politics might be a good thing.

From Feynman's Caltech commencement speech on this:

We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off, because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover that the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease.

[1] http://www.lhup.edu/~dsimanek/cargocul.htm

larsberg• 14 years ago

The NSF CISE guidelines (which apply to much of the funded CS research that folks on HN care about) currently require you to retain all data required to reproduce your experiment and make them available to any reasonable request: http://www.nsf.gov/cise/cise_dmp.jsp

My advisor is on appointment there right now, and they're working on guidelines that require you to provide full reproducibility of your program results (modulo system differences). At least for software, things are looking up. Of course, reproducibility is both infinitely easier (I can do it on any system!) and harder (what do you mean the kernel patchlevel or CPU/GPU sub-model-number matters?).

Science Exchange Reproducibility Initiative

Related Stories