Strategies for making reproducible research the norm

https://elifesciences.org/articles/89736

by Tomte • 3 years ago

97 91 3 years ago

An alternative approach is to encourage students tnduct replication studies, evidence synthesis, or meta-research as part of graduate theses.

Why not require it?

rowyourboat• 3 years ago

Isn't the main problem that academics are measured by the number of publications they publish, and reproductions of existing studies aren't published by the main journals, thus there is little incentive to try and reproduce findings? I never thought this was a problem of ability.

reignbeaux• 3 years ago

Unfortunately this is only part of the problem. Even studies on ML that use public datasets, which are the kinds of studies that when code is shared should be very easy to reproduce, are often surprisingly hard to repeat. Sometimes only parts of the code are published, the code has a lot of bugs (who knows why? Added intentionally?), the code is very badly documented, or the exact libraries are not specified properly.

And this is in a field where everything is based on code, where in principle reproducibility is easy. Go into materials science or chemistry and try to synthesize something following a published paper and you get all sorts of problems. Different equipment, different temperature, not all steps documented, ... Reproducing experimental findings can take you months.

mitthrowaway2• 3 years ago

Page limits certainly don't help!

sigtstp• 3 years ago

It still largely comes down to incentives from what I've seen. A lot of times all anyone (from the researcher to the reviewer) cares about is the paper. Journals don't check that code actually works, and a lot of researchers don't spend time on preparing their code. They feel there's no need, since they now got a new article on their CV. It's true that they may not have the skills and experience to produce good code they can share (depending on the area), but often 1) there's no time to prep code since they've got 3 other projects going on and a crazy work pace 2) the code is seen as something incidental and secondary - what matters to them is the figures and results 3) some groups want to milk a topic for a few papers so they're guarding their code and data. Luckily at least plenty of journals demand access to data or even making it public.

geokon2• 3 years ago

If you fail to reproduce some important research then I think that would absolutely get published. (see the recent superconductor drama)

So if you feel some impactful work is suspicious .. I think disproving it would absolutely be incentivised

If you show its actually correct.. Well then usually it's not that hard to push the envelope a bit further and say something new. That happens all the time

ttymck• 3 years ago

It seems to me, instead of funding a new college or traditional research institute, some benefactor ought to fund a "research reproduction institute", dedicated to identifying and reproducing suspicious publications.

sharma-arjun• 3 years ago

Yes, but in the vast majority of cases, it's hard to tell just by reading a paper if there's been dishonesty somewhere in the pipeline.

Also, the LK-99 example is an exception, not the norm–the chances of receiving significant attention for a replication study are near zero in almost all other cases.

empiko2• 3 years ago

Another issue is that making things reproducible costs you time and that is exactly what most researchers do not have. For example, many ML papers have code that is just a barely working Jupyter notebook. To make it reproducible you would have to create a reproducible environment, package the data, and prepare scripts that would rerun all the experiments you have done. That can take several weeks, but it will not increase the chance of acceptance for your paper at all.

BenoitP• 3 years ago

No. Several weeks is the time it takes to learn and master Docker.

About two hours is the cumulative time one must cater to the Dockerfile for a 3 weeks project.

But it requires institution insisting on reproducibility, and fostering best practices to make it even easier for the researchers to be compliant.

I get it that reproducibility can be quite hard for biology. But ML cannot be taken as an example of a hard problem.

sideshowb• 3 years ago

Perhaps the core issue is that academia excels at being a textbook case of goodhart's law. If/when reproducibility became a target then the academic system would/will likely make an equally bad mess as it has with its current targets.

tangjurine• 3 years ago

Maybe the cause is funding sources that fund researchers publishing too often, and not funding other researchers to double check their work

sundarurfriend3• 3 years ago

More precisely, making things reproducible after the fact costs you significant time - there are tools for reproducible setups that take maybe an hour (at most) to setup upfront, after which it takes very little effort to do your work within that framework and keep things reproducible (for eg. Julia has DrWatson, DataDeps, etc., I'd be surprised if Python doesn't have equivalents).

The problem is knowing upfront which of your work would need to be reproducible, or having the discipline to do all your hacking starting from such reproducible setups.

somenameforme2• 3 years ago

You're also leaving out the biggest issue. Journals generally don't want to produce negative results. If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published. It motivates everything from HARKing [1] to outright data manipulation. By contrast if negative results were seen as valuable, then none of this is an issue.

On the other hand, it really is the case that there's just not much of any value in learning that [shocking possibility] is, as everybody would naturally expect, indeed not the case. And filling up limited journal space with such discoveries would seem to be counter-productive, at best. And when you have limited space/funding for researchers, one guy who keeps proving everything everybody knows to be false, to be false, is always going to be perceived as less valuable than one making [shocking discovery] [... which ends up being proven false years later].

[1] https://en.wikipedia.org/wiki/HARKing

staunton2• 3 years ago

By "journal space" you of course mean journal prestige that isn't unlimited. The point of science journals is gatekeeping.

QuesnayJr2• 3 years ago

It can also be hard to judge whether replication failed because the result is bogus or replication failed because the replication team is themselves incompetent.

edgyquant4• 3 years ago

So just make sure someone unaffiliated has to be able to reproduce whatever research has been conducted. Tough luck if it doesn’t make it, that’s why you do your best to ensure you’ve verified it’s a real result.

bluGill3• 3 years ago

The biggest problem is honestly obtained incorrect results. If you run 1000 experiments across 1000 labs. Few will statistically not notice a mistake and get a wrong result. That wrong result is then published as it is surprising.

naasking7• 3 years ago

If you spend researching [shocking possibility] and it turns out that [shocking possibility] isn't true, you're not getting published

But this simply isn't true in physics where negative results are very common. This is at least an existence proof that this can work, people just have to get their heads straight on what research means.

crabbone• 3 years ago

Sort of. Yes and no. There has to be a metric to assess researcher's performance. Otherwise we won't know what research is worthwhile. When the rules of the game are known, players will find their way to cheat, or at least bend the rules to their advantage.

So, for example, suppose negative results become as valuable: well, they are easier to produce. They are also less valuable as stepping stones for further research. Given that, you'd still need to have a metric that compares publishing positive results to negative results. Even if you declare them to be equally important, the shared understanding will be that they aren't. And one would be more important than the other. And here were are back to square one.

There are some minor things that can be done in the near future. For example, results produced with code must come with the code that produced these results. A lot of research bodies resist this because they want to commercialize their code, or their code may inadvertently contain organization's secrets and therefore needs more auditing... but, in the end of the day, it needs to be made clear that this is a necessary and unavoidable price to pay.

Data sharing is even more problematic. Beside confidentiality concerns, data is always a bargaining chip in the game of getting collaborators (and grants). Should it be made public, it loses its value to those who collected it. Right now, the trend is: if you managed to collect a worthwhile dataset, then you'll cover yourself foot to head with NDAs, contracts of all kinds etc, and will sit on it, exploiting it for a series of research. And if anyone wants to do research on the same subject, you will only invite them if they bring grants or equipment etc.

But you cannot really verify results w/o having the data available. Even if you have the code.

---

It's really sad to see how research is doing wrt' programming in part because of the above, but I don't think the programs outlined in OP will have a noticeable effect. They don't paint a convincing picture in terms of incentives, i.e. they don't answer the question why would researches want to do any of that RepRes and OS training. Even in computationally-heavy research today you often find that all the computation work is outsourced by the researchers and they themselves have no clue what their code is doing.

Above were all sorts of arguments for why the current (or yours) approaches are ineffective. But I don't claim to know what needs to be done.

fsckboy• 3 years ago

it's not entirely important (theoretically) that research be reproducible, but it would be important that the attempt at reproduction is made in a timely way. In many cases today, it's not so easy to set up the experiments in the first place to attempt it.

naasking7• 3 years ago

it's not entirely important (theoretically) that research be reproducible

Well that's confusing, because reproducibility is a core tenet of science. How else could you tell whether your results were a statistical fluke or due to some flaw in the study design?

oddmiral• 3 years ago

12. Create a journal for reproduced papers only.

Qem2• 3 years ago

http://rescience.github.io/

seydor• 3 years ago

the root of the issue is trust in published science. if people focus too much on reproducibility, then you will get people trying to game the system to publish reproducible but still misleading/cherrypicked science

RandomLensman6• 3 years ago

The important thing to change would be that the consumers of research need to care about reproducibility. Currently, that is only the case in very few instances and there reproducibility is typically checked quickly.

robertlagrant2• 3 years ago

Idea: create a journal that is GitHub based.

It generates any data plots/tables through CI, which means you need to have your data available and consumable, and containerised code. Its PR reviewers are scientists doing peer review.

Then you get public review and comment, reproducible environments, and open data.

fleischhauf• 3 years ago

I like the idea a lot. I don't expect a lot of feedback/reviews from other scientists, there are just not that many and they are likely busy with something else

rsfern• 3 years ago

You could still have assigned reviewers like in a conventional journal, which maybe if done right could stimulate broader discussion

JOSS has a review process like this, but I don’t think I’ve heard of any journals focused on primary scientific results using this kind of GitHub based review process

https://joss.readthedocs.io/en/latest/submitting.html#the-re...

robertlagrant2• 3 years ago

Thanks! I was thinking an actual journal that would have a subscription.

I wonder if it could also get people to sponsor replication studies, and publish them linked alongside.

smokel• 3 years ago

I was recently interested in road traffic speed prediction.

The problem with academic research into predicting speeds is not as much the lack of reproducibility, but the lack of measures that make sense.

Hundreds of papers compare a short-term prediction (say 30 minutes ahead) using mean average error, or similar measures. This completely ignores the fact that vehicle speeds are mostly constant and only vary significantly during rush hours or other disruptions, at which time the process of breakdown is rather chaotic.

From what I gather this will not change easily, because it is considerable academic fun to improve on a number for existing benchmarks, instead of redefining the game.

I'd love to help academics in this field who need some guidance on practical applications!

daoboy• 3 years ago

It seems remiss to leave out the publish or perish dynamic that requires anyone pursuing a career in academia to churn out publications with little to no regard for their quality. Especially in light of the evidence in recent years that peer review is a poor filter for fraud.

Deincentivising an endless stream of trash publications will both cut down on the BS and narrow the field of research one might try to reproduce.

I don't have strong feelings about the optimal number of papers that should be published, but my sense is that we are running an order of magnitude above that number.

edgyquant4• 3 years ago

Why not just require independent reproduction to even publish?

mikrl• 3 years ago

That would also incentivize the paper writers to be exacting in describing their methodology and lead to higher quality papers.

If a paper needed to clear the bars of

- can be reproduced by reading the paper and/or a few meetings with the original team

- data is independently reproducible to within the usual statistical parameters

It would presumably increase the signal:noise of research.

bluGill3• 3 years ago

Most science isn't easily reproduced. Often it takes a lot of time and money to get the right equipment. The worst case is super colliders where there is one in the world capable of doing it and so we have to trust the staff. Even simple cases though is still significant effort. There isn't much low hanging you could reproduce it tonight in your basement science left.

danaris• 3 years ago

This would also require a reduction in the "publish or perish" pressure, because it would significantly lengthen the time from study design to publishing.

Furthermore, who's going to be doing that independent reproduction? How are you going to incentivize that?

If you require independent reproduction for every paper in a field where that's even possible, that's going to mean either you need to double the rate of running experiments in that field (unlikely), or double the number of researchers in that field (also unlikely) if you want to maintain anywhere close to the current rate of papers being published. (And again, even if you could double the number of researchers, you still need to provide enough of an incentive to reproduce that literally everyone is doing so for half of the studies they run. And you'd somehow have to ensure that you don't end up with fun, interesting studies getting 15 different groups reproducing them, while less-flashy studies that are still important fundamental science languish for years without anyone caring to try.)

Given how much of "publish or perish" is institutional inertia and culture, the idea that it could be systemically "nudged" to allow for this kind of shift is essentially a fantasy. Far too many of the Old Guard who are already in charge of the tenure committees would say, "Well, I had to publish 327 papers to be given tenure, and kids these days have so many newfangled things that let them write papers faster; why shouldn't we require them to publish 327 papers, too??"

RandomLensman6• 3 years ago

Is it even clear that it would be worth it? Or do you propose a drastic reduction in papers to go with it, i.e., only things valuable enough to have someone replicate it are published?

mnky9800n• 3 years ago

Whose going to pay for that?

civilized2• 3 years ago

I don't know if we will get a healthy research community by attempting to police unhealthy communities out of existence. This will only lead to research activity being further stifled by useless bureaucracy that fails to restrain most bad research while slowing down what actually matters, quality research. Instead, I think we need to focus our attention on outstanding communities and how to distinguish them from less outstanding ones.

We don't need to be threatened by the mere existence of fake journals that publish articles with "counterfeit consciousness" in them. No one actually reads that stuff. It only exists to feed badly-managed communities with terrible incentives. And the solution to such bad communities is to create good ones and showcase their work. Not waste our time obsessing over the fear that somewhere, someone is not a great researcher.

Poor research is the norm and should be ignored. Good research is the exception that should be recognized and nurtured.

naasking7• 3 years ago

I'm a little confused by your post. This paper is about increasing training about reproducible research and open science at all levels of education, with the goal that awareness of them will increase participation and reduce the incidence of pitfalls that lead to faulty research. Is this not exactly showing people how to conduct "good research that should be recognized and nurtured"? Is this really any different than requiring training in proper statistical methods?

civilized2• 3 years ago

Nevermind, it looks like I got mixed up responding to a different comment or article. Sounds dumb but it has happened to me a couple times over the years...

analog31• 3 years ago

Here's a thought: Teach it in elementary school, so that it becomes a habit, and benefits those who are not training to become professional scientists. It doesn't even need statistics or "code" since reproducible science predates those things.

If it becomes a habit, and a natural expectation, then teaching the specifics pertaining to any specific field at an advanced level should be easier. And maybe it would encourage the public to demand better science reporting.

Also, mark the topics in the introductory textbooks that are not based on reproducible research, so that students know the actual state of the field that they're studying.

I do industrial R&D, and my stuff doesn't get published at all, but I benefit from doing work that is at least "open and reproducible" within my organization. It actually improves the quality of my work.

corethree• 3 years ago

No. Teaching it in elementary school doesn't do shit. Everybody knows this stuff.

The reason why a lot of people don't make reproducible research is the same reason why communism tends not to work.

In order to make it work you need to put incentives in place. For example. No journal publishes a paper unless the work has been reproduced by someone from a different institution.

brightstep• 3 years ago

We need a publicly funded body whose purpose is to issue grants for firms to reproduce scientific findings. A scientific reproduction corps. Extra can be awarded for finding flawed/fraudulent research. This would create a community of organizations that counter-balance industry incentives and instill trust in our scientific process.

Strategies for making reproducible research the norm

Related Stories