Half of social-science studies fail replication test in years-long project

https://www.nature.com/articles/d41586-026-00955-5

by MBCook • 4 months ago

6 2 4 months ago

BeetleB• 4 months ago

When some of SCORE’s team members attempted to reproduce the data analyses of 600 papers, they found that only 145 contained enough details to do so.

...

However, many of the failures might have been caused by the SCORE researchers needing to make guesses about procedures or to recreate raw data, Errington says.

Incidentally, this is not unique to the social sciences. Back when I did condensed matter physics, the same was true for both experimental and theoretical papers. The PIs were pretty open about it: Once they perfect a technique, they want to milk it for as many papers as possible. If they reveal too many details, others will be able to outpublish them.

(Yes, it sucked).

h45x1• 4 months ago

I am surprised it's just half. The incentives are not there for replication or verification. You need to publish and it is easier to publish something new than to fight over existing results, especially if you're a young researcher.

I maintain something akin to google trends but for abstracts in economics papers. The word "novel" is growing rapidly, while "reproduce", "replicate", and "verify" have not changed much over the years: https://dubovik.eu/blog/repec?t=replicate&t=reproduce&t=veri...

Half of social-science studies fail replication test in years-long project

Related Stories