Psychology study that induced "reproducibility crisis" was wrong: researchers

http://www.slate.com/blogs/the_slatest/2016/03/03/psychology_study_that_induced_the_reproducibility_crisis_was_wrong.html

by signor_bosco • 10 years ago

28 44 10 years ago

slate.com

Certhas• 10 years ago

Or it wasn't, e.g.:

http://neuroneurotic.net/2016/03/03/the-non-replicability-of...

From where I’m standing, social and other forms of traditional psychology can’t say the same. Small contextual or methodological differences can quite likely skew the results because the mind is a damn complex thing. For that reason alone, we should expect psychology to have low replicability and the effect sizes should be pretty small (i.e. smaller than what is common in the literature) because they will always be diluted by a multitude of independent factors. Perhaps more than any other field, psychology can benefit from preregistering experimental protocols to delineate the exploratory garden-path from hypothesis-driven confirmatory results.

I agree that a direct replication of a contextually dependent effect in a different country and at a different time makes little sense but that is no excuse. If you just say that the effects are so context-specific it is difficult to replicate them, you are bound to end up chasing lots of phantoms. And that isn’t science – not even a “soft” one.

DanBC• 10 years ago

It's interesting to compare that link talking about how it's difficult to replicate psychology (and other) experiments, with this link that says it doesn't matter.

https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitch...

(I agree much more with your link).

nmrm2• 10 years ago

Interestingly, variations on "the exploratory garden-path" are widely hailed by qualitative social science researchers as sound methodology.

vacri5• 10 years ago

The article explicitly says that the 39% reproducibility isn't that far off what you would probably expect anyway.

The article then goes on to add the the meta-study was awful in it's study selection, giving the example: OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam.

The article then goes on to say that reproducibility is still a problem in science, but that the meta-study was simply terrible in it's own methodology. Low reproducability is less of an issue given the context of the problem space, the article says; the meta-study was wrong, both methodologically and in conclusion ('low reproducability' != 'unethical' or 'crisis').

I recommend reading the fine article.

logicrook2• 10 years ago

OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam.

If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. If they did conclude something on "Stanford University students", then it is a bad reproduction. You see, there's a very good justification there, that you can interpret as subtyping, or the liskov substitution principle.

'low reproducability' != 'unethical'

This is akin to saying "ok there's this theorem, but I won't give proof, just a bogus heuristic argument", or even worse, saying "that person is guilty but I won't justify why". Both these behaviors are unacceptable. Why would "social sciences" be above that?

or 'crisis'

A field is filled of politically oriented baseless claims, but this is not a crisis, just another day in Oceania.

That said, if the meta-study was sloppy, then it still justifies that research standards should have been upped a while ago; the "terrible"" meta-study was accepted precisely because terrible papers are accepted in the first place.

koentj• 10 years ago

There are many things wrong with the analyses performed by Gilbert et al. Sanjay Srivastava does a decent job at pointing them out:

https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-cr...

ars• 10 years ago

Am I reading this right?

They are saying there is no reproducibility problem, because it's perfectly normal to be able reproduce just 39% of studies?

rosser3• 10 years ago

If, for sake of a convenient number, half of the original studies had methodological errors that would make the result irreproducible — irrespective of whether the study's hypothesis was correct, mind! — and half of the studies attempting to reproduce the results of the original study committed some form (possibly even the same form) of methodological error as well, then just on that basis, we should reasonably expect to see a whopping 25% of studies being successfully reproduced.

Whatever the actual error rates turn out to be, something like that is going to cut pretty deeply into measured reproducibility rates, isn't it?

p4wnc63• 10 years ago

Why should we expect methodological errors for the replications to be anywhere near that? I'd expect them to be in the low single digit percentages. And across different replications, such methodological errors should rarely be correlated, so as long as the rate is merely less than 50%, then when given several distinct failures to replicate, it counts as solid evidence that the original result was spurious.

hackuser3• 10 years ago

It makes people like Darwin, Einstein, etc., whose results have held up for over a century, look even more impressive.

mucker2• 10 years ago

In this I hold Einstein much higher. If his work didn't hold up significant portions of modern society would fail.

mucker2• 10 years ago

Indeed that seems to be what they are claiming from the article. This to me sounds shady in the extreme. A later poster argues for a "50%" error rate on both sides.

Try building a bridge with that.

Social Science methodology is piss poor and needs to improve. People don't take it seriously for a reason.

analog31• 10 years ago

Perhaps a parallel question might be: Does the publication of psychology (or medical, etc.) studies have the net effect of producing a body of reliable knowledge?

edtechdev• 10 years ago

Too me a big problem with traditional psychology studies (at least in educational psychology) is generalizability (or what is called external validity), not so much reproducibility. A typical study “includes participants [like freshmen psych majors] who have no specific interest in learning the domain involved and who are also given a very short study time”, often mere minutes. When researched in real-world settings or even just real classroom settings, the effects might not only diminish but even reverse.

Most of the popular books on learning now suffer from this, emphasizing strategies that work fine for short-term rote memorization tasks (like memorizing random word lists), but have little impact on or can even hurt higher-order learning and understanding.

mettamage2• 10 years ago

IMO generalizability is a big problem in psychology as a field. This paper "the weirdest people around the world?" explains it quite well: http://www.econstor.eu/bitstream/10419/43616/1/626014360.pdf (open access)

Xcelerate• 10 years ago

That's a very interesting article by the way. I've been reading the whole thing front to back.

flycaliguy• 10 years ago

I've read a couple write ups on this situation and I still haven't figured out how they picked the 100 studies. Was it random? It must have been right?

hackuser3• 10 years ago

I had the same question at the time and never found an answer.

harry83• 10 years ago

Doesn't matter. If read the tea-leaves, listen to rumour or carefully select the 100 papers most likely. The fact you can select 100 papers, do the work to reproduce them and fail in over the half the cases is damning. It shows there exists a systemic problem.

Published, peer reviewed, cited work that can't be replicated is a problem. If it were a random sample of papers it is also a systemic problem.

Take your pick, it's a systemic problem that needs to be addressed either way.

neilc• 10 years ago

From the original paper:

"We constructed a sampling frame and selection process to minimize selection biases and maximize generalizability of the accumulated evidence. Simultaneously, to maintain high quality, within this sampling frame we matched individual replication projects with teams that had relevant interests and expertise. We pursued a quasi-random sample by defining the sampling frame as 2008 articles of three important psychology journals: Psychological Science (PSCI), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: Learning, Memory, and Cognition (JEP: LMC)...The first replication teams could select from a pool of the first 20 articles from each journal, starting with the first article published in the first 2008 issue. Project coordinators facilitated matching articles with replication teams by interests and expertise until the remaining articles were difficult to match. If there were still interested teams, then another 10 articles from one or more of the three journals were made available from the sampling frame. Further, project coordinators actively recruited teams from the community with relevant experience for particular articles."

hackuser3• 10 years ago

The claims of the original study, the one that found problems with reproducibility, are being overstated. From the news report at the time:

Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker.

Also: The only factor that did [affect the likelihood of successful reproduction] was the strength of the original effect — that is, the most robust findings tended to remain easily detectable, if not necessarily as strong.

Here's the HN discussion of the study:

https://news.ycombinator.com/item?id=10131387

nonbel3• 10 years ago

"Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance"

Nope!

AstralStorm2• 10 years ago

Indeed, what it might have meant is "strictly on the basis of p-values" or confidence intervals. Still, that is very risky, p-values attained using different tests cannot be compared. Not to the point, p-values have no meaning if you do not have a control group, because you do not have access to a population that might be under the null hypothesis. P-values are sometimes computed against a non-null hypotheses, e.g. null test is giving people $5 not nothing, while the main gives $100. This is not valid. Or against the same subgroup, which might not really be representative.

aaron6953• 10 years ago

This article is just plain scary.

The 'researchers' chose to do a exclusive pre-release with a journalist/media before you could check the data says it all, they are not scientists.

The more I read about Psychology the more I think it really really isn't currently a real science.

https://xkcd.com/435/ - Currently I think it starts to cut off around chemistry to be honest.

aaron6953• 10 years ago

This is incorrect, right?

"If all 100 of the original studies examined by OSC had reported true effects, then sampling error alone should cause 5% of the replication studies to “fail” by producing results that fall outside the 95% confidence interval of the original study"

It'd have to be a pretty poor(read corrupt) field of study of all the random-ish studies chosen at close to 95%?

If the original studies are legit, than many/most of them would be at much higher confidence intervals?

Only scam fields would they all be around 95%?

https://xkcd.com/882/ - this is more about how bad science creeps in legitimate fields, the authors would be implying all the studies in this fields are this bad.

This would be my reading, not sure....

maxander• 10 years ago

You aren't by any chance a physicist or something, are you? :)

95% confidence is just about the lowest that a scientist in any field would think about publishing with, but in many fields- especially those whose subjects are the delicate and high-variance animals known as humans- statistical confidence is expensive. It wouldn't surprise me that lots of papers are put out that just barely reach the lowest standards of evidence, because grant-funded researchers couldn't afford the 1000 more test subjects it would take to get another sigma.

Which isn't great, but strictly speaking its not doing society a disservice, either- even uncertain knowledge decreases the entropy of our vast ignorance. It does, however, add an important dimension to how these studies should be interpreted- that small chance that any given study is incorrect matters, particularly when there's such a mind-bogglingly huge amount of research being done in the modern day. Its entirely unsurprising to read about a study claiming with p < 0.01 that a glass of wine a day will make your hair fall out, when you consider that there's been many thousands of papers published about things like wine. Confidence- and p-values alone aren't going to save us there.

aaron6953• 10 years ago

Yes, I see you're correct.

Most studies would chose a 95% confidence interval so their statement stands.

vacri5• 10 years ago

The article gives a clear example of why it is difficult to replicate psychology-based studies. Iron is iron everywhere, and behaves similarly. Put iron at STP, and it will behave the same in Massachusetts and Amsterdam. The same is not true of humans - the study they give as an example is "students opinions of affirmative action policies", a clearly political set of questions, taken of people who are not only not in the same political environment, but speak different native languages!

Not being able to reproduce a political questionnaire done of Harvard students on U of A students should surprise exactly zero people, and should never have been used for verifying reproducibility.

harry83• 10 years ago

I'm not particularly interested in Harvard students. I'm pretty sure they're not super interesting to anyone outside of the Harvard diaspora and their friends.

I'll bet 100% the paper was generalising from Harvard students to some wider population, or why would you publish it. If the paper says "this thing we're publishing is only applies to Harvard students and has zero wider significance." I'll eat humble pie and apologise.

Psychology has a big issue of claiming things as fact that are no such thing. It has had this issue since forever. The history of psychology research is truly ethically hideous both in terms of what it did to people in experiments and what it claimed to have found that was then used to maim people in the general population. Research lobotomy if you have any doubts at all about that statement. Some fraud got a Nobel prize for research that he used to advocate "safe lobotomy with no adverse affects." (But maybe he fooled himself and it wasn't blatant fraud. Bah.)

These are the screams from the psychology establishment as they get called out for claiming so much that is just totally false. It's really hard to know something. There is an immense amount of work you have to do and you all you might find after decades is "dead end". Not in Psych research papers and they absolutely deserve the contempt they get for it.

jkldotio• 10 years ago

The Bayesian take on this issue is interesting.[0] In their analysis of 72 replications only 8 had evidence pointing in the other direction and with a strong Bayes Factor.

[0] http://alexanderetz.com/2016/02/26/the-bayesian-rpp-take-2/ and http://journals.plos.org/plosone/article?id=10.1371/journal....

AstralStorm2• 10 years ago

The real question is whether the research commits the so-called type 3 error: provides a correct answer to the different question than posed. This might be very common in psychology.

Typical example would be premature generalisation of results.

Psychology study that induced "reproducibility crisis" was wrong: researchers

Related Stories