Garden of forking paths: Why multiple comparisons can be a problem (2013) [pdf]

http://www.stat.columbia.edu/~gelman/research/unpublished/forking.pdf

by luu • 4 years ago

40 9 4 years ago

stat.columbia.edu

smegsicle• 4 years ago

tl;dr a problem in research data analysis, not computer science or branch prediction or anything

uniqueuid3• 4 years ago

It's actually a very common problem in e.g. AI research, because your choices of hyperparameters, folds and training material may unconsciously or intentionally lead to overestimation of performance.

rwilson4• 4 years ago

There are some great technical solutions offered, but the real one is to publish your data and your code, and be as transparent as you can about what you did.

conformist• 4 years ago

Yes, that helps. But it’s not quite sufficient, because transparency does not actually adjust the p-value of a study correctly (or makes it easy to do so). For this approach to work, you would have to sort of preregister all the code, too.

uniqueuid3• 4 years ago

The by far best way to handle forking paths are of course pre-registrations (where you formally commit to a plan of analysis before performing the research).

But for ex post diagnosis, specification curves are also awesome: https://www.nature.com/articles/s41562-020-0912-z

origin_path2• 4 years ago

Pre-registration is becoming more common but unfortunately, it isn't enough. For it to work there's an implied second stage: journals and other funding gatekeepers have to commit to blocking research if it doesn't actually implement the pre-registered plan. This does not seem to be happening reliably, or at least it's relatively easy to find counter-examples.

There's also a lot of scope for drama and problems at that second stage, because there can be a lot of wiggle room in whether what was done meets or doesn't meet the pre-registered plan. It might sound easy to follow a pre-written plan but for instance if what you're doing turns out to need adjustment due to discovering something you hadn't anticipated, then technically you might violate your pre-registration. In turn that can yield perverse incentives to not fix things.

Having been through a few rounds of "here's a new procedure that will yield better science, oh no it isn't really working" I'm not really enthused anymore by this kind of tactical fix. From a software dev perspective it's sort of like constantly hot-patching a system in prod to try and cover up bugs when the underlying problem is actually bad architecture that creates too many bugs in the first place. Getting overly fixated on P-hacking or non-replicability distracts from the deeper underlying problem of a mis-specified utility function.

Example: replication was meant to be the big hammer but most studies still aren't being replicated and when they are they sometimes use lax definitions of "replicate" that wasn't what people had in mind, e.g. they'll get an effect size 10% of what was originally reported and still label that study "replicated" or they'll get totally different numbers but claim that because the trend is sort of similar, that's replication.

uniqueuid3• 4 years ago

(I'm answering here even though the story has already dropped, because I think you raise a very important point.)

All that is true, and we should realize that there are no silver bullets anywhere. Cohen/Gigerenzer's painful papers on power were some 40 years apart, yet nothing happened in between. There needs to be a systemic answer, you're completely right about that.

But then, It's important to consider what the assumed model of change is. I believe deeply that influential individual cases can often push progress more than gradual toiling. Having a replication crisis with Daniel Kahneman writing "fucking solve it" emails definitely helps. That's because progress here is a ratchet (new methodological standards set a new lower bound). And more and more people realize that your research today is what you may be judged on tomorrow. So it's a dangerous game to intentionally deceive.

In the end, science needs to be eventually correct in finite time. The steeper the slope, the better. The one thing we must not allow is for the slope to become negative.

netsharc• 4 years ago

The full Borges short story referenced by the title: https://genius.com/Jorge-luis-borges-the-garden-of-forking-p...

Garden of forking paths: Why multiple comparisons can be a problem (2013) [pdf]

Related Stories