Not Frequentist Enough

https://statmodeling.stat.columbia.edu/2022/10/05/not-frequentist-enough/

by luu • 4 years ago

89 53 4 years ago

statmodeling.stat.columbia.edu

hammock• 4 years ago

A lot of words to say something that essentially boils down to, "Make sure your findings replicate."

Most people care about their findings replicating. The hard part is knowing what you need to do to improve your chances.

hackerlight2• 4 years ago

More like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothesis by chance, but that should only happen 1 in 20 times at an alpha of 0.05. With all the bad practices that alter test power, such as p-hacking and the file drawer effect, that 1 in 20 blows up to 1 in 2.

hackandthink4• 4 years ago

Gelman makes a distinction between p-hacking and choosing the test based on your data.

p-hacking is confirming your preferred hypothesis - just try again.

the latter let's you write a paper.

(test power is different: if there's really an effect, would you detect it)

jldugger• 4 years ago

In principle the robustness approach (redoing the analysis with some different parameters) tries to show good faith. This matters because observational studies have no meaningful replication, no second collection of data. In practice it just means practitioners gravitate towards methods more likely to show false positives.

Like, this discontinuity at zero approach seems geared to show large effects from noise. When the linear regression across the dataset is a flat line, splitting at the discontinuity pretty much mathematically requires _any_ slope have an opposite slope on the other side.

civilized• 4 years ago

As if that was a thing you could just do? "Oh, forgot to make sure my findings replicated. I'll be sure to check that box next time, easy enough!"

hackandthink4• 4 years ago

A famous Bayesian arguing for frequentist statistics?

Gelman tries to steal the concept "Frequentism" from simple minded frequentist statisticians.

His argument seems to be:

Simple minded frequentists statisticians perform a statistical procedure once - they do not think about performing the procedure many times.

They fall into this trap (from Gelman's paper):

"3. Researcher degrees of freedom without fishing: computing a single test based on the data, but in an environment where a different test would have been performed given different data"

rwilson43• 4 years ago

Gelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.)

Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer.

People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!

hackandthink4• 4 years ago

yes

https://stats.stackexchange.com/questions/115157/what-are-po...

kgwgk11• 4 years ago

running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do

Is looking at probability distributions “a pretty frequentist thing to do”? Even when those models and simulations include _prior_ probability distributions? Sure, one can (re)define frequentist to include Bayesian models - as Gelman seems to want to do in that post. I just don’t see how this helps to clarify anything.

whatshisface8• 4 years ago

In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer.

Bayesian and frequentist methods always give the same answer because they represent two different ways of translating the same mathematical ideas into English.

AstralStorm• 4 years ago

The frequentist test for this attempts to see what would happen with a variety of test designs using likelihood ratio and similar statistical tests. Relaxing it you end up with Generalized Method of Moments family.

A Bayesian would attempt to compute the Bayes factor using approximate Bayesian computation resulting in more or less the same thing. You end up with various information criteria.

Both approaches then converge in using Monte Carlo techniques to evaluate the features of the whole experimental setup using simulated data.

All of the above approaches replace the problem of choosing the test/design based on data by the researcher with one by a data driven algorithm with known properties.

youainti• 4 years ago

My understanding of the post is that given a small actual effect size, for a fixed experiment, you are more likely to get a significant p-value on a "large" measured effect size.

konschubert2• 4 years ago

When studying statistics, I never understood the apparent conflict between frequentism and bayesianism.

To me, they seemed like complementary tools, each with their own strengths and weaknesses.

You can use one or the other depending on what your goal is. Do you want to figure out how much you should believe something?

Or are you trying to figure out how compatible a hypothesis is with reality? In the first case, go bayesian, in the second, frequentist.

kgwgk11• 4 years ago

Nothing prevents you from "going Bayesian" in figuring out how compatible a hypothesis is with reality. Bayesians have no issues with probabilities representing frequencies even though frequentists cannot understand probabilities representing uncertainty.

jxy• 4 years ago

Seriously? Frequentists simply build infinite amount of parallel Universes and ask what percentage of the Universes It occurs with certainty.

hackandthink4• 4 years ago

If you want to know: do Higgs Bosons exist, you can go frequentist or bayesian:

Wasserman: p-value is fine for Higgs experiment:

https://normaldeviate.wordpress.com/2012/07/11/the-higgs-bos...

Tom Campbell-Ricketts: Bayes is better even for Higgs experiment

https://maximum-entropy-blog.blogspot.com/2012/07/higgs-boso...

My take: frequentist p-value is simpler but Bayes is what you really want.

rwilson43• 4 years ago

There is less of a conflict than many would have you believe. In many situations, both approaches yield the same answer. There are some edge cases. For example, in A/B testing, is early peeking bad? From a frequentist perspective the answer is "yes, either use a sequential method, or don't early peek at all". From a Bayesian perspective the answer is "early peeking is fine".

It boils down to what properties you want your analysis to have. Cox and Hinkley's "Theoretical Statistics" has a great discussion (section 2.4). Basically, you might want your analysis to have a certain kind of internal consistency. But you might also want your analysis to be replicable either by yourself or by another researcher. Those both seem like pretty important things! But there are edge cases (like the early peeking example) where you can't have it both ways. So you have to pick which one you want, and use the corresponding methods.

zozbot234• 4 years ago

The likelihood principle actually supports the Bayesian perspective on these issues of experiment design, and is regarded as foundational by many frequentists.

timy2shoes• 4 years ago

From a Bayesian perspective the answer is “early peeking is fine”.

That’s definitely not true. Early peeking increases your type I error rate, even with Bayesian methods. For a full explanation see http://varianceexplained.org/r/bayesian-ab-testing/

juped• 4 years ago

I think the "conflict" is basically just a matter of Jaynes having used the word "frequentist" like a hardcore Calvinist uses the word "Arminian".

BeetleB• 4 years ago

When studying statistics, I never understood the apparent conflict between frequentism and bayesianism.

I remember a statistician once saying: There are two types of statisticians: Those that are Bayesian and those that are both Bayesian and frequentist.

pdonis• 4 years ago

On the view of at least some Bayesians, "frequentist" is just the special case of "Bayesian" that you get when you are computing credences based on a large number of identically prepared, independent trials over a known sample space. So on this view (with which I tend to agree), the two are certainly not incompatible.

It is of course possible to do both frequentist and Bayesian statistics badly. I would say bad frequentism comes when one fails to realize that standard frequentist methods tell you the probability of the data given a hypothesis, when what you really need to know is the probability of the hypothesis given the data. Bayesianism at least starts right out with the latter approach, so it avoids the former (unforfunately all too common) error.

Bad Bayesianism, OTOH, I would say comes when one fails to realize that Bayes' rule is not a drop-in replacement for your brain. You still need to exercise judgment and common sense, and you still need to make an honest evaluation of the information you have. You can't just blindly plug numbers into Bayes' rule and expect to get useful answers.

grayclhn• 4 years ago

IME “bad Bayesian analysis” is when people use plausibly defensible priors to manufacture outcomes… which happens all the fucking time.

credit_guy• 4 years ago

There's no conflict. Nowadays whoever takes some courses in statistics will be exposed to Bayesian statistics. The universal reaction when exposed to Bayesian statistics is to fall in love with it, to think it's the answer to all of the world's problems (with the possible exception of war and hunger). But most people get to the point where they realize that Bayesian statistics is just a set of tools. Frequentist statistics can often times get you essentially the same results, with less effort.

And by the way, what do you call MLE (maximum likelihood estimation)? Is it frequentist? Because it looks awfully close to Bayesian.

kgwgk11• 4 years ago

MLE is usually considered frequentist. See for example the table in https://en.wikipedia.org/wiki/Frequentist_inference

Bayesian inference uses the likelihood function but I wouldn't say that this fact alone makes MLE "close to Bayesian".

MLE neither considers a (prior) probability distribution nor produces a (posterior) probability distribution for the parameter of interest.

tpoacher2• 4 years ago

There's nothing stopping the definition of a p-value from incorporating bayesian priors.

There. I said it.

mgraczyk• 4 years ago

P values already incorporate assumptions about the distribution under the null hypothesis, so it's not such a stretch to report bayesian statistics

whatshisface8• 4 years ago

There's nothing stopping the Bayesian posterior from being the degree to which you can reject the null hypothesis except for its definition. There's nothing except their definition stopping real numbers from including i, too.

tpoacher2• 4 years ago

No, that's redefining.

The p-value's ACTUAL definition, as accepted today in the statistical sense, can incorporate bayesian priors easily by making the model-data compatibility score upon which it acts compatible with Bayesian priors. Such as, e.g., a posterior probability.

Not Frequentist Enough

Related Stories