Frequentist Statistics are Frequently Subjective

http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/

by yummyfajitas • 17 years ago

55 28 17 years ago

smanek3• 17 years ago

To my (limited) understanding, Bayesian statistics are just as subjective. If you and I start with different priors, and are then are fed the same evidence, we can end up with vastly different conclusions.

Since priors are defined as a function of my state of mind (or state of ignorance, as it were), it seems pretty subjective that the 'result' should depend so heavily on my initial state of mind.

E.T. Jaynes' book has a example of this with regard to E.S.P. research (I believe http://omega.albany.edu:8008/ETJ-PS/cc5d.ps is the relevant chapter - but it's been a year or two since I've read the book).

A real statistician should feel free to correct me though - I'm more of an algebra guy ...

mnemonicsloth• 17 years ago

To my (limited) understanding, Bayesian statistics are just as subjective.

I'm still reading, but that seems to be the point. As Yudkowsky puts it, "probability is ignorance and ignorance is a state of mind"

yummyfajitas• 17 years ago

The argument is that Bayesian statistics are less subjective because while we won't come to the same conclusion, we will perform the same update to our prior probabilities.

An example: consider a medical diagnostic test with a 1% false positive rate and no false negatives.

If I understand him correctly, Eliezer is arguing that the scientist (i.e. the lab running the test) should present an update rule, not the posterior probability. I.e., the lab report should say: "Whatever your prior estimate of P(sick) is, your posterior should now be P(sick)/(0.01 P(not sick) + P(sick))" instead of saying "the patient has an x% chance of being sick".

Suppose before the test I believe the odds the patient is sick are 1% and you think they are 10%. Once the test comes out positive, I've updated my probability estimate to 50%, you to 92%.

The update rule provided by Bayesian statistics is objective, even if the posterior is subjective. The frequentists have nothing comparable which is objective.

See also this link:

http://www.overcomingbias.com/2009/02/share-likelihood-ratio...

smanek3• 17 years ago

Ah, that makes sense - thanks.

jules• 17 years ago

The difference is that bayesians acknowledge this explicitly. You publish the way to calculate the posterior given someones prior so that everyone can fill in their own priors to determine their posteriors. This calculation is objective. With frequentist statistics everything is subjective, including the results you publish.

btilly4• 17 years ago

Your understanding is incomplete. Bayesian statistics are not subjective at all. Instead they objectively describe the correlation between prior belief and conclusion. People may draw different conclusions for subjective reasons, but that subjectivity is in the people, not the statistics.

Now it seems you are objecting to the fact that your initial state of mind affects your conclusion. But it is unavoidable that there is no way to make sense of observation except through the lens of prior beliefs. You can be explicit about it and draw conclusions with a correct methodology about it as in Bayesian statistics. You can be implicit about it and draw conclusions with an incorrect methodology as in frequentist statistics. (See my other post in this thread for an explanation of why frequentist methodology is incorrect.)

Let me offer a simple example. Suppose a pregnant woman you know gives birth to a boy. Is that evidence that babies are more likely to be boys than girls? Obviously it is. Is it strong evidence? Obviously not. Should we upon observing that conclude that boys are more likely than girls? Obviously not.

Now suppose that you have no prior knowledge other than we see about similar numbers of boys and girls. I know that of the last 100,000 babies born in the USA, 51,157 are boys. Suppose we are both told that at the local hospital, 92 of the last 200 babies born were boys. I submit that we both will and should wind up with different conclusions about the relative likelyhood of boys and girls. Why? Because different prior knowledge lead to different prior beliefs, and those prior beliefs when modified by identical evidence lead to different conclusions.

smanek3• 17 years ago

You have to admit that your example is pretty contrived though - usually setting priors isn't so clean.

How do you 'properly' set the prior probability that someone really has E.S.P.? That a researcher is secretly colluding with a test subject? That a researcher is flat out fabricating their results? That this is just an instance of the Hans effect? Or a thousand other possible hypotheses ...

frig2• 17 years ago

Say a proper Bayesian who's a fan of 'da Bears' just saw the bears lose again, badly. How is he supposed to update these two beliefs:

- this season 'da Bears' will tear shit up every Sunday (fwiw assume prior on this has it as likely)

- I am usually over-optimistic about 'da Bears''s performance (fwiw assume prior on this also has it as likely)

...in light of the new evidence?

(Side note: I'm specifically not talking about bayesian analysis or statistics in a formal setting with a prespecified, finite list of allowed hypotheses (like spam or ham).

If I wrote this out in extreme detail I'd get nothing done today, I think you can charitably fill in the gaps and missing pieces of what I'm getting at; if not I'll fill them out in a few.)

btilly4• 17 years ago

The real problem with statistics is that people want an answer to an impossible problem. Namely they want to be told the probability of the world being a particular way. But anyone familiar with conditional probability can easily see that there is no way to come up with that answer, because the conditional probability of something after an observation depends on the assumed probability before the observation.

There are multiple approaches to this problem. What frequentist statistics does is answers a different question. Namely, "What is the probability of getting a result this strongly against the null hypothesis if the null hypothesis is true." This is appealing in that it is an objective probability that seems to say something about the problem under discussion. However people consistently read it as, "What is the probability that the null hypothesis is true?" Which is simply wrong.

There is a second problem with frequentist statistics, which is what this article is about. Which is that the objective-looking probability you get depends on the null hypothesis chosen in ways it shouldn't. Basic familiarity with Bayes' Theorem and conditional probability demonstrates that it is impossible for it to matter whether your intention was to flip a coin 6 times, flip it until you get both heads and tails, or flip it until you get heads. This factor cannot affect the conditional probability of the coin being biased. But those three different intentions translate into 3 different calculations, and 3 different answers in frequentist statistics.

So that's the frequentist approach. What is the Bayesian approach? It is to come up with statistics that inform us on how prior beliefs before observation turn into posterior beliefs after observation. The advantage of this method is that it is intellectually honest. The disadvantages are that it is complicated and people notice that it is confusing. (The frequentist approach is confusing as well, but people don't notice their confusion. Instead the confidently draw the wrong conclusion that the null hypothesis has been proven.)

There are other approaches. The article touched on my favorite when it pointed out that we should report likelyhood ratios rather than probabilities. This is absolutely right. The effect of an observation is to modify our prior beliefs, and likelyhood ratios concisely describe how we should modify them. Plus they have the great ability to stack - you can take 3 experiments and combine their likelyhood ratios to come up with the likelyhood ratio for having seen all three experiments.

Unfortunately, though, everyone knows frequentist methods, people accept them, and it is very hard to get people to see what is wrong with them. So alternate approaches, though theoretically superior, face an uphill battle towards acceptance.

tel7• 17 years ago

The difficulty I tend to encounter in promoting Bayesian statistics among scientists is the "sudden" appearance of a statistical model. Too many people go one further than believing frequentist statistics are answering "What is the probability that H0 is true" but instead just fully over the analysis and believe that frequentist methods tell you, simply, accurately, objectively, whether an experimental treatment is "significant".

If you try to press on what people believe "significant" to mean it gets ugly fast, but it's generally a good thing and definitely necessary to publish. If you don't get significance it's just because you need a bigger n. If you can think of some factors or covariates then you really need to use ANOVA.

Stating that there is anything more complex to looking at data and deciding what it means is practically unthinkable.

Likelihood ratios are definitely nice, though. I've sort of gotten people to think about it at a high level by talking about "information flows" and log likelihood values.

Eliezer• 17 years ago

A separate problem, not dealt with in that particular essay, is the quite hideous degree to which average scientists don't understand the statistics they use.

I would put a good deal of the blame for this squarely on frequentism as well. Bayesianism isn't hard to understand, it's just takes an effort of the teacher to explain well - I've made certain notable efforts in that direction myself. Once you do get it, you get it.

hamilton3• 17 years ago

I think you're on to something regarding "significance." Over in my dept. we like to say that significance is a measure of sample size. The question, then, hinges on whether or not something has practical significance. Because we've built the whole research reputation incentive structure on the .05 significance level, studies can be designed to get that.

A great article on the problem with statistical significance (by a Frequentist) is here, called Why Most Published Research Findings Are False: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

The punchline seems to be, well, that there's also a large human element that contributes to the problem. I think it's one thing to rail on the Frequentist way-of-thinking; it's entirely another to state that the institution of scientific research built on it creates unwanted incentives.

gort• 17 years ago

To me, more interesting than Bayesianism versus Frequentism was the observation that scientific procedure as it currently stands allows a field like Parapsychology to survive.

mattmcknight• 17 years ago

I get lost in the argument with these simple coin flipping examples. The way it's presented is so confusing for the weird second case of how many trials it takes to get a tail.

In the first case, the probability of getting "five tails or more" from a fair coin is 11%, while in the second case, the probability of a fair coin requiring "at least five tails before seeing one heads" is 3%.

I didn't check the numbers here, but this seems perfectly reasonable to me in terms of hypothesis testing. He's playing on the fact that for cases like THXXXX you would get a result of 2 in the second experiment. You are taking the same results, but applying a different metric to them (position of first head versus count of tails). Of course p will be different.

I understand that the value chosen for significance of p is subjective, but the experiment itself makes perfect sense to me.

btilly4• 17 years ago

The problem is that the conditional probability of the coin being biased given the observation of TTTTTH absolutely does not and cannot depend one whether you intended to flip until you saw a heads, or were going to flip 6 times. Therefore you've got a subjective factor influencing your conclusion that probability theory says should not be a factor if you are reasoning consistently.

hamilton3• 17 years ago

This debate has been raging for 250 years. Brad Efron, one of the greatest living statisticians today and one of the few who can transcend the debate, has VERY interesting things to say about it. He believes that Empirical Bayes, or using some of your data to model your prior, will win the day.

http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode...

nova2• 17 years ago

> or using some of your data to model your prior

I think that kinda missing the point.

hamilton3• 17 years ago

I beg to differ (not sure if you actually read the Efron talk I just posted). I get the thrust of the Less Wrong article, though I think that frankly his language and vitriol is direly misplaced. He's railing against some class of Frequentists who attack Bayesians. This is sort of a Fox News tactic; pick someone, in this case, a commenter on a blog, and generalize their comment to the entire population. It's fairly unstatistical, if you ask me. I don't get why anyone would do this. A good Frequentist is not a lost idiot, nor does she rail against Bayesians. In fact, a good Frequentist understands the beauty of Bayesian methods, too. I'd even wager that a good Bayesian understands why Frequentist methods have dominated the 20th century.

I'm attempting to realize that rare event in these vitriolic philosophical academic debates - add a bit of middle-groundness. In this case, the middle ground of Empirical Bayes has proven to be very, very useful in large-scale simultaneous inference problems. It's proof that there IS such thing as convergence in this debate. Taking the shrill tone the author takes as a signal of how little regard the two sides have for each other, it's obvious that we need some convergence.

tel7• 17 years ago

Hamilton, I'd like to email you about your work at Stanford if you don't mind. Could you drop me a line?

Frequentist Statistics are Frequently Subjective

Related Stories