Thomas Bayes and the crisis in science

https://www.the-tls.co.uk/articles/public/thomas-bayes-science-crisis/

by agonz253 • 8 years ago

105 62 8 years ago

the-tls.co.uk

It's amusing that the people who are most militantly Bayesian aren't Bayesian statisticians. It's almost as if there are advantages and drawbacks to the Bayesian perspective. By the way, frequentism vs. Bayesianism has very little to do with Thomas Bayes. All statisticians accept the validity of Bayes' Theorem. Moreover, the two approaches are not mutually exclusive. It's always interesting to investigate the frequentist properties of Bayesian estimators and the implicit priors of frequentist estimators.

nkozyra• 8 years ago

That seems like a vast generalization across the board.

RobertRoberts6• 8 years ago

I don't mean to segue here, but I got caught in a logical argument that ended abruptly with no definitive answer.

I mentioned the Monty Hall problem to a friend (an engineer) and discussed the statical analysis done on this issue. (ie, statistically it's better to switch doors after the first is opened, as it improves your odds of winning slightly)

But he only answered with "nope, Bayes' Theorem says the odds are the same no matter what you do". The only other point he added was that if you do something only once, you have the same odds every single time.

I found this frustrating simplistic, because we _know_ through testing that this isn't true. That the odds are better if you switch doors.

Is this the fanaticism of Bayesians? He even called me a "frequentist" as if it were some kind of pejorative.

I've researched this issue, and even mentioned to him his logical flaw (you choose twice, not once) and still "Bayes theorum says there's no difference and you're wrong because you are a frequentist"

sigh.

LeanderK2• 8 years ago

I am not really sure what this has to do with bayesian vs. frequentists?

the bayesian theorem is a mathematical fact. It's true if we just accept some basic axioms. Bayesian/Frequentist divide is also pretty independent from the problem, since it's purely math-based? We are not estimating something...In fact, an easy solution to the monty-hall problem is to just apply the base theorem and calculate the probabilities.

jkabrg4• 8 years ago

Does this comment deserve a downvote? I think he's asking a genuine question.

selimthegrim• 8 years ago

He’s wrong, you can show switching doors improves your performance with Bayes’ theorem.

rvern2• 8 years ago

Bayes’ theorem says no such thing. Your friend is wrong, but please do not let this reflect badly on the Bayesian interpretation of probability in general—it has nothing to do with this.

AlexCoventry4• 8 years ago

The easiest way to convince your friend of his error is to imagine the odds when there are n doors, and Monty opens n-2 of them, the ones which don't contain the prize and aren't your first pick.

md224• 8 years ago

Here's how I would explain the Monty Hall problem:

First, get the person to realize that switching doors will always flip your initial outcome. If you picked a losing door, switching gives you the winning one. If you picked the winning door, switching gives you a losing one. There is no winning-to-winning or losing-to-losing. If you switch, you will always get the opposite of whatever you originally picked.

Once this idea has been understood, ask them to consider the probability that your original choice was a losing door. Once they agree that it's 2/3 (2 losing doors out of 3 possible choices), you can reply "that's right, and if there's a 2/3 chance that your current choice is a losing door, then there's a 2/3 chance that you're in a situation where switching will give you the winning door. Therefore, the chance that switching will give you the winning door is 2/3!"

Bromskloss5• 8 years ago

"frequentist" as if it were some kind of pejorative.

Ah, so he was right about that, at least! :-)

eli_gottlieb6• 8 years ago

Judea Pearl uses the Monty Hall problem in his latest book, and when he draws out the causal network, the issue becomes obvious. The game-show host chooses which door to open based on two factors: he has to choose a door you didn't, and he has to choose a door with a goat.

Hence, the door he chooses to open is probabilistically informative about your initial choice. Treating the door he opened as information on which to condition does change the posterior.

jhbadger• 8 years ago

I think the that the virulent all-or-nothing Bayesian vs. frequentist debates are really more from earlier decades. These days most people are open to using techniques from both camps and are more interested in solving problems than being philosophically "pure".

trentmb• 8 years ago

Not gonna lie- I don't think I understand the difference. Every explanation seems to bounce off and over my thick skull.

Bromskloss5• 8 years ago

Moreover, the two approaches are not mutually exclusive.

I would say that they are based on two different view of what probability is, and that subscribing to one means that you cannot accept the other.

rossdavidh• 8 years ago

On the one hand, I have (in the semiconductor manufacturing industry) encountered statisticians who were greatly averse to making any prior assumptions about the likelihood of something. Also, any article with the phrase "nonsense on stilts" in it is not entirely unworthy. It also does a credible job of explaining Bayes' theorem.

On the other hand, I think it somewhat exaggerates the extent to which Bayes' theorem and its associated work are rejected by statisticians, and also the extent to which a magical p > 0.05 limit was advocated. I believe it was non-statisticians who wanted easy tests that they could apply without understanding much about statistics, who were to blame in that case.

eli_gottlieb6• 8 years ago

Statistics is formalized philosophy of science, so of course scientists outside the stats departments wanted "easy tests". Arguably, a lot of the new applications of computational Bayesian methods in the natural science are the same statistical witch-doctoring with a bit more rigor.

(Ok, I do computational Bayes for a living with natural scientists, so I'm inherently biased to view our work as shakier than it really is.)

Karrot_Kream• 8 years ago

Aside: What do you do for a living?

sewercake• 8 years ago

'Statistics is formalized philosophy of science' -- care to expand? Or provide supplementary literature?

analog312• 8 years ago

A statistician could talk me out of this, but I've always been puzzled by the use of "prior." Nothing in Bayes's Theorem says that the prior has to be established before anything else. It makes Bayesian statistics seem like a matter of letting your expectations influence your results.

Instead, an interpretation that seems more favorable to me, is that you consider all of the information at your disposal, that can be brought to bear on a problem, and this could include known constraints on likelihoods. Bayes's Theorem becomes a tool for working through problems where a single statistic can't be readily used to analyze an entire data set, e.g., when data come from disparate sources and can't be readily combined.

lgas• 8 years ago

I'm not a statistician but as I understand it, in Bayesian terms, when you get some new piece of information you must update your beliefs to incorporate the new information. So you take your belief prior to the new information and incorporate the new information to get your updated belief.

hooloovoo_zoo3• 8 years ago

It sounds like you're talking about just using the maximum likelihood estimator (with a special likelihood) which is similar to using a flat (improper) prior. One disadvantage of doing so is that the results aren't invariant to reparameterization of the problem. The latter problem can be fixed by using Jeffreys' priors. However, the MLE is very effective in its own right.

sjg007• 8 years ago

A prior could be a population level statistic which is then updated by a likelihood to reveal a posterior probability. Take for example a medical test. We know that on average that 1/10 people have some disease X. We have a test that tests for X (with some false positive rate).. Using that we can calculate the posterior probability that you actually have disease X. If you relied just on the test you don't discount the population level statistics which might be relevant. You can also use the tests FP rate in the same way as your prior... e.g. the people who test positive who don't have the disease etc...

sl8r3• 8 years ago

Part of the issue is that you can't not assume a prior; it's unavoidable. The Bayesian POV just makes this assumption clear / explicit, while many frequentist methods (if naively applied) amount to choosing a uniform prior.

E.g., imagine you have an e-commerce site that has, historically, had a 2% conversion rate (landing page to purchase). Now you run an A/B test with two variants, a control (A) and a treatment (B). Both buckets get 10,000 landings, of which A converts 200 of them and B converts 250. How can you tell if A is better than B? Cutting to the chase, the frequentist approach (if applied naively) would be to model B as some distribution centered around 2.5%, for example N(2.5%, 0.15%) or Beta(251, 9751).

A Bayesian would say that this assumes a uniform prior - but that this is probably a bad prior because it ignores what we know about the historical conversion rate of 2.0%. Said another way, the above amounts to saying (before we run the test) that we think it's just as likely for B to have a conversion rate of 2.5% as it is to have a conversion rate of 100%. Clearly we don't actually believe this.

valgor• 8 years ago

It makes Bayesian statistics seem like a matter of letting your expectations influence your results.

The smart approach is to have a set of a prior expectations, run through the data, and see what comes out the other side. Essentially, through a large amount of experimentations, you can narrow down what assumptions were indeed correct.

gweinberg• 8 years ago

There's probably some truth to this. But if statisticians weren't able to convince the academic world that arbitrarily setting a "statistical significance" threshold is a gross misuse of statistics, then what good are they? It would be like MDs failing to convince the rest of the medical community that bloodletting isn't a universal cure for whatever the hell ails you.

smallnamespace2• 8 years ago

Responsibility in communication is a two-way street, and also depends on incentives and relative power.

- Are security researchers grossly incompetent for failing to convince the IT industry to adopt best practices?

- Are voters in democracies grossly incompetent for failing to police their politicians and keep them honest?

- Are media outlets grossly incompetent for not squeezing out 'fake news', whatever your definition?

- Are epidemiologists and public health officials grossly incompetent for not convincing anti-vaxxers of their foolishness?

Anytime there is a breakdown in communications, both the listener and the speaker should probably shoulder some of it.

Bromskloss5• 8 years ago

statisticians weren't able to convince the academic world that arbitrarily setting a "statistical significance" threshold is a gross misuse of statistics

What would the proper use of statistics be? I believe it is to produce a Bayesian posterior probability distribution, but that wouldn't be the opinion of a frequentist.

wirrbel• 8 years ago

I believe it was non-statisticians who wanted easy tests that they could apply without understanding much about statistics, who were to blame in that case.

Yes and no. I think most journal articles feature a copy-and-paste approach to statistics, reusing what other papers do, or worse, tinkering with test functions in excel. Yet, this seems to be ok with statisticians it seems.

From my personal experience applying statistics in the software world (and trying to build alerting models for monitoring applications, etc.) is, that since we have a lot of systems that continuously record date (instead of the way scientific studies usually work, collecting a set of data points, N=x), we get cross-validation of our results, which has drastically made me aware of the pitfalls of frequentist statistics.

eli_gottlieb6• 8 years ago

We are living in new Bayesian age. Applications of Bayesian probability are taking over our lives. Doctors, lawyers, engineers and financiers use computerized Bayesian networks to aid their decision-making. Psychologists and neuroscientists explore the Bayesian workings of our brains. Statisticians increasingly rely on Bayesian logic. Even our email spam filters work on Bayesian principles.

While that's true, strictly speaking, it's akin to writing that, "We are living in a new logical age" when Boolean algebra was first finding wide application in engineering and the natural sciences. "Bayesian methods" just mean using statistical modeling techniques that conform to Bayes' rule as their normative guide. In "machine learning" or "frequentist" terms, it just means that good approximate-Bayesian reasoning minimizes the KL divergence between the true posterior and the approximate model (whether variational or by sampling or by training a neural network, whatever), as opposed to minimizing the classification hinge-loss or the mean squared error (though some of those losses have formally equivalent Bayesian priors).

plassma• 8 years ago

Strictly speaking, health care and genocide are the same thing, you just minimize different metrics.

gfodor• 8 years ago

Is there a metholodgy for bayesian analysis that avoids the choice of a specific prior but instead provides conclusions in the form of boundary/regions in "prior-space" and their effect on belief? For example, it would be incredibly useful if the output of research allowed a reader to gauge support of the conclusion in a minimally subjective way by explaining what effect choices in prior have on results. I'm not a statistician so I'm assuming this is a well understood thing, but would be curious to know if and how it is practiced.

hooloovoo_zoo3• 8 years ago

Generally speaking, a good Bayesian analysis includes what's known as a "sensitivity analysis" which seeks to measure how sensitive the results are to the particular choice of prior. Additionally, if strong prior assumptions are not available, an "uninformative" prior is used. In such cases, the results tend to be pretty close to those from frequentist methods, except the frequentist methods lack the Bayesian probabilistic interpretation.

Bromskloss5• 8 years ago

The final result of a Bayesian analysis is the posterior probability, which is the product of the prior probability (which can incorporate previous knowledge or convictions) and the likelihood (which is determined by the data you just collected). You can leave out the prior, and just report the likelihood. Another comment mentions this: https://news.ycombinator.com/item?id=17438643

Eliezer• 8 years ago

For a more detailed take advocating the particular solution "report likelihoods, not posteriors or p-values", see "Likelihoods, p-values, and the replication crisis": https://arbital.com/p/likelihoods_not_pvalues/?l=4xx

AlexCoventry4• 8 years ago

You can abuse Bayesian methods just as easily you can hack a p-value. Maybe more easily, since fewer people would be aware of the issues.

What's needed is a shift in researcher attitudes and incentives, to emphasize development of reliable knowledge instead of publication record. Just changing the rules of the game slightly will only lead to people adjusting their game slightly.

http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

outlace• 8 years ago

Yes but with traditional frequentist approaches most scientists don’t understand what’s going on under the hood of the statistical tools they’re using. There are dozens and dozens of named tests like “Fisher’s exact test” and “Wilcoxon signed-rank test” and most scientists just sort of follow received wisdom about which test to use in which situation.

Bayesian methods force you to actually think about how your data and model parameters are distributed, and explicitly specify a model.

eli_gottlieb6• 8 years ago

Bayesian methods force you to actually think about how your data and model parameters are distributed, and explicitly specify a model.

Yep. I know some very senior scientists who get a lot of mileage out of finding places where "model-free" methods are implicitly assuming a bloody stupid model, and then attacking them.

Thomas Bayes and the crisis in science

Related Stories