Paradoxes of probability and other statistical strangeness

https://theconversation.com/paradoxes-of-probability-and-other-statistical-strangeness-74440

by seycombi • 9 years ago

132 106 9 years ago

theconversation.com

pmoriarty2• 9 years ago

My favorite probability paradox has always been the Monty Hall problem[1]:

Suppose you're on a game show, and you're given the choice of three doors:

Behind one door is a car; behind the others, goats.

You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat.

He then says to you, "Do you want to pick door No. 2?"

Is it to your advantage to switch your choice?

[1] https://en.wikipedia.org/wiki/Monty_Hall_problem

haddr3• 9 years ago

I love this one, but I guess it must be less common to have such a problem in the real life. Or I'm wrong?

justinpombrio2• 9 years ago

As the Wikipedia article points out, that problem as stated is ambiguous. You need to know under what circumstances the host would have opened that door. The standard assumption is that the host always reveals a door with a goat behind it.

However, another possibility is that: (i) if both of the remaining doors contain goats then the host reveals one of them; but (ii) if one of the remaining doors contains the car then the host doesn't open a door at all.

kharms• 9 years ago

That variation doesn't make much sense as a paradox. If (i) you know that you should stay 100% success. If (ii) you know you should switch, 50% success.

casta2• 9 years ago

If you see this problem as a special case of another problem the answer becomes pretty intuitive.

Suppose there are N (let's pick 1000) doors: Behind one door is a car; behind the others N-1 (999), goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens N-2 (998) doors, excluding No. 1, which have have goats.

Now there are only two doors closed, the one you picked No. 1, and one that was part of N-1 (999) doors. He then says to you, "Do you want to pick door No. 1 or the other one?" Is it to your advantage to switch your choice?

std_throwaway• 9 years ago

Suppose there's two doors. You can't decide. The host opens one. There is no goat behind that one. Now you know for certain.

p1esk6• 9 years ago

It did not get any more intuitive for me.

kaoD2• 9 years ago

It didn't get any more intuitive for me like that. If anything, less.

When it clicked for me is when I thought of it as being 1/3 probability the car is behind your door, and 2/3 it isn't. That doesn't change whether they show you one of the other two doors or not, so switching to the 2/3 option is better.

gerdesj• 9 years ago

You are only a small step away from "assume a spherical cow" or something involving Hilbert's Hotel.

Which part of three (doors) are you having snags with?

YCode• 9 years ago

This one is uniquely impressive in that often when first described to someone they will be all but certain it doesn't matter.

Many I told were just about ready to argue their view to death and asserted the switch your choice conclusion is wrong despite them having no formal learning on the subject.

This and the birthday problem effectively illustrate how intuitively bad we are at probability and statistics in general.

j7ake• 9 years ago

Properly applying Bayesian inference gives you the correct answer.

Also the answer becomes intuitive once this scales to more than 3 doors (someone below already mentioned).

denzil_correa• 9 years ago

The best explanation on the solution for this problem is from Numberphile.

https://www.youtube.com/watch?v=4Lb-6rxZxx0

pmoriarty2• 9 years ago

One of the things that interests me about this paradox is that when you're left with just two doors, the odds of either of them being the one with the car is 50%, so what does it matter if the one you happened to pick was picked before the host opened some other doors? How could that possibly affect the probability? It seems like choosing to switch would be admitting that the host's action has some kind of reverse-causality.

The other thing I find interesting about this puzzle is wondering whether the people who find the arguments in favor of switching convincing are actually reasoning better than I am. If so, how are they doing this? What is it that makes those (allegedly superior) arguments convincing to them?

The thing that kind of convinced me that the pro-switch arguments must be correct is that people have run simulations of this problem on computers and the switching strategy turns out to win in line with the pro-switch argument probabilities. But the tricky thing about probabilities and randomness is that such successes could be "just a matter of luck", as it were. Maybe they were wrong, but just got lucky. It's improbable, but still possible.

Only mathematical proof independent of experimentation should be rock-solid correct. But mathematical proof seems to be a matter of being convincing enough to mathematicians, and no so independent after all, and that seems disappointing somehow. There's a feeling that mathematical proof should be above mere argumentative skill and the audience's susceptibility to being convinced.

rndmio• 9 years ago

One of the things that interests me about this paradox is that when you're left with just two doors, the odds of either of them being the one with the car is 50%.

Well, no, that's the point of this, it's not 50%. Before you pick there is a 1/3 chance of the car being behind any of the doors, so there is a 2/3 chance of it cumulatively being behind the other two. But there is only one car, so the set of doors you didn't pick must include at least one non-car. The host knows the location of the car so can always open a non-car door from this set. But still as a set they have a 2/3 chance of containing the car. By swapping you can effectively think of it as choosing both other doors. An easy way to get your head around this is increase the number of doors, say to 100. You pick 1 of those 100, the host opens 98 other doors, do you swap?

kevhito• 9 years ago

Alternate reality rules of the game are: nobody speaks, you pick any two doors you like, and you win the prize behind both of them. The odds of wining this game is obviously 2/3. Monty will gesture at you to point to a door. You point at the one you don't like. Monty will then show you one of your prizes behind one of your two chosen doors. Monty will then gesture for you to point to a door. You point at the remaining one you liked. Monty then gives you your second prize behind that door.

Notice that all of the externally visible actions are identical to the original game.

babyrainbow2• 9 years ago

You pick a door, say No. 1, and the host, who knows what's behind the doors...

I don't get this. Why does't the player insist that they open door no 1. Or is it part of the game that the host will open a different door other than the one the player picked...

If the host knows what is behind the doors, and you are only given two chances, and the host can open a different door from what you picked, isn't your chance of winning zero?

I don't get this...

Retra2• 9 years ago

The host's job is to create suspense by opening a door and offering a chance to switch. If they opened the door you picked first, there would be no suspense, and if they didn't know what was behind the door, they could show you that you lose, which also removes suspense.

Turns out the host just ends up giving you more information that you can use to make a better decision.

z3t42• 9 years ago

When we discussed this in school I quickly wrote a program that simulated it ...

danbruc14• 9 years ago

No need to look at fancy paradoxes, just think about the following.

What does it mean that tossing a fair coin has a 50 % probability of showing heads?

If you think you know the answer, you are probably wrong.

EDIT: Instead of just voting this down, try to give an answer. If you think it is easy, you have not thought about it careful enough.

yellowstuff4• 9 years ago

The frequentist answer is that you could get the proportion of tosses that come up heads arbitrarily close to 50% in a large enough sample.

The Bayesian answer is that given all the evidence available to me I would be willing to bet $1 to win $1 if the next toss comes up heads. (Since I work in finance I'll add that this assumes I'm risk neutral, so losing $1 is exactly as bad for me as winning $1 is good.)

Except for the risk-neutrality detail this is all Probability 101, right? Or are you thinking of something else?

danbruc14• 9 years ago

That the relative frequency converges to 50 % is obviously not true. You would have to be exceptionally lucky, but you could get heads, heads, tails repeated for ever and therefore the relative frequency of heads would fluctuate increasingly tiny amounts around 66.(6) %. This of course has probability zero, but it is not impossible. And there are of course many other sequences of outcomes for which the relative frequency does not converge to 50 %. So at best you could say that the relative frequency converges to 50 % with high probability, but now you have a circular definition because you make use of probabilities while defining probabilities.

The Bayesian view is problematic for several reasons. Why do I need someone with believes about the coin, we are talking about intrinsic properties of tossing a coin after all. And if that is not enough, we also throw some betting in. Tossing a coin does certainly not depends on the invention of money and gambling, at least ignoring that coins are usually money. Last but not least you have to explain where your believe about a 50 % probability for heads comes from and how it is justified. I could certainly believe that the probability for heads is 25 %, that would not be a good believe.

b_emery• 9 years ago

It means that there is a 100% chance that one side of the coin has a head on it.

danbruc14• 9 years ago

Well, that is already in the setup, there is also a 100 % chance that the other side is tails. I don't think that observation will turn out to be very useful.

llasram• 9 years ago

That we're modeling the outcome of a coin toss as a sample space set containing at least two event elements, one of which we call "heads," and that we have a probability measure which assigns 0.5 to the subset consisting of only the "heads" event.

danbruc14• 9 years ago

You are presupposing an understanding of probability, aren't you? What does it mean that a probability measure assigns 0.5 to heads?

criddell6• 9 years ago

If it's a fair coin, then by definition there's a 50% probability of showing heads.

danbruc14• 9 years ago

Yes, but what are the consequences of this definition? What does a probability of 50 % for heads imply about the coin? Or what is the difference to an unfair coin with a 40 % probability for heads and a 60 % probability for tails?

TekMol3• 9 years ago

You can play this game for every statement.

What does it mean that an elephant is bigger then a dog?

If you think you know the answer, you are probably wrong.

danbruc14• 9 years ago

That one is easy. First let's clarify what exactly we mean with bigger. Let's say we mean larger in volume. Now we pick a procedure for determining the volume of animals, say we submerge them briefly in a large water tank and mark the resulting water level. That animal that lead to the highest water level is the bigger one. Any objections?

naftaliharris2• 9 years ago

A less popular but perhaps more influential phenomenon is Stein's Paradox [1]. Here's a provocative example often given to illustrate it: Say you have a baseball player, soccer player, and football player, and you wish to estimate the true mean number of home runs, goals, and touchdowns each scores per year. If you have their last ten seasons worth of data for each, then the obvious thing to do, for each player, is to estimate the true yearly mean score for each player by their average yearly score from the last ten years. (E.g., the baseball player hits an average of 20 home runs each year, so let's estimate their true mean yearly home runs by 20). Stein's Paradox says that you can actually do a lot better than this.

Even more crazy, the James-Stein Estimator which does this actually uses data about the football player and soccer player to make predictions about the baseball player, (and vice-versa). This is deeply unintuitive to most people since the players aren't related to each other at all. The phenomenon only holds with at least three players; it doesn't work for two.

(More generally, Stein's Paradox is the fact that if you have p >= 3 independent Gaussians with a known variance, you can do better in estimating their p-dimensional mean than just using their sample means).

I've spent a bunch of time trying to understand why this actually works [2]; to be honest I still don't deeply understand. But nonetheless the consensus is that the same shrinkage phenomenon is what causes improved performance for a variety of high-dimensional estimators, (lasso or ridge regression, e.g.), making the paradox very very influential.

[1] https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator [2] https://www.naftaliharris.com/blog/steinviz/

projectorlochsa2• 9 years ago

Well this does make a little bit of sense if after estimating things get worse for a particular case.

If MSEs were 5, 5, 100, and after they are 10, 10, 80, the total got better but the prediction for the other two got worse.

mafuyu• 9 years ago

Yeah, reading through the Wikipedia, it looks like this reduces the total error of the combined estimator, but the error compared to an estimator of any one single parameter could be worse. So you can combine whatever crazy parameters you want, but it's only really relevant when you have things that are associated with each other somehow, and you want to reduce the total error of estimating all of them.

ryanmonroe3• 9 years ago

This is not why it works. The James-Stein estimator applies in the case of independently distributed Normal variables with equal variance, so the individually optimal estimators for each parameter have the same MSE.

tmyklebu• 9 years ago

Not sure I understand. Why should the number of home runs per year, the number of goals per year, and the number of touchdowns per year have equal variance?

naftaliharris2• 9 years ago

You probably wouldn't expect them to. But the same kind of Stein phenomenon holds under a much broader set of conditions, including arbitrary covariance matrices and arbitrary quadratic loss, (see e.g. [1]). It's a very general phenomenon!

[1] https://projecteuclid.org/euclid.aos/1176345691

terminado• 9 years ago

I'm going to go out on a limb with my intuition, and hypothesize that the underlying premise of the method zeroes in on a common characteristic of games humans tend to enjoy.

A universe of people wouldn't find these games generally interesting, if they didn't present outcomes above a certain threshold of unexpectedness. The underlying rules of each game are tuned into the equipment used, and a balance is reached, where game play is fair, but still requires players to develop skills.

Because each sport adheres to the premise of capturing interest in players and spectators, they all present the same scoring tendencies, when aggregating and generalizing.

If you change the motive of the activity (mix games with non-games), and the artifact that represents success (mix freely tallied points with rare physical tokens or discovered evidence), so that the behaviors being compared are dissimilar, the predictions will become unreliable.

For example, when comparing "victories" across lawyers, geological prospectors, and sports players you probably would not be able to make predictions about all, by lumping each area's statistics together. A gold-mining prospector probably wouldn't encounter success in the same way a trial lawyer would, and neither would help you predict or generalize a hockey game.

But, an oil driller, a diamond prospector, and a gold prospector would likely compare, based on the geological goal sought. A forensic analyst, a private detective, and a trial lawyer might compare, also, based on the human factors of investigation. And, thus, so too, with sports where freely tallied points measure a player's skill at achieving an event in game play.

ralfd2• 9 years ago

I don't understand. If the average of the last 10 seasons is 20 home runs, what would be a better predicted value? You are a bit short in explaining here?

Your site, and the Wiki link, is very math formular heavy. Is there an explanation for someone who forgot all his statistic courses and greek letter thingys?

stinkytaco• 9 years ago

This is maybe a better explanation:

https://jmanton.wordpress.com/2010/06/05/comments-on-james-s...

It's still math heavy, but there is some explanation. It's hard to explain without the math, since the math if fairly integral to it, that's why it's such an amazing discovery. My understanding is that it's saying that the variables are independent, but the measurement is not. So in the case of the athletes, it's not that home runs predicts touchdowns or goals, but that by using a Stein Estimation we would get a more accurate measure of all three in aggregate. The example used in the article is less interesting, but probably better for understanding:

For example, if i=1,...3 represents the financial cost of claims a multi-national insurance company will incur in the next year in three different countries, the company may be less concerned with estimating the values of the individual means accurately and more concerned with getting an accurate overall estimate.

credit_guy• 9 years ago

Here's my intuition. Let's say you have 1000 coin flippers. They flip a coin 10 times, and none of them has any special powers, and the coin is fair. Some of them will get an equal number of heads as tails, but there's a good chance you'll get tsome who get 9 or 10 heads, and also some who get 9 or 10 tails. As the probability to get 10 heads in a row is 1/1024, if you see one or two guys how get only heads, or only tails, you will attribute that to the natural variability of the outcomes.

Now imagine that these are not coin flippers, but some guys who have some skills to do something, but the outcome has a large variability nonetheless. For example running backs in the NFL league. There are running backs (RB) who average 2 years per carry (ypc), and others who average 5. 5 ypc is stellar by the way, 4 is very good, 3 is decent, and 1 or 2 not so much. But obviously, RBs get a different yardage for each carry. Now, let's say you follow the first 4 games of the season and get the average ypc for each RB. You would like to predict for each RB the average ypc for the rest of the year. The classical statistical estimation is that the current average is the best estimator for the future average, but from the extreme example with the coin flippers above, we know that this is not quite the case. Using a bayesian estimation, we get that a better estimator is if we move the current average towards the overall mean. This is called a shrinkage or James-Stein estimator. In the case of the coin flippers, you move the average all the way to 1/2, and that estimator is correct. In the case of the running backs, you don't shrink that much, and it's a cute exercise in math to see how much you shrink if you assume some distributions around the overall ypc for RBs in the league and around the ypc of an RB given his average ypc.

If you want some further intuition, think of the Sports Illustrated curse. It was observed that NFL players who make it to the cover of the SI magazine are generally "cursed", i.e. they don't do as well after as they did before. One amusing case is the (former) New England Patriot Jonas Gray, who made the cover of SI after a phenomenal game with the Indianapolis Colts in 2014 (201 rushing yards, 4 touchdowns), but then he showed up late to work and was promptly benched for the rest of the season. Generally though, players don't do anything stupid like that, but simply "regress to the mean". That regression to the mean is what explains the shrinkage estimator, and the Stein paradox.

sampo• 9 years ago

If the average of the last 10 seasons is 20 home runs, what would be a better predicted value?

You are correct, 20 is the best estimate for this single variable (or similarly for any single variable in isolation).

Only if the objective is to minimize the total MSE (Mean Square Error),

          (Ph - h)² + (Pg - g)² + (Pt - t)²
    MSE = ---------------------------------
                          3

then it pays off to bias each estimate – Ph, Pg, Pt – slightly towards zero. If any of the observed values is larger than the true value, we do improve the estimation by using a correction coefficient slightly under 1. If the observation happens to be smaller than the true value, we do make a mistake. But we make a smaller mistake when the observed value was small because it was small, than what we improve when it was large. A set of 3 independent variables is already large enough that this gamble pays off in average (in the combined total error of the 3 estimates).

haddr3• 9 years ago

I think there is a whole class of statistical "strangeness" with using p values for hypothesis testing. For instance, p = 0.05 means that we have ~30% chance that our hypothesis is a false positive [1], which is far from what intuition tells us.

[1] http://www.nature.com/news/scientific-method-statistical-err...

TekMol3• 9 years ago

"our hypothesis is a false positive"

I wonder what you mean by this. In what sense can a hypothesis be a false positive?

haddr3• 9 years ago

I mean false positive that null hypothesis is rejected.

Houshalter3• 9 years ago

By far the most unintuitive paradox for me personally is the one presented here: https://youtu.be/go3xtDdsNQM?t=3m27s

"Mr. Jones has 2 children. What is the probability he has a girl if he has a boy born on Tuesday?" Somehow knowing the day of the week the boy was born changes the result. It's completely bizarre.

prolways• 9 years ago

Is there a more rigorous explanation of why they count the probability space how they do? Watching that video I feel like the ordering of the kids and the striking of one of the "b2b2" entries seems wrong to me. If we care which kid was first... which doesn't seem to matter... then the first b2b2 and the second b2b2 seem like they're different and shouldn't "cancel."

Then again... it took me multiple explanations to understand the Monty Hall problem.

kevinwang2• 9 years ago

Yeah, it definitely seems wrong to me. B2B2 has twice the probability of any other event listed in that table.

edit: I guess it's more nuanced than that. The explanation and interpretation sections on this blog post [1] and on wikipedia talk about the controversy.

[1] https://jakubmarian.com/the-day-of-the-week-boy-or-girl-para...

[2] https://en.wikipedia.org/wiki/Boy_or_Girl_paradox

mrcactu5• 9 years ago

my favorite data science paradox is the "curse of dimensionality"

https://en.wikipedia.org/wiki/Curse_of_dimensionality

prmph4• 9 years ago

Here is another seemingly basic question that leads down the rabbit hole: What does it mean to say two things are the same?

z3t42• 9 years ago

If you throw a six sided dice two times, there's 1/6*1/6=~3% chance to hit six both times. But if you throw one six, there's now ~17% change to hit six again ...

Paradoxes of probability and other statistical strangeness

Related Stories