The Central Limit Theorem and Its Misuse

https://lambdaclass.com/data_etudes/central_limit_theorem_misuse/

by lelf • 7 years ago

176 67 7 years ago

fyp• 7 years ago

Can someone give me an example on how to properly use the central limit theorem in the real world?

For example with stuff that are commonly assumed to be bell-curved like test scores, IQ, etc. What are the iid variables being averaged? Each test question?

throwlaplace4• 7 years ago

there's a distinction between random variables that are assumed to actually be normal and RVs that we conceptually apply the clt to and then treat as normal.

in the case of test scores it could be either. for an individual test it's possible the distribution is roughly normal actually. but if you, for example, look at subsets of all SAT scores and take their means (the mean of the subset) then those averages will be Bell shaped because of the clt.

Just to directly answer your question: the iid random variable is the test score or the iq.

markkvdb• 7 years ago

Suppose you have a school with 30 groups of students. All students are randomly assigned to the groups (so independent of their skill at IQ tests). The distribution of measured IQ within classes can be any distribution. But let’s take the 30 averages of the groups. The mean of these averages of groups (asymptotically) follows the normal distribution according to the CLT.

throwlaplace4• 7 years ago

not the mean of the means - just the means themselves.

The distribution of the original population of iqs in the school is what you're sampling. Each group is a sample. The distribution of the sample mean approaches a normal according to clt as the sample sizes increase.

raegis• 7 years ago

He's the example I have my students do in class. Roll a die repeatedly and tally the results. You'll get a (roughly) uniform distribution of 1s, 2s, 3s, 4s, 5s, and 6s.

Now to illustrate the CLT, you roll a die 50 times, and average the result. AND your 300 classmates do the same. If you tally the 301 averages, the distribution of the averages will not be uniform but bell-shaped, with average (approximately) 3.5.

The CLT says (roughly) the distribution of the averages will be approximately normal, regardless of the original distribution.

ganzuul• 7 years ago

Short and concise, as things should be. Thank you. :)

laichzeit03• 7 years ago

Here’s something that’s bothered me for a while. Why is it necessary to roll it 50 times? Why can’t I roll it just once? Does each X_i have to be a sample greater than 1 and if so is there some kind of minimum required for each random sample? Does CLT still hold if each random sample’s size is 1?

amelius• 7 years ago

Ok, but the real pdf is cut off at 1 and 6, while the normal distribution extends to +- infinity and can never model the cutoffs, no matter how large N is (though of course those tails get less significant).

roenxi• 7 years ago

The CLT can be used to justify modelling an effect with a normal distribution and raises questions about model assumptions when normal distributions don't appear where they should. The Normal Distribution is special because it is the highest-entropy distribution that is completely characterised by mean and standard deviation [0].

So what the CLT is saying is that, surprisingly, sums of i.i.d. random variables lose information from the individual variables quite quickly but reliably retain a little data about mean and second moment. Initially a given X_i has all sorts of information associated with it (higher order moments, other distribution characteristics, etc) that disappears as many X_i are summed together. All that is left is information about a mean and variance.

So say I have a situation where a large number of i.i.d. variables are going to be summed together. The CLT tells me that summing N of these variables together is going to be similar to summing together an equivalent number of normally distributed variables (!!). This is because the sum variable is equal in value to n * \bar{x}, but the CLT imposes a distribution on \frac{\bar{x}}{n}.

This justifies why the normal turns up everywhere in practical measurements. A lot of measurements (say, number of people at the beach) are probably really measures of a sum of random variables (maybe there is some non-normal variables that captures the chance a given person goes to the beach). So if the number of people at the beach turns out to be normally distributed (maybe I measure it each day for a few weeks) it isn't shocking. If the total number isn't normally distributed then that implies that there is no i.i.d. variable representing probability a given individual goes to the beach (eg, high correlation between the individual variables).

I didn't quite fail any of the statistics courses I've ever done, YMMV.

[0] https://en.wikipedia.org/wiki/Normal_distribution#Maximum_en...

MichaelRazum2• 7 years ago

Not 100% sure, but for me "The three coins (non) example" is false. You could easily constract a random variable wich flips the free coins at random. Then you get again an iid sequence.

I'm writing this basically because I think it is troubeling how "blogs" work. Explaining, lets say complex stuff, with simple diagrams. Especially in this case it even looks wrong and gives you the wrong intuition.

Schiphol3• 7 years ago

The RV you suggest you could construct is not how the example works. From the article:

Let's consider the following scenario: say we have three coins with different biases (their probability of coming up heads): 0.4, 0.5 and 0.6. We pick one of the three coins at random, toss it 300 times and count the number of heads.

So, not choosing a random coin every time, but choosing it once and tossing it 300 times.

MichaelRazum2• 7 years ago

Maybe that is correct. What I'm just missing is a simple "proof" or at least an idea of a proof why some assumptions are violated. In this case you could a gain argue: picking a coin random and then toss 300 times --> Multidimensional CLT. Not saying the article is wrong. I'm just saying, where should be a proof or at least a bit more evidence... Just a simple plot is not enough from my perspective

em5003• 7 years ago

You're correct, and the blogpost (and several replies) are wrong. The sum or sample average of a simple mixture of Bernoulli distributions still converges to a Normal.

There are too many errors and misconceptions in this blog and thread, confusion between mixture distributions and the sample mean of said mixture, confusion between sufficient and necessary conditions (IID is sufficient to proof the Lindeberg–Levy CLT, but it's not necessary for more general forms Lindeberg-Feller, Lyapunov).

Unfortunately I'm at work and don't have time for a comprehensive rebuttal today. But people with some numpy or R skills can simply do the simulations themselves and see that the blog is not correct.

LaserPineapple2• 7 years ago

The sum or sample average of a simple mixture of Bernoulli distributions still converges to a Normal.

The procedure given in the example is not a sum of samples from a mixture of Bernoulli distributions. It is a mixture of sums of Bernoulli distributions.

pmiller2• 7 years ago

The TL;DR here seems to be that the CLT is false when it’s hypotheses are violated. This does not seem to be news.

bsaul• 7 years ago

you’re kind of missing the point imo... the goal is to explain misconceptions and misuses of the CLT. it’s useful to people that actually don’t understand properly the hypothesis of the CLT.

hgibbs• 7 years ago

Interestingly, the O(1/sqrt(n)) error in the CLT provided by the Berry-Eseen Theorem is the best one can do for general sequences of our RVs. However, if you are happy to put certain 'bumps' in the normal distribution then you can improve the error to O(n^(-r/2)), where I think r is some integer depending on the smoothness of your random variables. These bumps are called an edegworth expansion.

manthideaal• 7 years ago

The example of misuse of the CLT illustrates that a mixture of normals is not a normal distribution. For example the height of people is a mixture of two normals, one for men and the other for women, this mixture has two modes and so it is not bell shaped and is not a normal distribution.

elcomet2• 7 years ago

CLT is about the sum. In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman)

BeetleB2• 7 years ago

In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman)

Yes and no. The sample must be independent and identically distributed. In your case the "identical" part is not correct, as men and women have different distributions (both are normal but with different mean and std). However, if both distributions are normal, then their sum is normal (even with different mean and std).

The fact that the sum is normal in this case has nothing to do with the CLT - it's just a quirk of the normal distribution that the sum is normal. Had men/women had non-normal distributions with different means/stds, then the sum would not be normal.

joker3• 7 years ago

1) Human height isn't actually bimodal even if some histograms display two modes. See https://amstat.tandfonline.com/doi/abs/10.1198/00031300265#.... for an analysis.

2) The central limit theorem applies to the distribution of the sample average. It applies whenever the samples are iid and the second moment is finite. The fact that the samples are coming from a mixture of normals doesn't change that.

paulddraper4• 7 years ago

Human height isn't actually bimodal even if some histograms display two modes.

"Bimodal" doesn't really have a precise definition or test, if you don't assume normal distrubtions.

That paper argues that only if means are separated by 2σ should the distribution be considered bimodal.

But there are many measures of bimodality. [1] [2] [3] [4] [5] [6]

---

In any case, I would be very much surprised if the population couldn't be selected enough (age, race, country, diet, family) to have human height be bimodal by any measure.

[1] https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/97WR...

[2] https://journals.sagepub.com/doi/10.4137/CIN.S2846

[3] https://link.springer.com/article/10.1007%2Fs11207-008-9170-...

[4] https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809....

[5] https://link.springer.com/article/10.3758%2FBF03205709

[6] https://esj-journals.onlinelibrary.wiley.com/doi/abs/10.1007...

YeGoblynQueenne• 7 years ago

This is a nice article but if it's really meant as an explanation of CLT for people who don't know much about it, then the (pretty) graphs should have their axes labelled clearly and the symbols used in the formulae presented should be also clearly explained (e.g. the article uses E for expectation without immediately explaining it).

It's no use "explaining" what is supposed to be a novel concept by presenting it in a way that is difficult to understand if one is not already familiar with it.

wodenokoto• 7 years ago

What does the histogram of cointosses represent?[1]

We have 300 coin throws, so basically a list of H's T's.

My go-to for a histogram would be to have the buckets represent categories. I Throw a coin 300 times and plot the number of heads on one column, and the number of tails on the other column.

Next I would maybe do a bucket-size of 10-throws and count outcomes chronologically in each bucket.

Lastly, I would consider the same, but only with comulative counts.

[1] https://lambdaclass.com/data_etudes/central_limit_theorem_mi...

gbrown• 7 years ago

It’s the estimated (via simulation) sampling distribution of the number of heads observed out of three hundred coin tosses.

eanzenberg6• 7 years ago

CLT is abused among data scientists quite often, and usually due to the points in the article.

- Convergence takes infinite time when the sampling variable has infinite variance (imagine a random number generator).

- While the distribution of an aggregate statistic tends to a Gaussian, most underlying distributions are not and most stats is done using the underlying raw data.

Junior data scientists assume everything is Gaussian (sometimes attributing it to CLT) when doing linear regression when most distributions being modeled are not. Then, analysis done using variance on the coefficients is meaningless because the assumptions are incorrect.

kxxsc• 7 years ago

This comment is surprising to me. Most data scientists use results and measures of statistical significance provided by the program they're using, which accounts for the distribution used. Do you have examples where either data scientists are not presenting aggregate statistics or where someone is using the wrong kinds of p-values?

To your specific examples - the coefficient of a linear regression is distributed normally (?). Similarly, we know the expected distribution of most maximum likelihood estimators (logistic regression, etc.), and programs will give you the right p-value.

Of course, omitted variable bias is still a problem and it is possible to mis-specify your model. However, I think most data scientists are presenting aggregate statistics (means, regression coefficients) like you said, and that we have a pretty good handle on the underlying distributions.

throwlaplace4• 7 years ago

which accounts for the distribution used

what does this mean? are you trying to say the code uses the empirical distribution as a proxy for the true distribution?

wbl• 7 years ago

Mispecification error is hard to deal with.

eanzenberg6• 7 years ago

For example, there were times where features were excluded because their "p-value was too large to be significant", regardless that the underlying distribution was not Gaussian. p-value from a t-test, like in lots of regression software, requires Gaussian distributions.

kyllo2• 7 years ago

Linear regression doesn't require or assume the input data values to be normally distributed, just that the residuals are normally distributed and have constant variance.

eanzenberg6• 7 years ago

Linear regression doesn't require much, honestly, just non-null numeric data. My point was more on the interpretations of the outputs.

clircle3• 7 years ago

Further, those assumptions are only necessary for confidence intervals and hypothesis tests.

paulddraper4• 7 years ago

Convergence takes infinite time when the sampling variable has infinite variance

When does that really even happen in practice though?

most distributions being modeled are not

That a rather assertive statement. I dare say most distributions are (due to CLT) Gaussian.

clircle3• 7 years ago

Distributions are only assumed. There is no such thing as truly normal data or sampling distribution.

srean• 7 years ago

That a rather assertive statement. I dare say most distributions are (due to CLT) Gaussian.

I have been doing stats/ML/data-science for more than a decade. The above has rarely been true ever in the datasets that I have looked at. Unless of course the raw data has been transformed in ways to make them look Gaussian. Gaussian is the exception, not the rule.

clatan• 7 years ago

Everyone assumes things that aren't true, not just "junior" data scientists. There is not standard methodology for non gaussian non independent random variables.

bobbyd2323• 7 years ago

There are MANY CLTs. In general there is a memory-moment trade-off in assumptions for a given CLT. For example, you can relax the iid assumption to allow dependence between observations, such as a mixing condition, with the cost being you need a little bit stronger moment condition where the moment condition is a function of the mixing condition - more dependence (up to a point) requires a stronger moment.

xpe• 7 years ago

Dear authors: Please add timestamps showing when the article was first published and updated. This helps situate the article in a larger context.

paulddraper4• 7 years ago

Agreed.

It appears to have been published yesterday-ish. [1]

[1] https://web.archive.org/web/*/https://lambdaclass.com/data_e...

The Central Limit Theorem and Its Misuse

Related Stories