The backlash against big data

http://www.economist.com/blogs/economist-explains/2014/04/economist-explains-10?fsrc=scn/tw_ec/the_backlash_against_big_data

by rrtwo • 12 years ago

8 7 12 years ago

economist.com

j2kun4• 12 years ago

Third, the problem of spurious correlations—associations that are statistically robust but only happen by chance—increases with more data.

Can someone explain why this is the case? I don't see how spurious correlations wouldn't appear in small data just as well as big data.

ZenPro3• 12 years ago

Probability theory.

If I flip one coin 100 times the outcome is likely to be close to the perception you would expect.

If I flip 8 million coins simultaneously for 100 iterations it is possible that a number of the coins will display massive outlier behaviour (98 Heads / 2 tails etc).

j2kun4• 12 years ago

Now I'm confused what is meant by spurious correlation. You're talking about an event that is known to happen, and happens infrequently, and if you have enough samples it's guaranteed to happen at least once with high probability.

Spurious correlation seems to be related to the use of ratios of independent variables making correlation "out of thin air." The word "spurious" also seems to be used in relation to hidden causation.

Neither of these two uses are related to this example, nor does the problem (in the sense in the previous paragraph) "increase" with more data.

The backlash against big data

Related Stories