Statistics have lost their power (2017)

https://www.theguardian.com/politics/2017/jan/19/crisis-of-statistics-big-data-democracy

by prostoalex • 7 years ago

58 59 7 years ago

al2o3cr• 7 years ago

Describing this entirely as "the declining authority of statistics" is like saying the WTC towers suffered "declining structural integrity" before they collapsed - there are people behind both...

Google234• 7 years ago

God damnit, you should know better than bringing up WTC towers here. It attracts conspiracy theorist like how moth are drawn to a flame.

freejulian85• 7 years ago

At one point does policy targeting a statistical measurement render that statistic useless? I’m looking at you, CPI.

Majromax2• 7 years ago

At one point does policy targeting a statistical measurement render that statistic useless?

When the statistic is a proxy for a quantity of interest, and when policy exploits ways of manipulating that proxy without commensurate change in the underlying quantity of interest. My driving policy of targeting my speedometer does a fine job of controlling my vehicle's speed even though the former is just a potentially-inaccurate measurement.

In the case of the CPI (actually the PCE for the US Fed [1]), the quantity of interest is the "[nominal] cost of living." To the extent the CPI doesn't in fact measure the cost of living, the easy solution is to change to a price index that does measure the cost of living. Since living tends to involve buying and consuming things, a price index, suitably defined, can be a fine measure of this.

To the extent you are concerned about the exact ways in which the CPI is computed, independent measurements [2] generally reproduce the broad trends in US data, which in turn suggests there is no significant manipulation by the BLS.

[1] -- https://en.wikipedia.org/wiki/Personal_consumption_expenditu...

[2] -- http://www.thebillionpricesproject.com/

cjslep2• 7 years ago

As usual, the title's description is wrong, and the real news is buried and overlooked in the story:

"People assume that the numbers are manipulated and dislike the elitism of resorting to quantitative evidence."

If people don't believe that something is useful, it doesn't matter what that "something" is, it will start to have a diminishing affect our society. It goes for statistics and liberal democracies just as much as ancient gods and ancient languages of yore.

traderjane2• 7 years ago

That's like saying if people don't like military drones, then military drones will start to have less relevance; but some technologies are sufficiently powerful in their own right that it needs only a few people to recognize their power. Then the stress of competition can do the rest.

I doubt statistics or mathematics gained its influence because various cultures were fond of symbolic play. We study these things because we expect concrete payoff, not because of cultural affection.

So is social science lame? Or is Google insight scary? At some point a deliberate posture of mis-devaluation will come to terms with reality.

cjslep2• 7 years ago

That's like saying if people don't like military drones, then military drones will start to have less relevance;

Sure I'll back that up. Thought experiment: if everyone, including all the drone pilots, suddenly magically stand up all at once, walk away, and never talk about it again, then yeah it won't impact humanity.

Is this particular scenario realistic? No. Has mathematic or chemical knowledge been collectively lost before in human history, and been eliminated from society? Yes.

but some technologies are sufficiently powerful in their own right that it needs only a few people to recognize their power. Then the stress of competition can do the rest.

Sure, those few power can recognize its power, but if it's anything short of immortality, never underestimate the power of every other human being being a Luddite wielding a pitchfork ready to kill over progress, because it violates a cultural norm. Doesn't matter if your statistics can find a cure, societies and collective humans' irrationalities are expert suppressors of technology as both a mean and an end. Look no further than the US stance on nuclear waste reprocessing. The science and power are there. The will of people, is not. Or, look at South Africa and Ukraine giving up nuclear weapons. Power? Yes. Popular will to hold onto this power? No.

We study these things because we expect concrete payoff, not because of cultural affection.

Exactly, because society values concrete payoff, things like "derivations from first principles" and "following the scientific method". But if society as a whole pressures people to never pursue ideologies along these lines, or turn actively hostile against it (book burning, HDD erasing).

I take this article as an alarming warning. Scientific progress is itself not an end. And it is not an inevitability. I like scientific progress and statistical tools, this article just goes to show how careful -- ethically responsible -- those of us wielding them must be when living in our society. And call out bad actors.

This is why engineering professions have ethics boards. If engineers build bridges and bad actors build bad bridges, people won't trust bridges anymore. And if people don't trust bridges, they don't get used, so more don't get built.

The greatest failure of the "modern" education has precisely been the removal of religious ethics without an equivalent amount of time working to help students develop their sense of personal ethics.

aeternum• 7 years ago

I see this as a good thing. Many 'scientific' studies are manipulated and paid for by special interests. Our existing peer review & journal system hides the actual experimental setup behind paywalls, there is often no repeatability.

The scientific system needs to be opened up or replaced. Distributing information and experimental results no longer requires expensive printing.

People are right not to trust it, scientific journals are now the gatekeepers rather than the gateways they once were.

deogeo2• 7 years ago

I can't help but laugh. Make the slightest mention of statistics broken down by race, and the same publication that is despairing of how people no longer trust statistics, would be finding innumerable reasons why those statistics are invalid.

commandlinefan• 7 years ago

Hey now, in the Guardian's defense, it's not as if any world government has ever been caught lying and misrepresenting facts to its population in the past.

bpodgursky• 7 years ago

/cue handwringing op-ed about biased AI algorithms

jayd162• 7 years ago

Why does that make you laugh? Your take away should not be "well clearly everything someone calls stats should be supported". The article is about how stats are becoming dismissed before examination, not that they shouldn't be scrutinized.

deogeo2• 7 years ago

And that all the examples given of uneducated masses distrusting statistics just happen to align with the Guardian's pet causes is of course pure coincidence.

Really, that's the perfect example of why statistics are distrusted - only those statistics that align with someone's agenda get presented. And sometimes, they're not even gathered in the first place: https://www.thelocal.se/20180508/why-sweden-doesnt-keep-stat...

vanniv• 7 years ago

I guess the point is that, when the statistics support their political bent, they decry people questioning the validity of them, and consider not blindly accepting them as being proof of the idiocy of the reader.

But, give them statistics that go against their pre-ordained narrative, and suddenly it is the reader who doesn't immediately refute the statistics who is the uneducated rube.

prvc• 7 years ago

Journalism has been one of the worst offenders with regard to the misuse of statistics, not only in "soft" topics, but also in science reporting. Perhaps statistical literacy ought to be normative in that profession the same way that checking sources and using correct grammar (supposedly) is.

mikelyons• 7 years ago

Even if statistical literacy was normative wouldn't people's selfishness still cause them to use/abuse statistics to influence those who are less literate in them?

lostmyoldone3• 7 years ago

Knowledge can be abused, but lack of knowledge will almost certainly lead to, unintentional, abuse. Especially in the case of statistics.

In many cases, reporters want to be dramatic, but not liars. However, the lack of deeper statistical knowledge often lead to them becoming liars, even though they only wanted to be dramatic.

PaulKeeble• 7 years ago

It is more about declining trust in the source of that information and the institutions that spread it around. People have been given plenty of reasons to distrust the various sources of information over the past decades.

glofish8• 7 years ago

It is a 17th-century concept applied to 21st-century data collection.

Plus it gets misused all the time - even by statisticians that should know better.

threwawasy12283• 7 years ago

This is an objectively good thing, most statistics don't actually tell most people anything interesting about the world. They don't aid the average untrained guy in the street in gaining a better understanding of the world around them. They are bombarded with opinions that are based on contradictory studies and statistics all day, this is bad in politics and sociology but medical statistics are probably the worst offenders.

Wine is good for you, chocolate is bad for you, cholesterol is good for you, wine is bad for you, red meat good for you, Fluoride bad for you! I could find you a statistically significant study supporting any conclusion I want right this minute. Since that is the case, why should the average Joe believe any of them? This is made even worse by news articles and popsci blogs not even linking to the source papers, so avg Jane can't even investigate further if she wants to.

lotsofpulp8• 7 years ago

Statistics aren’t bad. Extrapolating unwarranted conclusions is bad.

threwawasy12283• 7 years ago

I am not a trained medical professional, and I don't have time to be scrutinizing and reading background literature to gain a full understanding of the context each specific study has within its sub-field. I don't have access to whether these conclusions are generally accepted or are commonly disregarded for some technicality in the methodology that I didn't understand. Me personally, I can't be an expert in everything all the time, and I can't waste my life gaining a full breadth of understanding of every topic. I rely on extrapolations to do this for me, if those extrapolations are commonly faulty, I won't pay any mind to them.

willturman• 7 years ago

Sure, but is it significantly bad?

glofish8• 7 years ago

the problem is that in many (most cases?) even trained scientist would have a hard time deciding which statements are statistically valid.

There are endless ways to be unintentionally obscure, to semi-intentionally obfuscate or to just flat out lie. And it very hard to tell these apart - one would need to dedicate more effort than usually worth.

chhickman• 7 years ago

Extrapolating unwarranted conclusions is bad.

But that's practically all they're used for!

heavenlyblue• 7 years ago

Wouldn’t knowing statistics allow the average Joe to realise that most of these studies are simply not representative?

willis936• 7 years ago

The fact that the statement “most statistics don't actually tell most people anything interesting about the world” relies on two statistical conditions is humorous to me.

traderjane2• 7 years ago

That a current generation is under-equipped to use quantitative discourse is sad, but it also suggests that what communities need are experts and not evidence. They want to listen to a trustworthy leader and not conflicting data -- because data will conflict.

airstrike• 7 years ago

but it also suggests that what communities need are experts and not evidence

IMHO it actually suggests we need to improve the average person's educational level

peteradio• 7 years ago

Bet you dollars to donuts none of the underlying studies you complain about pass judgment in the form of x good x now bad. Those are always added by the pop media or implied by the reading.

tonixie• 7 years ago

There's an interesting blog post about something similar:

https://slatestarcodex.com/2014/04/28/the-control-group-is-o...

Basically it boils down to the fact that studies & statistics used are still very, very, prone to human bias.

lostmyoldone3• 7 years ago

Which can as well be seen an argument why statistics are necessary, as even with them we are stuck with a lot of bias. Imagine where we would be if we didn't have statistics at all!

ljw1001• 7 years ago

A statement like "most statistics don't actually tell most people anything" demonstrates both a profound lack of understanding of statistics. So why comment?

As the old saying goes: "Better to remain silent and be thought a fool than to speak and to remove all doubt."

lostmyoldone3• 7 years ago

To understand statistics, you have to either know some statistics, or be willing to learn.

It is it's own thing, the art and science of making data and observations into information. Yes newspapers and the regular pundits abuse it all the time, but don't pin that on statistics, pin that on those that abuse it with, or without (usually) knowing better.

Statistics are unavoidable, unavoidable enough that much of our perception seem to make use of the same statistical calculations sciententists use, only without us having the ability to scrutinize its workings.

jammygit• 7 years ago

Too many science projects reporting statistical results that do not hold up in repeat experiments; contradictory media coverage of the same topic repeatedly as a result.

Reduces trust in science, statistics, and the media all at once.

lotsofpulp8• 7 years ago

The problem is verifying experiments gets no funding.

munk-a• 7 years ago

I think the bigger problem is that the initial experiments are usually funded by interests with a desired outcome in the study - instead of being funded by grant money or educational funds.

Gimpei• 7 years ago

I think the solution to this problem is more statistics. In particular, we need a mandatory statistics course at the high school level. The problem is that people are bombarded with "studies" right now claiming to prove stuff and many of them do no such thing. If people had a better sense of what is provable and what isn't, what makes for a good study, how much you should trust the results of a single study etc, I'd hope that they'd be far more informed and effective citizens.

glofish8• 7 years ago

that is the cliche definition of insanity:

"doing the same thing over and over and expecting a different result."

probably the most common mistake students everywhere make. They keep taking more courses - it never helps just makes you more confused.

Majromax2• 7 years ago

Ultimately, I feel this headline ("Statistics have lost their power," in case the Guardian changes it later) buries the lede. The dispute is not really about statistics as a way of understanding the world, it is about the epistemology of it, and at its core the idea that a national aggregate is a valid proxy for well-being.

It's easy to decry:

Presented with official estimates of how many immigrants are in the country illegally, a common response is to scoff. Far from increasing support for immigration, British Future found, pointing to its positive effect on GDP can actually make people more hostile to it.

... but the underlying belief appears to be closer to:

If you live in one of the towns in the Welsh valleys that was once dependent on steel manufacturing or mining for jobs, politicians talking of how “the economy” is “doing well” are likely to breed additional resentment. From that standpoint, the term “GDP” fails to capture anything meaningful or credible.

The first statement makes no sense in and of itself, and it would seemingly suggest a knee-jerk reaction against the very idea of data. However, in the context provided by the second statement we see a different view: a core distrust that "average well-being" (represented by the statistic of choice) in fact represents the common good, suitably defined[1].

As a result, I feel that the article's later musings -- that we're merely collecting the wrong kind of statistics -- misses the core point. If we start reporting on underemployment, then the right circumstances might come along to discredit that indicator, and so on. The article begins to grapple with this issue in its digression into big data (and how there are no national agencies for it in the manner of the ONS), but even there the author deals with a practical problem of said data being more useful in the short term than long term than the core problem:

Ultimately, if society as a whole does not have a shared view of the 'common good', then _no quantity at all_ can measure it. The question of "is this good?" becomes fundamentally unanswerable in a democracy with an identity crisis.

[1] -- That could mean "median well-being," "well-being of people I know / in my region / like me," or "personal, individual well-being of the speaker."

chongli• 7 years ago

I feel like all of this is overcomplicating the issue. Now, at the risk of oversimplifying, I offer this:

If I am not doing well and some journalist or some economist tells me the GDP has grown a lot and everything is great, I am going to feel resentment toward that person. Doubly so if they see my less-than-enthusiasm and start talking about the “common good.” Why? Because the implication is either that there’s something wrong with me (for not keeping up with the Joneses) or that I am an acceptable sacrifice for the benefit of the common good.

Okay, it’s fine if you (the journalist or economist) think I’m an acceptable sacrifice, but don’t expect me to be happy about it, let alone agree with you. Don’t expect me to support the status quo, come Election Day. DO expect me to try to throw a spanner into the gears. DO expect me to support a strongman who says he’ll champion my cause over the cause of those who think I’m the acceptable sacrifice.

caseysoftware• 7 years ago

Even if the underlying statistics accurately represent the situation correctly, bad analysis on top of it creates the same results. From last week:

"Paul Krugman and other mainstream trade experts are now admitting that they were wrong about globalization: It hurt American workers far more than they thought it would."

Ref: https://news.ycombinator.com/item?id=21357096

QuesnayJr• 7 years ago

That Foreign Policy article illustrates exactly why the influence of statistics is on the decline. If you read Krugman's original article (https://www.bloomberg.com/opinion/articles/2019-10-10/inequa...), you can see Krugman make several points, all using statistics. One point is that the effect of globalization in the 90s was small, but it had an unexpectedly large impact on manufacturing in the 2000s. The second point is that the overall effect disguised the fact that certain industries were particularly hard hit. He then cites the papers where you can find the details. These papers only exist because of the statistics they do. Krugman is a clear writer, and he is taking the unusual step of admitting he was wrong, so his article should be interesting.

But nobody cares about that, because nobody cares about statistics anymore. They want that dopamine hit of outrage, so they read the FP article, that deliberately misrepresents the argument (notice how "manufacturing" suddenly becomes "workers", despite the fact that most workers are not in manufacturing), and doesn't give any evidence. It just quotes a bunch of people settling scores with Krugman. Even there, it mischaracterizes, for example, Krugman's review of Greider's book to heighten the outrage.

Statistics is boring, and rage is interesting, so the Internet era optimizes for rage.

motohagiography• 7 years ago

The quote, "using statistics like a drunk uses a lamppost, for support and not for illumination" comes to mind.

Data is great for illuminating things, but for arguments, it's usually cherry picked. I do think that writers for the guardian vastly underestimate the intelligence of people who don't agree with them. I generally mistrust statistical arguments because someone forearmed with statistical talking points has usually already signalled their intransigence. Data is nice, but alignment of interests and incentives is better.

It reduces to, "according to criteria and data I have selected, your argument does not bear out." It is a passive appeal to the authority of whoever provided the number, and not meaningful or logical information in itself.

Logical arguments tend not to persuade anyway, but to say someones argument is illogical because it does not resist a coded illogical premise (appeal to authority fallacy), is certainly a way to make them give up on arguing with you.

08-15• 7 years ago

This single sentence explains the problem very well:

The thinktank British Future has studied how best to win arguments in favour of immigration and multiculturalism.

BF didn't set out to figure out whether immigration and multiculturalism is favorable. They only tried to figure out how to use statistics (or anything else) in support of their foregone conclusion.

Maybe the headline should be "Statistics have lost their usefulness for propaganda."

MR4D• 7 years ago

90% of statisticians agree with the conclusion.

The other 90% challenge the sources of this article.

(With apologies you Yogi Berra)

Statistics have lost their power (2017)

Related Stories