Cory Doctorow: Our dangerous statistical ignorance

http://www.guardian.co.uk/technology/2008/may/20/rare.events

by graywh • 18 years ago

65 56 18 years ago

guardian.co.uk

There's a quote I like that I think came from Marvin Minsky, "the US needs a Department of Homeland Arithmetic." We're protecting ourselves from all the wrong risks.

aggieben2• 18 years ago

Uh, wouldn't this actually be called something like "the US Department of Education"?

It's been a rousing success, too - it's allowed our Ministry of Truth to claim things like "comrades! Our production of students that pass the standardized test is up 85% this quarter!".

It's all about families, man. Good families teach people what's really important - not federal departments of anonymoustaxeatingbureaucrat.

sabat• 18 years ago

And when a large proportion of families (broken, dysfunctional, otherwise unable) fail to teach kids what's important: what then?

byrneseyeview• 18 years ago

I don't understand why Marvin Minsky hates America so much that he wants to put the government in charge of assessing such risks. The government is an incredibly complex device for distilling a tiny amount of individual ignorance from each voter into a moonshine of batshittery.

SwellJoe7• 18 years ago

I'm not exactly sure what a "moonshine of batshittery" is, but you've got my vote.

rw• 18 years ago

It's presumptuous to say that voters effect any real change in government policy.

gojomo• 18 years ago

We could fund the Department of Homeland Arithmetic with lottery revenues!

mynameishere• 18 years ago

People who are quite aware of the mathematics behind gambling still do it. Gambling exploits instincts rather than ignorance.

dhimes3• 18 years ago

If I had an intuitive grasp of statistics, I may not have a start-up!

SwellJoe7• 18 years ago

Why not? It's a one-in-four (or one-in-three, depending on who you believe) chance of being rich in a few years--that means that in the worst case scenario, if you keep at it, starting over when you fail, in about 10-15 years you're very nearly guaranteed to be rich. And you almost certainly won't starve in the process.

Of course, that assumes a reasonable level of intelligence, education, and drive.

Startups are about the best game going, as far as I can tell--I wouldn't be playing if the game was rigged against me (more than a little, anyway...sure, small companies have higher relative regulatory burden, but on the whole the technology game is actually rigged in favor of new companies, from a growth perspective).

iamwil2• 18 years ago

I don't get how you're guaranteed to be rich in 10-15 years. If it's a 1/4 chance of being rich every x years, it's still just 1/4 chance over the span of 10-15 years, right?

However, I'm going to guess that you mean every time you try is going to influence the next time you try for the better (as evidenced by some paper about higher success rates for 2nd+ time entrepreneurs I remember on here), which makes sense. You learn from your mistakes, you make contacts, you have a better view of the market--so it shouldn't stay one in four every time you try.

epi0Bauqu• 18 years ago

Where do you get this 1/4 or 1/3 from?

dhimes3• 18 years ago

I guess it depends on what your goal is. My goal is making this startup work, but I'm in a dysfunctional market. I think I'd rather not know the real odds.

helveticaman2• 18 years ago

So:

25% chance, 4 cycles: 68.4% chance of one or more successes. 33.3% chance, 4 cycles: 80.2% chance at least one is a winner.

33.3% chance, 10 cycles (your entire career): 98.3%. Though it would be pretty hard to keep at it at 58, after 9 failures.

michael_dorfman• 18 years ago

I agree with all of his points, but it was a very slim article-- I was kind of hoping for a more profound analysis from someone like Doctorow.

spydez• 18 years ago

If you want a profound analysis, he wrote a (fiction) book recently on this subject, and put it out on the internet under a CC license. It's an enjoyable read; he pretty much distilled a few bits from the book to make that article.

Here tis: http://craphound.com/littlebrother/Cory_Doctorow_-_Little_Br...

billroberts• 18 years ago

I think it was about as in-depth as you are likely to get published in a (UK) national newspaper. One small but well-written step in the fight against the innumerate sensationalists that dominate the press and no doubt contribute to the populist over-reactions from the UK government.

jpeterson• 18 years ago

"Here's how that works: imagine that you've got a disease that strikes one in a million people, and a test for the disease that's 99% accurate. You administer the test to a million people, and it will be positive for around 10,000 of them – because for every hundred people, it will be wrong once (that's what 99% accurate means). Yet, statistically, we know that there's only one infected person in the entire sample. That means that your "99% accurate" test is wrong 9,999 times out of 10,000!"

No, it means that the "99% accurate" test is wrong 9,999 times out of 1,000,000. It would be clear to anyone when stated that way. What's counterintuitive is the author's statement of the result, not the result itself.

graywh5• 18 years ago

http://en.wikipedia.org/wiki/Bayes'_theorem#Example_.232:_Dr...

Basically, an extremely low rate of incidence (in Cory's example, 1 in a million) will render a "highly accurate" test meaningless.

tel3• 18 years ago

A better wording is that your chance of having the disease if given a positive result from the test is 1/10000 (0.1%).

That's a huge increase from a 0.0001% chance of having the disease, but it's still not flat out terrifying. Repeat testing can weed through the false positives at a speed proportional to its accuracy.

graywh5• 18 years ago

Not necessarily. If you get the same results every time, repeated testing provides nothing.

andrewparker• 18 years ago

Riding on the subway earlier today, a guy got on an starting preaching how "the lord saves" and "you must embrace Jesus." I wish math could create equivalent evangelism. I'd love to live in a world where a guy gets on the subway and starts preaching the Pythagorean theorem?

graywh5• 18 years ago

That one's not the problem. Perhaps Bayes' theorem?

henning• 18 years ago

a more basic problem is just plain ignorance.

many if not most citizens of the USA do not understand the basic organization and functions of their government. most of them cannot name a single sitting supreme court justice, have never read the constitution, do not know how a bill becomes a law.

and these people vote.

redorb• 18 years ago

good case against data mining... and the "if it saved one life, would it be worth it?" argument doesn't change my mind.

ken• 18 years ago

Statistician Peter Donnelly on the same subject:

http://youtube.com/watch?v=kLmzxmRcUTo

niels_olson2• 18 years ago

As someone who deals in diagnostic tests all day, nobody's diagnosed on one test. There's a similar problem I worked out on the odds of getting into medical school, I'm sure this is elementary stats and has someone's name on it, but I'll call it the acceptance problem: how many medical schools do you have to apply to in order to have 90% chance of getting into medical school? For any school there is a acceptance quotient: Q = (acceptances sent out)/(number of applications received)

For any given student applying to some schools 1 thru n, the goal is getting at least once acceptance, and applying to more schools, mathematically, can't possibly hurt in the closed case (neglecting social engineering, time spent on applications, etc), so the chance of acceptance, Ca, approaches 1 with every new application in the following fashion:

Ca = (1 - (1 - Q1)(1 - Q2)(1 - Q3)...(1 - Qn))100

There's a visual and some worked out examples here: http://nielsolson.us/MedSchool/

Similarly, if the sensitivity of a test is 90%, that means the test identifies 9 of every 10 people with the diagnosis. If I administer n different tests each with a sensitivity of S, then the chances of accurately diagnosing the disease, Cd, goes up* with each additional positive but never gets to 1.

Cd = (1 - (1 - S1)(1 - S2)(1 - S3) . . . (1 - Sn))100%

So lets say you are doing, say, genetic testing, and any one gene is 1% sensitive for the disease. If you tested 300 genes you could be no more than 95% certain of the diagnosis.

(1 - (1 - .99)^300)100 = 95.09591...%

Now, if your genetic tests were 5% accurate, you're panel could be no more than 95% accurate with 59 tests.

If your test was 50% accurate, you're panel could consist of 6 tests and be no more than 95% accurate.

Of course, if some of the tests are negative, things get more complicated. One of the problems with these data sets is that we have no idea how predictive they are. You can't even calculate the predictive power of the database. There simply haven't been enough events. Then we get into surrogate measures (how many were positive on tests 1 - n and were found to have razor blades in their homes, etc).

The claim that these databases can't be effective isn't true. They could be. P might also equal NP. Whether the hypothesis is strictly true or not, the vague but real set of 'practical concerns' suggest that the truth of the hypothesis is sufficiently difficult to test as to render the null hypothesis the de facto assumption until proven otherwise.

aston2• 18 years ago

The assumption of independence underlying the math you're doing is probably outright false (albeit mathematically simple). As one example, imagine the case where all n schools use the exact same admissions criteria. Unless you're applying with a random application to each school, your math is shot; you will either get into every school or none of them. I won't even go into the genetic independence issue.

niels_olson2• 18 years ago

Every mathematical model is a false representation of reality. The predictive accuracy must be validated by experiment. And using the link I reference, you'll see there's actually some pretty strong data to start from in this case.

breck• 18 years ago

"But the fact is that attacks by strangers are so rare as to be practically nonexistent. If your child is assaulted, the perpetrator is almost certainly a relative."

Maybe that statement is true in today's world. But for tens of thousands of years, while our brains evolved, I would guess attacks from strangers were a lot more common.

tel3• 18 years ago

The causation implied here is unlikely at best. More likely might be that until the last century no one had to deal with numbers large enough to need these kinds of statistics.

seshadripv• 18 years ago

It's not just the probability of an occurence that leads us to behave in an unreasonable manner. The other important thing is the impact of that 'rare occurence'. Most critics don't seem to take that in to account at all...

__• 18 years ago

Something doesn't seem right about Doctorow's example of attacks on children. He writes, "But the fact is that attacks by strangers are so rare as to be practically nonexistent. If your child is assaulted, the perpetrator is almost certainly a relative (most likely a parent)."

I'm sure that's true, but it doesn't answer the real question. For most parents, the question is: given my child's particular environment, what is the greatest threat to him?

Doctorow is saying: given that your child was attacked (and no additional information), the attacker is more likely than not to be a relative. That seems backwards.

tel3• 18 years ago

He's stating that the evidence is P(Was Relative | Child Attacked) is relatively high, especially compared to P(Was Total Stranger | Child Attacked). This means that it should, if you use Bayes' Theorem correctly in everyday life, shift your suspicion away from the random photographer.

Of course, since P(Child Attacked) is so very low it's still not a huge deal.

It's not really backwards. That sort of inversion is exactly part of Bayes' Theorem.

Retric• 18 years ago

If you are related to X people then the odds that one of them attacks your child is greater than that anyone else on the planet.

Picking numbers from thin air:

Let's say are related to 30 people know 200 and you encounter 10,000 random strangers and you have a 1 in 100 chance of an attack. Well the odds a specific random stranger attacks your child given the above assumptions is less than 1/2,000,000. The odds a specific person they know attacks your child would be less than 1/20,000 and the odds that a given relative attacks your child would be more than 1/6,000. Now who would you focus on? (What about a predator that has a 50% chance of attacking someone well he is still under 1/6000 because there are so many other people for them to attack.)

PS: Given my child's particular environment you should focus on relatives and people you know.

JacobAldridge2• 18 years ago

I believe the worst damage to statistics is specious reasoning. My favourite chocolate promotion is Mars' '1 in 6 Wins a Free Bar' (on currently here in Oz). If I buy 6 bars, most people would assume I would win once. In fact, I have only a 2/3 chance of winning a free one

1 - (5^6/6^6)

Buy 12 bars, and there's still a more than 10% chance I won't have won yet...most of the chocolate-buying government-voting lottery-praying public would be stunned.

aston2• 18 years ago

Although the average number of bars you'd need to buy before you win is indeed six. I think almost no one would put the odds of actually winning one by buying six at 100%.

JacobAldridge2• 18 years ago

"Any 6 bars may not contain a free bar"

You're probably right, and I hope that you are. Still, funny to note that Mars has even put this disclaimer on the bottom of their promo site!

http://www.marsfreebars.com.au/

graywh5• 18 years ago

And as N approaches infinity, the probability of winning after N tries with a p=1/N approaches 1 - 1/e.

(Proof left to the reader as an exercise.)

Cory Doctorow: Our dangerous statistical ignorance

Related Stories