Type I and Type II Errors: The Inevitable Errors in Optimization Experiments

99 35 6 years ago

Scea91• 6 years ago

It is funny that although I have published peer-reviewed papers about evaluation of ML models and could probably discuss with you about many nuances of the process if you woke me at 3 AM, I always find it difficult to remember which of [Type I, Type II] is false positive and which is a false negative. If I have this problem I suppose almost everyone has it.

I wish people stopped using [Type I, Type II] when we can use clearly superior terminology. This feels to me very similar to amateur software engineers using non-descriptive variable names.

pfortuny• 6 years ago

This is exactly my experience. I have never known what type the error I was talking about was...

Imagine calling (for instance) type I groups those which are abelian and type II those which are not... And then not being a professional mathematician. So is xy=yx here? Mmmmhhh type I says yes or was it no?

twanvl2• 6 years ago

Imagine mathematicians calling commutative groups abelian? How do you remember if xy=yx there?

laichzeit0• 6 years ago

I have the same problem with specificity and sensitivity. Always have to look up the damned terms even though I’ve used them countless of times.

jackallis• 6 years ago

what are those "superior terminology"?

amluto• 6 years ago

“False positive” and “false negative” are pretty good.

beagle3• 6 years ago

Maybe I can help you with that ... the mnemonic I developed is:

Type I error, with probability often denoted by α (alpha) which is the first (I) letter of the alpha bet. AL-PH-A stands for al ALlegedly PHalse Alarm. Or just fALse-Positive-HA!

(yes, it's stupid, and weird, but has been helping me remember it for nearly three decades now).

andrewla2• 6 years ago

As another commenter noted, the story of the Boy Who Cried Wolf is an easy mnemonic -- the villagers committed a series of Type I errors, and then a Type II error.

jhayward• 6 years ago

I'm reminded of this apt tweet [1]

If a Type I error is a false positive, a Type II error is a false negative, and a Type III error is getting the right answer to the wrong question, is a Type IV error GIVING IMPORTANT CONCEPTS NUMBERS INSTEAD OF NAMES FOR NO GODDAMN REASON AND CONFUSING GENERATIONS OF STUDENTS

Also, I find the Confusion Matrix[2] to be a helpful reference, especially the multi-colored table with formulae for each condition.

[1] https://twitter.com/mjskay/status/1201380151356989440

[2] https://en.wikipedia.org/wiki/Confusion_matrix

henrikeh2• 6 years ago

Remember Peter and the wolf. The villagers committed several Type I errors first and then a Type II error second.

But, yes, a prime example of terrible naming.

andrewla2• 6 years ago

Not "Peter and the wolf", you mean "The Boy Who Cried Wolf". There are no villagers in Peter and the Wolf, and no statistical errors committed.

yourpalkeith• 6 years ago

Thank you for this! May it be shared far and wide until we can stop using these silly codewords.

earthboundkid• 6 years ago

One of the greatest terrible technical names of all time has got to be the philosophers who knew than "intention" was already a word but decided that the opposite of "extension" should be "intension" because WHY SHOULDN'T WE JUST USE HOMOPHONES FOR SIMILAR BUT SUBTLY DIFFERENT CONCEPTS FML

yters2• 6 years ago

So that's what that means! I remember reading a philosophy article using that word and not being able to make heads or tails of it, constantly thinking "how is intension related to intention?"

oarabbus_• 6 years ago

I particularly liked this:

I like that "Type 1 error" and "Type 2 error" are both 4 syllables, just like "false negative" and "false positive". So they aren't briefer than the thing they describe (at least out loud, and even by char count it's not a huge win).

avz• 6 years ago

To highlight the persistence of the confusion compare the confusion matrix in two wikipedia articles: [1] and [2].

[1] https://en.wikipedia.org/wiki/Confusion_matrix

[2] https://en.wikipedia.org/wiki/Pre-_and_post-test_probability...

(In case edits are made one way or the other: both articles display similar confusion matrices but disagree on the names of the off-diagonal elements. In [1] false positives are called "type I errors" and false negatives are called "type II errors". In [2] false positives are called "type II errors" and false negatives are called "type I errors".)

(Edit: layout)

ImaCake2• 6 years ago

I hate the number naming system. I have no idea why anyone would find it preferable to use them for literally any reason. Specificity and Sensitivity already almost mean the right things in common language already, they are perfect words for this. The only use for these is to confuse people so they don't realise you have no idea what you are talking about.

I think some people struggle to grasp that these are actually strict mathematical concepts. But personally, that makes me feel comfortable with them. It is a lot easier to get a math concept consitently right than some of the fuzzier concepts in epidemiology.

yters2• 6 years ago

If I accidentally swap "Type I" and "Type II" in my mind, is that a Type I or Type II error?

ashfromconvert2• 6 years ago

You've just made a Type I error because you've lacked 'specificity' in your mind by swapping the two! :O

Volt• 6 years ago

What's with the shortened URL?

selimthegrim• 6 years ago

Just to mix things up:

https://statmodeling.stat.columbia.edu/2019/03/12/r-package-...

Type I and Type II Errors: The Inevitable Errors in Optimization Experiments

Related Stories