Who's testing the tests? The case of an interesting false negative

https://www.nu42.com/2021/09/who-is-testing-tests-false-negatives.html

by nanis • 5 years ago

2 2 5 years ago

newbamboo• 5 years ago

If tests are widely used we really don’t need to worry about false negatives. Self correcting problem. False positives are also not a problem. Tests should be ubiquitous and cheap. They are not a thing to fear and over regulate.

nanis• 5 years ago

false negatives. Self correcting problem.

A false negative is something that is broken, but you are confident it is not. It will not be known without either 1) someone taking it upon themselves to study test code/output which no one really is incentivized to do if all tests show green or 2) something going wrong at an unanticipated point that forces you to seek answers in the you spot the problem.

So, tautologically, a false negative cannot be self-correcting.

False positives are also not a problem

A false positive is a test that fails even though the code is good. Fixing it requires one to prove that the code is good and the test is wrong which tends not be so simple (if it were, the test would not have been wrong in the first place). That takes effort. Therefore, still not self correcting.

Tests should be ubiquitous and cheap.

Sure. Correct tests which pass if, and only if, code is good.

If you think that it is OK to expect some X% of tests to fail at any time, you have my condolences.

They are not a thing to fear and over regulate.

Tests are code. They should be treated as such. Since all code has bugs, tests have bugs. These bugs themselves can impact the ability deliver, maintain, and sustain code that solves the business goals.

If you think identifying bugs in tests is "over regulation", then we don't really have a common understanding of the situation.

Who's testing the tests? The case of an interesting false negative

Related Stories