100% test coverage

https://brandur.org/fragments/100-percent-coverage

by craigkerstiens • 3 years ago

10 40 3 years ago

brandur.org

sghiassy2• 3 years ago

I don’t get the 100% branch coverage requirement. If the code is off of ‘master’ why would you have to go and fix it?

hinkley4• 3 years ago

Conditional branches, not repository branches.

kristiandupont3• 3 years ago

I think he's referring to branching in the code (if-then-else blocks).

anotherhue5• 3 years ago

I was confused also but decided they meant if/else branching

abdusco• 3 years ago

Maybe the author means something else. How else could you hit 100% of the lines if you're not covering all branches.

hooksfordays• 3 years ago

Branch coverage means something different. If you have a function with 2 if statements, there are 4 branches. 100% code coverage could be 1 test and would run the code before + after the if statements, and enter each if statement block. 100% branch coverage would test this code 4 times: once without entering either if statement, once only entering the first, once only entering the second, and once entering both.

morelisp5• 3 years ago

You can achieve branch coverage of two if statements by running it twice, taking the different result on each the second time. You are describing stronger / orthogonal types of coverage.

sghiassy2• 3 years ago

Oh, so like cyclomatic complexity coverage

bewaretheirs• 3 years ago

Branch coverage refers to branch instructions (in the underlying machine code), not a branch in whatever source control system you're using.

hinkley4• 3 years ago

We run one of our projects not only with 100% test coverage, but 100% branch coverage. It’s deeply unpleasant.

anotherhue5• 3 years ago

I've seen bad codebases rightfully elevated to greatness because of their test suite.

The confidence it adds allows new people to make changes much faster.

DragonStrength5• 3 years ago

I've seen great teams grind to a halt because of edicts.

The elevation of process over results is demoralizing to builders.

jameshart2• 3 years ago

But builder morale is not a product that generates revenue

anotherhue5• 3 years ago

Sure, but I've seen enough premadonnas walk away from a tyre fire that fed their ego to build that I don't take their egos too seriously. Stifling process? No, unwavering diligence? Yes.

hinkley4• 3 years ago

Confidence is only sexy when it’s deserved.

People who jump straight to 100% coverage often cheat, and now we have to manually inspect all of the tests to see if they are actual tests. Sometimes life is easier when you are seeing that there are spots that need more tests because coverage is only 85% in a file.

anotherhue5• 3 years ago

Those people are shown out.

ramraj07• 3 years ago

How do we test for 100% branch coverage on python that covers list comprehensions?

DragonStrength5• 3 years ago

But 100% coverage definitely helps, especially in languages like Ruby where an untaken branch can contain a serious error that’s accepted by the interpreter, but will take down production if hit.

And if you're in a language providing more constraints, the benefits can quickly turn into a cargo cult. This is certainly the mindset of Ruby folks, and such a mindset left a native client team with 0 system devs and all web dev converts who refused to engage with our actual issues and focused only on dogma within 2 years of hiring a manager with that background. Color me skeptical of this applying to all projects.

kristiandupont3• 3 years ago

I don't think I've ever worked with a code base that had 100% coverage so I can't speak from experience per se. But my general feeling is that the cost increases exponentially and the return on investment diminishes exponentially with those last bits of coverage.

Also, I don't think I need to tell anyone here but "100% branch coverage" still doesn't mean "100% branch combination coverage", which would quickly make your test code base grow to the size of the universe anyway.

morelisp5• 3 years ago

The term you're looking for is "path coverage".

kristiandupont3• 3 years ago

So it is, thank you!

Veuxdo• 3 years ago

You probably can get to 100% branch test coverage while being able to tie all tests to business requirements. It'll be expensive, though.

AndrewKemendo• 3 years ago

How do you manage breaking changes from changes in third party APIs and libraries then?

Woof

iron2disulfide• 3 years ago

I started my career off in the chip design world, where 100% line, branch, FSM transition, functional group, and toggle (i.e. individual bits in a bus switching 0 -> 1 _and_ 1 -> 0) was "table stakes". There are a lot of comments in here saying that achieving 100% coverage would be expensive - and they're right. The majority of headcount in modern chip design houses is taken up by verification engineers, whose sole job is to architect, implement and maintain a minimal test corpus that achieves that high bar. The cost of failure is simply too high to omit this step.

It was unsettling to me after moving to a SWE job where coverage was kind of... not as important.

rTX5CMRXIfFG• 3 years ago

I can think of certain types of software that are so critical that they must be 100% thoroughly verified to be logically correct, e.g. software in surgical machines, airplanes, or high-frequency trading. The vast majority of applications, though, would survive with less coverage, though I still think that the sweet spot is somewhere between 70-80%.

pictur• 3 years ago

%100 test coverage is absolutely essential for dependencies that are used in too many places and can affect too many places if something goes wrong. but it is necessary to decide whether it is really necessary to approach all addictions in this way. what I'm talking about is of course valid for a large codebase. And yes, 100% test coverage does not eliminate the possibility of error :D

jameshart2• 3 years ago

How do you get 100% branch coverage in a dynamic language?

Every method dispatch or property access is a branch to an arbitrary piece of code, or a source of an arbitrary error to handle, depending on what object it happens to end up dealing with.

pawelduda2• 3 years ago

I wouldn't worry too much about getting to 100%... With tests the spectrum can be ALWAYS extended to the right side. E.g. you could argue that you should also do mutation testing because 100% coverage doesn't mean tests are sensitive enough to changes in code branches, etc... Once you realize this you never really arrive at 100%

That said having coverage target is more often positive than not having one. Coverage target kind of sends "we care about test suite" message

morelisp5• 3 years ago

I mean, this is the kind of thing I agree with, and would even say I'd trust a project with 90% coverage but no surviving test mutants vs. 100% and lots of mutants.

But in reality I'm usually looking at piles of like... 60-70%ish? Thousands of mutants. Maybe a round-trip fuzz or property test if I'm lucky.

So I keep aiming for 100% and hoping anyone else follows...

pawelduda2• 3 years ago

These are also more expensive kinds of tests to setup and run so it's often not feasible, maybe in extreme cases. In a few Rails codebases I tried to mutation test something critical only to see that getting it to work would take more time than worth the effort

lucumo• 3 years ago

The tradeoff between the cost of many tests and the benefits of them is why I usually use a lower coverage requirement. Something like 80% or sometimes 60% seems to give most of the benefits with less of the cost.

TheChaplain• 3 years ago

well, in my experience 100% line and branch coverage is not enough.

Countless times I see people write tests "to pass", but nothing to make them fail. I speak about null-pointers, invalid dates, strings with garbage, out of bounds indices etc.

nathants• 3 years ago

quickcheck is eating popcorn with 3d glasses on.

100% test coverage

Related Stories