Why Most Unit Testing Is Waste (2014) [pdf]

https://wikileaks.org/ciav7p1/cms/files/Why-Most-Unit-Testing-is-Waste.pdf

by dumindunuwan • 9 years ago

43 24 9 years ago

wikileaks.org

majewsky• 9 years ago

Link is down for me. Can someone give a tl;dr?

jaccarmac• 9 years ago

The TL;DR: I did a quick read, no evaluation here, just what's in the paper.

1. Unit tests only work in "functional" code composed of easy-to-reason-about components that support other similar components. The OO storm introduced a style of code which is impossible to inspect unless it is running.

2. Because Turing machines you need as many bits of test code as bits of real code.

3. Code coverage requirements can only lead to tests for their own sake, i.e. wasted lines of code.

4. If there are many more lines of tests than production code, your developers are quite possibly paranoid about what they're writing, and that's bad.

5. Tests that have not failed in a long time don't convey information, they only provide noise. Optimize your test suite to provide information.

6. The reward of seeing a green bar in an IDE encourages appeasing the test gods instead of designing a good system.

7. Assertions are better than tests.

8. The Agile culture has turned unit tests into something worse than what they could be without it.

mikekchar• 9 years ago

This is such an awesome summary, I'm just going to start here without actually reading the paper :-). I'll say up front that I'm a TDD/Unit test kind of guy. I'm also not generally a fan of London School/Outside-In style (but will obviously use it when circumstances demand it). So I'm probably exactly the kind of person that should disagree with the above. But I don't really disagree that much.

One thing I've noticed is that there really is more than one way to skin a cat. Some people like to have a more linear style in their code. I've found that such people tend to have good memories and considerable ability at reasoning. They can see a fairly large piece of code and see quickly where the errors are. They can remember that one piece of code on the other side of the project did something one way and code on this side of the project does it another way.

These people will often look at my code and it looks disjointed to them. They can't read the code and get a sense of what it is doing. They find themselves jumping around through hundreds of files just to figure out what's going on. If I have any ability at all it's really much more in the spatial relations side. I can visualise networks easily and reason about them.

I can never remember the details about how things are implemented and I'm a very slow reader (I'm dyslexic). I depend on trust. I want to be able to see a function and tell in 5 seconds whether or not it is correct. I then rely on tests to tell me when I have violated some assumptions somewhere -- because I don't bother to learn and have no capacity for remembering all the assumptions in the system.

Unit tests and the intense factoring that occurs when doing TDD (at least in a non-mocking style) favours a style where you admit imperfect knowledge about the system. It then gives you a framework for reasoning about the system. People (even on my team) are often surprised that I haven't used a debugger for at least 10-15 years. Tests are really a misnomer -- they don't verify the correct execution of the code. What they do is probe the execution of the code. It tells me what the code is actually doing and documents my assumptions about what it should be doing. If there is an error, there will be a mismatch in the two. I may or may not have an explicit test that fails during that mismatch, but I should be able to reason about it effectively.

It's a different way to think and a different way to work. IMHO, I think it requires considerably less mental capacity to work in a TDD way. However, on the other side of the coin it requires considerably more training/experience to do well. You definitely want to decide upfront which way your team wants to operate, but I'm not about to say that you're "wrong" if you pick a different approach than I would.

Anyway, thanks for doing the summary! It was very evocative for me :-)

AnimalMuppet• 9 years ago

1. Say what? Unit tests only work in functional, not in OO? That seems highly bogus to me, since I've used them in OO all over the place.

Now, if your object architecture is "throw everything into one object", then OO can make it harder to test (or do anything else). But even on a very large class (5000+ lines - I know, I don't like that it's that big, but it's really hard to break up), I've used unit tests very successfully.

AstralStorm• 9 years ago

I long for a language or library that would be better at contracts than just pre- and post-condition assertions.

Then when assertions fail, you write a regression test. But a unit test that verifies of the contract itself is verified correctly is needed too. More than once such a test actually found that it can be violated under common conditions or does not check what you thought of does. That is the main reason for "unit" tests.

It is even more required for multithreaded or parallel programming.

blatherard• 9 years ago

This paper has been public for years at http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste....

rhizome• 9 years ago

Previous discussion 6mos ago: https://news.ycombinator.com/item?id=12666454

WalterGR• 9 years ago

There have been a ton. For more, click the "past" link.

chillacy• 9 years ago

If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away

And that's how you get regressions. I get the rationale that these tests cover code which probably never changes, and tests are most useful for code that changes a lot. But if the guy who wrote the code left the company a year ago and then you have to change this code... it's going to be a lot easier if there are good tests.

arthur_pryor• 9 years ago

also, how do you know a test didn't fail locally on a developer's machine, causing them to fix work in progress? IME, test failures are only tracked on dev/stage/prod deployed systems.

jernfrost2• 9 years ago

He did cover that. He said specifically that regressions tests are a good thing, but that they are typically not done at the unit level.

sv123• 9 years ago

He means that's how you get regressions of the same bugs popping up in the future, not regression tests.

rst• 9 years ago

Not at all sure how anyone following this advice with, say, a Rails app ever upgrades to a newer version of the framework. When the framework APIs change, you need something exercising the code just to verify that it interoperates with the new versions.

(Yes, apps and frameworks based on a statically type-checked language would catch some of these cases. But not the ones that change behavior of something while leaving the relevant type signatures intact.)

astrodust• 9 years ago

There's a lot of out-dated junk in this latest dump, so do take all of this with a gigantic grain of salt.

jjawssd• 9 years ago

out-dated junk

You just accurately described half of the Intelligence world accurately

gillianlish• 9 years ago

the programming people i admire most just randomly seem to be the same people who never write angry rants about programming topics.

took me 20 years to notice this.

kevinmhickey• 9 years ago

I'm a bit skeptical of a paper written by a consulting company that specializes in QA services and training. Understanding a codebase well enough to write useful unit tests is very time consuming and difficult. Teaching devs the same requires domain, development and process skills. Other forms of testing (e.g. end-to-end, browser based, and manual) are much easier for an outside organization to apply based on rules in a BRD or user manual. It stands to reason that a company would write a white paper that nudges customers to a methodology that favors them.

jernfrost2• 9 years ago

He has some points though. Unit testing has become a bit of a religion, where people almost write tests for the hell of it regardless of their value. Now I do usually test e.g. algorithms, but I find it rather wasteful to test dumb classes at the unit level as well as UI stuff.

I've certainly felt what he pointed out about tests becoming a real hassle to maintain while not necessarily being very valuable to keep.

mempko2• 9 years ago

Cope, the author, was writing about this for a long time. His history precedes him. It's not some random consultant with no experience.

xwvvvvwx• 9 years ago

When I was programming on a daily basis, I did make code for testability purposes but I hardly wrote any unit tests. However I was renowned for my code quality and my nearly bug free software. I like to investigate WHY did this work for me?

This made me giggle. It's like magic. don't write tests, don't find bugs....

mempko2• 9 years ago

Actually he provides much better alternatives. Design by Contract combined with system level tests will catch more bugs than you can shake an editor at.

Why Most Unit Testing Is Waste (2014) [pdf]

Related Stories