Lean Testing or Why Unit Tests Are Worse Than You Think

https://blog.usejournal.com/lean-testing-or-why-unit-tests-are-worse-than-you-think-b6500139a009

by saranshk • 8 years ago

199 155 8 years ago

blog.usejournal.com

guygurari• 8 years ago

The article makes some fair points for performing end-to-end tests. There is however a benefit of unit tests over end-to-end tests that is not mentioned: code locality. The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. Unit tests typically cover a small part of the code. If a unit test fails, isolating and fixing the bug is typically cheap. On the other hand, if an end-to-end test fails (due to the same bug) then isolating and fixing the bug can be quite costly because there are many different parts of the code that need to be checked. To me, code locality is the main advantage of writing unit tests.

At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.

lbill• 8 years ago

You are right! I work on test automation for end-to-end testing. I dare say my work is very useful and has prevented a lot of bugs from hitting our prod. But at my workplace we also all agree on the fact that it is often quite painful to locate the cause of the bugs that I report, precisely because we don't have enough unit tests on our old code.

Since we live in a world of limited budget and time to spend, I agree with the conclusion of the article: "use unit tests where it makes sense". It is what we try to implement on our new code (and when refactoring legacy code).

mosselman2• 8 years ago

I hear this argument a lot, but I do not think it is valid. What happens is that you simply put in the time before a bug occurs with every unit test that you write. Which isn't very efficient, seeing as not all of your units will lead to bugs. So yes, once you have all of your Units covered 100% it will be easier to find bugs, but you have invested a lot of time in order to get here and in order to keep the unit-test-suite maintained.

Klathmon3• 8 years ago

It's a trade off of when you want to spend the time.

Sure, it's less efficient to write unit tests ahead of time for every case, but in a lot of cases fixing a bug faster when it's discovered is much more important than even double the hours spent during less "pressing" times.

pron2• 8 years ago

But the goal of tests isn't just to find bugs in newly-written code. It's to defend against regressions, and a regression is even harder to localize, given the team may not even be well-familiar with the failing code.

ams6110• 8 years ago

Maybe I'm a worse than average coder, but I find it nearly impossible to write a "unit" of code without also writing one or two bugs along the way. So whether you do your testing ad hoc as you develop, or write unit tests, you do have to spend time in testing each unit of code you write. It doesn't seem to me that formalizing this into a repeatable unit test is adding a lot of real extra work.

eeZah7Ux• 8 years ago

Units covered 100%

This metric means very little. It does not measure the extent of code path coverage and much more.

thephyber• 8 years ago

I have found unit tests are useful for the original developer to very explicitly describe what test cases the code was written for (including parameter data types). The lack of unit tests for specific types of use cases can also signal to the inheriting developer what the code wasn't originally expected to do (which can signal performance limitations).

StavrosK2• 8 years ago

That's exactly what the article is saying. Also, your point about unit tests assumes that tests cost the same to write. Sure, a unit test will have ten times better locality than an integration test, but you have to write ten of them to catch a bug. Also, unit tests don't catch bugs where they're very likely to crop up: the boundaries.

conradk7• 8 years ago

You have to write 10 of them if they are not part of your workflow. If you TDD (which is applicable in a lot of cases in my experience with web development), then instead of testing with Curl or a web browser, you write tests. This takes the same amount of time upfront, so the additional cost of writing the tests is actually 0.

Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.

crdoconnor4• 8 years ago

At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.

IME unit tests work acceptably in one very specific scenario and fail pretty badly in all others. That scenario being:

1) You're surrounding a self contained block of code that interacts "with the outside world" via a code API.

2) That code API is a very stable and clean abstraction.

3) It has minimal interactions with modules outside of it and those interactions that it does have are tightly scoped (i.e. minimal to zero mock objects are required to write the test).

4) The logic of the code is relatively complex and most bugs that crop up are logical in nature (e.g. off by one, things getting swapped around, incorrect calculations, wrong behavior with negative numbers).

Meanwhile, integration tests (at varying levels) work well for pretty much every case apart from this and still work okay for this type of code. They make much more sense as a go-to default.

I've also worked on several projects where there was little to no code that it actually made sense to unit test. It's not uncommon that an entire codebase is predicated mainly on hooking systems together and doing some shallow calculations. IMHO, having zero unit tests in that environment is actually desirable.

The worst unit tests I've seen have been written when two or more of those preconditions have failed. They would fail constantly, require massive maintenance and, somewhat comically, almost never fail in the presence of an actual bug.

Mouse47• 8 years ago

This is spot on IMO. I often see people touting the benefits of unit tests during a refactoring...but 95% of the time refactoring involves modifying class APIs since the hardest part of development is getting the object model right. Unit tests only assist refactoring when you don't modify the APIs - in other cases they are a burden.

scott_s2• 8 years ago

I mostly agree, except that for parts of the code that meet these conditions, I aggressively unit test as the default. To the point that if an end-to-end test uncovers a potential bug in this component, my first action is to write a new unit test for this component inspired by the end-to-end test. If that unit test fails, then I can focus on just the unit test, and not worry about the end-to-end test until the unit test passes.

Another condition I would add to the list, which the component of mine I am thinking about meets is:

5) Failures in this component have cascading effects to multiple other components, causing seemingly "impossible" failures that are not obvious to others that they were caused by this component.

marcosdumay5• 8 years ago

The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear.

That's completely at odds with my experience.

I find that for local bugs, the cost of locating them grows with O(log n) of the amount of code. And for non-local bugs (interface bugs, system bugs, incompatible specs...) unit tests don't catch them anyway.

JamesBarney• 8 years ago

The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear.

I don't think this is true. Fixing a bug is comprised of four parts. 1. Understanding and reproducing the bug.

2. Finding the code responsible for the bug.

3. Coming up with code that fixes the bug.

4. Verifying that your fixing code does not introduce any new bugs.

#2 is the only one that could even theoretically be exponential. The only bugs I've found where I've spent days on 2, were hard to reproduce, intermittent, race condition bugs, which units tests aren't very good at finding anyway.

calyth2018• 8 years ago

At the practical level, I find that most end-to-end test tells you something failed, but not which part failed.

A unit test, or a component test, would tell you that something failed, and it's in this general area, which narrows down the search quite a bit.

They're both useful, but I've seen far more problems with people arguing that end-to-end test is more than what they need, while a bad conversion from seconds to nanoseconds would be caught quickly if a unit test was actually written.

cies2• 8 years ago

Interesting article.

I'm a big fan of "economic testing". Test that are not going to "be economically viable" to write during the dev't of new software, usually do not get my permission to be written in that phase.

Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly. Mostly that functionality can be extracted into a library. For instance a lib that deal with scheduling of events of arbitrary length: we do not want to have issues with those getting messed up, so we try to get 100% unit test coverage of the functionality in that lib.

Another thing with unit test is that I found they mean something different in dynamic languages than in strongly typed languages. In the first unit test are often something to provide confidence that the type system can give you in the latter. Sure both will also test the logic. But this understanding has pushed me to love strong types (as a rubyist), as I can go with less unit test and still be more confident about my code base when it is being heavily worked on. Refactoring becomes so much easier with strong types, as all is "in the code".

When it comes to integration tests I usually want to make a biz case for them. How much does it cost to have it all manually tested, and how much do the automatic integration tests cost to make. While including the benefit of being able to run the automated integration tests at near zero cost (thus several times per day if we need to).

pards• 8 years ago

Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.

I hear this kind of rhetoric a lot, but rarely do I hear of code that doesn't need to work correctly.

poopchute• 8 years ago

Code robustness clearly has different priority levels though. A graphical error is much less serious than a part of your code that could nuke your server.

cies2• 8 years ago

If the e.g. scheduling code is messed up, and we find out 5 weeks in, then there is a lot of damage. We need to call all parties to review their schedules with them.

On the other hand if the code responsible for the styling of some menu is not working correct, the functionality may still work, and once we discover the bug we can simply fix it.

rarely do I hear of code that doesn't need to work correctly.

So there are different grades. That's why I used the word maximum. The cost for these unit test should pay bank, even during the dev't phase, IHMO. There are often areas in a large code base for which this hold true, and often they can be identified as such beforehand; then writing these tests should be a priority.

Fradow2• 8 years ago

The difference is not between code that need or doesn't need to work.

The difference is having code for which having some bugs is acceptable, vs having code for which bugs are not acceptable.

There is definitely code for which having some bugs is acceptable: think about some small feature that the user can totally do without. I had to do one recently, and bugs were acceptable, as long as it was shipped in time, and working for the client specific devices. That code shipped with no automated test (testing is done manually), and a few known bugs not worth solving. On the other hand, anything that touch core features must be tested.

AllegedAlec2• 8 years ago

Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.

I think this is a partial answer.

Yes, I write unit tests for such pieces of software. However, what I also write a lot of unit tests for a pieces of software which are hard to write good integration test for because of combinatorial explosion. If I have a part for integration method which consists of a chain of 4 methods, each of which can have 5 different results, that'd mean I'd have to write 4^5 integration tests, which is clearly not possible. If I were to write unit + integration tests, however, I could do with 4x5 unit tests plus a couple of integration test which prove that the overall combination of those methods work well together.

crdoconnor4• 8 years ago

Combinatorial explosions are best mitigated by doing property based testing, design by contract and creating clean, low level abstractions which you test against.

In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:

* 30 different versions of library A

* 50 different versions of library B

* 25 different versions of the language runtime

* 30 different browsers

That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.

drakonka• 8 years ago

There is an interesting article which responds to this and mentions some misinterpretations of references: https://www.tedmyoung.com/looking-at-tdd-an-academic-survey/

keithnz• 8 years ago

I think that article is a necessary read after the original posted article. The original posted article seems a bit too keen to convince you of its point of view that it seems a bit disingenuous.

My feeling about unit testing / integration test is you do as much of it as you can, trying to sincerely learn the lessons of how to do it it effectively, until you know why you don't need to do it. Or until you know what the right mix is. I started with Extreme Programming and testing around 18 years ago and my thinking has constantly evolved over that time. One of the biggest factors, I think, is project context and the nature of the software you are developing.

I find in embedded systems, unit tests done extensively are very very useful. As is integration testing.

In web systems, I find integration tests are often most useful as so much is all about gluing together various tech, and it's often the gluing the comes unstuck. However there is often chunks that have some interesting logic that lends itself to more extensive unit tests.

Sometimes testing is just a mess because of the design. Too many things that are not really needed, too many layers, too many levels of abstractions, too much "future proofing". Then you see people start trying to throw unit testing around it, and it is mostly just useless and explodes out the amount of work to do. Often I think articles like the original are reactions to this scenario.

jfjfjfjfyy2• 8 years ago

The dogma around unit testing, TDD and the book selling test hype cycles have caused immeasurable damage to software development as technical and scientific disciplines.

bacro• 8 years ago

I agree with you, but could you care to elaborate on that?

jfjfjfjfyy2• 8 years ago

Where to begin.. perhaps the projects with code metrics tied into the check in system - want to commit code changes? You're now on the hook to provide at least 85% unit test coverage or the source control tool will not accept it. So what did most people do? They wrote useless tests, assert true equals true, no-ops or pass statements, basically anything to just get past the arbitrary metric.

Or how about another project that had lots of real tests, that were old, unmaintained, did humongous amounts of work in each single unit test and took a weekend to run, and of course, fail.

Or perhaps the team that obsessed about the testing for each user story, spent all the time doing that, then running out of time to deliver the actual stories.

And those were the less egregious examples!

kozak• 8 years ago

"Nobody got fired for writing unit tests", and this makes it hard to justify writing less of them, especially to people that are exposed only to development side of software business.

rectang• 8 years ago

Meh. Thinking about ROI when writing tests is important, but my analysis of ROI is mostly at odds with this anti-unit-test screed: the greatest ROI comes from writing thorough unit tests against stable low-level sub-components.

Integration tests, in contrast, require attention more frequently -- because not all subcomponents are stable and when one changes, multiple integration tests need superficial readjustment.

I'm skeptical that my own ROI analysis is more "right" -- it probably varies by project and team. The conclusion I draw instead is that ROI analysis is important in general when writing tests, but that one particular ROI analysis is unlikely to apply universally.

2T1Qka0rEiPr• 8 years ago

This article makes lots of good claims as to why integration tests can be the most useful area to focus on. I do however think that unit tests can be really useful in encouraging junior developers to improve their plan vs. write approach to software development (if properly guided).

hdi• 8 years ago

Somewhat interesting but calling an old idea something catchy isn't really revolutionary.

The article makes a lot of assumptions like "In practice, most agree as most projects set the lower bound for coverage to around 80%". Most projects where, which industry, who are these 80% that agree? This is just taken out of thin air in this case. It might be true but should we take the author's word for it?

  There are also some psychological arguments. For 
  example, if you value unit-testability, you would prefer 
  a program design that is easier to test than a design 
  that is harder to test but is otherwise better, because 
  you know that you’ll spend a lot more time writing tests.

This is very subjective without examples. The opposite argument can also be made that code which is easier to test is sometimes better. It felt like reading a collection of quotes and articles by other people.

I think TDD, unit, integration and E2E testing works. How much of which to apply is entirely project and industry specific and it's up to teams to decide what testing strategy works best for them.

LoSboccacc• 8 years ago

"The outlook for the effectiveness of testing used by itself is bleak. Jones points out that a combination of unit testing, functional testing, and system testing often results in a cumulative defect detection of less than 60 percent, which is usually inadequate for production software."

https://medium.com/@TuckerConnelly/94-gems-from-code-complet...

personally I like write some cursory unit test because it ensures the code logic is correctly decoupled from data retrieval, but most of my unit tests are written from user reports, as part of a large no regression suite.

sverhagen• 8 years ago

There's two things that humble me on a daily basis, in no particular order: collaboration and testing.

While I use all sorts of testing, I still write a good amount of unit tests. For whatever kind of tests I write: not a day goes by without tests or the act of writing them exposing bugs that would otherwise have moved on to the "next level". Doesn't mean they'd have ended up in production, but they'd still have been a lot more expensive to fix.

IanCal• 8 years ago

Integration tests lie somewhere between unit tests and end-to-end tests so they provide the best balance. Therefore, they have the highest ROI.

Being in between does not mean it gives the highest ROI or the best balance. They can easily share the worst of both.

Lean Testing takes an economic point of view to reconsider the Return on Investment of unit tests.

But does not in any way actually try and work this out, which means that you can't make the conclusions that are in this.

The Return on investment (ROI) of an end-to-end test is higher than that of a unit test. This is because an end-to-end test covers a much greater area of the code base. Even taking into account higher costs, it provides disproportionally more confidence.

And yet no figures or evidence.

Many e2e tests can cover the same path segments, meaning adding a new test may not increase your confidence by much at all. But they're still taking longer to run, and despite the articles insistence that integration and unit tests are brittle, I've found E2E tests are also brittle, just in different ways.

I shouldn't have to change my string normalisation tests just because the website changed moving the resulting output field to a new place on the page. They shouldn't break because the login flow changes.

I have definitely seen (and written) E2E tests that provide almost no real value, and unit tests that provide huge amounts.

Plus, end-to-end tests test the critical paths that your users actually take. Whereas unit tests may test corner cases that are never or very seldomly encountered in practice

Unfortunately low proportions of errors still can account for extremely large actual numbers of problems. 1/1000 bugs will happen all the time if you've got a reasonable number of users. Those bugs also may happen to most of your users if they're somewhat random, or worse may heavily impact one group.

For many products, it is acceptable to have the common cases work but not the exotic ones (‘Unit Test Fetish’). If you miss a corner case bug due to low code coverage that affects 0.1% of your users you might survive. If your time to market increases due to high code coverage demands you might not survive.

Or if roughly one in 1000 customers keep taking your production server down due to a bug you might not, and you may have been just fine waiting another week to go live.

I agree with the idea, you should consider what it is you're trying to achieve and what the best ways of doing that are. What you absolutely shouldn't do is use terms like ROI and economic without any analysis, purely to justify not doing something you don't really want to do.

edejong2• 8 years ago

Edsger Dijkstra: "The first moral of the story is that program testing can be used very effectively to show the presence of bugs but never to show their absence." [1]

"I suggest that the programmer should continue to understand what he is doing, that his growing product remains firmly within his intellectual grip." [1]

[1] E.W. Dijkstra: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...

nradov3• 8 years ago

The trouble is that advice isn't really actionable. In the real world of commercial software often the only way to satisfy customer requirements is to build a product more complex than the programmer can really understand. Most customers would rather have buggy software (which eventually gets fixed) rather than no software at all.

edejong2• 8 years ago

But it is actionable. Each and every time you chose a certain technology (FP over OOP, immutable vs. mutable state, static typing vs. dynamic typing, shared vs. specific terminology), we make a choice. These choices together drive our community towards a certain mode of programming.

So, your argument is true for individuals, but for the community it is void. Together we should focus on identifying correct, effective, simple and reliable building blocks on which we can build our larger systems.

Perhaps you haven't seen "Simple made easy" by Rich Hickey, which illustrates this concept rather well.

djhworld• 8 years ago

I'm working on a ticket at the moment that involves a lot of refactoring work to support a new feature.

In the initial run on this code base we went a bit unit test crazy, which was fine at the time, but I'm seeing the drawbacks now where I'm doing a lot of surgery on the components.

I've ended up just beefing up the current integration tests with some extra edge case testing and, after the refactoring work, they still pass, which has given me a lot more confidence.

I think a lot of this comes to head when you unit test the glue logic that orchestrates the entire program, unit testing those becomes painful with mocks etc because they become fragile whenever you do any refactoring work.

I'm not sure what to think really.

paulmd• 8 years ago

Integration testing and unit testing aren't direct substitutes. You would never ever say "I have unit tests, I don't need to run the product before shipping it". Furthermore, automated integration testing is strictly less capable than a human doing integration testing.

Unit tests also have the advantage of being cheap to set up and cheap to run, so their limitations in whole-system testing are offset by the fact that you can have many, many more of them. You can easily blow through a couple thousand unit tests in a matter of 10 minutes or less, whereas even a modest battery of integration testing will be an overnight process at best. That means your OODA loop is a lot faster on unit testing.

(the ideal is of course structuring your application so you can deploy many tests in parallel, but that's not always possible.)

Fixing Selenium so WebDriver instances don't take 60+ seconds to launch would be a major help. Until then, integration tests that work at a controller level (minus the UI) are probably the best compromise between speed and integration. You can pass in a mocked request as input, and assert on the output. Runs much faster than a "true" integration test with a driven browser client, but hits the whole stack better than a unit test. They are also really easy to set up, since you can just have a tester or an integration test go through the motions and then dump request data to JSON, then feed it back as a test. The lack of state persisted between requests makes implementation a lot simpler.

discreteevent• 8 years ago

"Unit tests also have the advantage of being cheap to set up and cheap to run".

They are not cheap to develop and maintain however. And given enough developers a certain percentage of them will completely distort the design of the code to enable unit testing making the code itself much more complex.

luord• 8 years ago

I'm having trouble thinking of how unit tests are "expensive to maintain"; you don't really need to do anything other than having them there.

If it's because you need to change them after a feature changes or breaks, isn't that the entire purpose of testing and not exclusive to unit tests?

simula67• 8 years ago

I have seen statements like 'even bad tests are better than no tests'. I feel that for unit tests, 'no tests are better than bad tests'. Failure in unit tests often break CI and I have seen many tests which tests the structure of the code and not it's intent. For example, I have seen some tests where the product code is putting some element to a list and the test code checks if the element is present in the list. I can understand why people do it ( they are trying to improve "code coverage" ), however if tomorrow you refactor the code to not put that element in the list, but still accomplish the same thing, the test would break. Often, this breaks CI also and now you are sitting late at office trying to figure out what went wrong so you can push your change into the release. This is valuable time that could be spent on new features, fixing bugs or just going home and relaxing. End to end tests ( or integration tests ) are better since they are real user interactions.

If a test fails and it breaks CI, there should be a problem in the product. Move everything else to some "additional testing" bucket and not block developers.

irundebian2• 8 years ago

I think that's not right way to test units. You should test the interface resp. the right execution of the specification of the unit and not testing implementation details.

That means you're either testing the return value or testing the inputs which were modified by the unit.

If the behavior of a unit changes in such a way that it does something else with its inputs, it's a specification change which requires other test cases.

nradov3• 8 years ago

This is why you need coding standards for ensuring good quality unit tests, and enforce those standards through rigorous code reviews. Developers shouldn't be able to check in crappy unit tests on the release branch.

nimblegorilla• 8 years ago

In an ideal world developers never push something with broken tests to CI. When developers don't run the tests locally it seems like a sign that there are too many bloated integration tests and not enough fast unit tests.

pshirali• 8 years ago

I find this article misleading; with a biased PoV. The author says "empirical evidence is missing", yet advocates a testing methodology, banking largely on articles and PoVs of other individuals. Is there any software project the author would like to quote as a case-study? Is there any evidence to prove that the Lean method is better OR, the definition of what is better?

It is OK to be economical in what one would see as a cost of testing a product end-to-end. Call it whatever term you like, but functionally testing a product end-to-end has always followed an economical approach, as all such testing is a factor of time & resources. You'd perhaps be less economical if you were testing life-safety systems, but you could choose to be more economical if you considered your software to be not-that-critical, or, you needn't have to set exceptionally high standards.

I find it extremely odd that unittests are even considered into this equation of 'testing-costs', despite decades of improvements in software development processes. Unittests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unittests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future.

It's a sad state of affairs if one gives code 1st level treatment, but treats unittests, documentation, build scripts and other support infra as something less important; worth economizing. This is really what _matters_ when it comes to overall quality of a software.

One must remember that software is always improved, refactored, expanded, ported or worked on in some way or another. When unittests are missing, then the very boundaries that were meant to dictate the rules of the software don't apply anymore. This leads to human errors causing portions of software to break.

Critical pieces of software that exist today, exist strongly because they were engineered right (Linux kernel for example). Not because their developers followed an _economical_ approach to testing. If an OS disto claimed to perform economical end-to-end testing (of a potential user's most commonly performed paths), would the author of this article want to use that OS over one that has had strong ground-up unittests and testing of interfaces where they matter?

Brotkrumen• 8 years ago

His claim of lacking empirical evidence shows his lack of research on the subject. There's tons of research that consistently shows comprehensive testing during development makes your product better and cheaper over its lifetime.

Just a quick google and not even the paper I was thinking of: https://softwareengineering.stackexchange.com/questions/6050...

mdip2• 8 years ago

I'd love to see some research around time cost in this area. My personal experience is that writing unit, integration and end-to-end testing ends up either being a wash or reducing the total time it takes to build a product[0].

The rule of thumb I always come back to is "the minute you decide it's time to attach a debugger, be prepared to burn an hour." I used to not mind firing up the debugger -- heck, I used to run my code with a debugger attached every time I ran it. These days, I write tests. I don't aim for 100% code coverage, though I aim for a majority and 100% coverage of critical paths. When I encounter a problem that isn't covered by a test (rare), I write a test centered around the problem. This usually surfaces the bug before I've even ran the actual test -- and once that test is there, any refactoring that affects that code will avoid that bug if the test passes.

This rule has been so effective for me that I've now gone through four different teams, encouraging each to do "the next project" with automated testing. The result has been a commitment to working that way from that point forward[1].

[0] Assuming some baseline quality requirements -- automated build, and a general requirement to publish a mostly functional product. :)

[1] I can't take full credit for the idea -- it was a blog post that convinced me to try it on myself and I advocated that idea after it convinced me.

danielovichdk• 8 years ago

There is a test for that.

Try building a tennis game, just in code with no gui, and take note of how much you would either write tests for things like the score, set points etc., or how much time you spend debugging to justify if your code works.

Unit tests are a great documentation tool as well, both for developers but also for the business needs of the software — the value.

usrusr• 8 years ago

Another rule of thumb I like to apply when mix-and-matching different levels of tests: how visible would a bug be in higher level tests? If things will fail in obvious ways, a unit test will give no benefit besides narrowing down the bug location in case of a failure.

But if symptoms would likely be too subtle to stand out, e.g. a typical bug would manifest itself in results that seem plausible despite being wrong, then it's a strong indication for unit tests.

irundebian2• 8 years ago

Time-to-market is the only valid argument for writing crap code and not unit tests. I'm not sure if you can seriously argue with economic reasons when talking about testing. The most costs in software development arise in the operations/maintenance phase and are probably (there are much papers on this topic) multiple times higher than the costs of the implementation phase.

I agree that every software development team has to make their own decision to balance development time and maintenance cost but I don't agree with this low entitlement mentality of writing correct code in this article.

Of course there are also costs when implementing tests and your unit tests are rendered useless on design changes. But so what? You're also decreasing code assurance and it's better to increase it with tests so that you won't encounter bugs in production which is much more expensive than writing tests.

Software development is not only about adding functionality but also about ensuring its correctness.

rc_bhg• 8 years ago

sometimes (often times) unit tests make for crap code.

GuB-42• 8 years ago

It reminds me of a project I worked on. It was for critical embedded software in aircraft ("make an emergency landing" critical, not "we are all dead"). Obviously, the code is certified, with specific requirements (DO-178B level B) and third party audits.

And we made the choice not to have unit tests. Everything was done through integration tests. What made up for it is that we had very good traceability. So for we knew which lines of code were related to which requirement and which test. So if we changed some code, we knew which test to run, and if a test fails, we knew where to look at.

For the edge cases, we had the ability to inject data directly into memory, triggering the otherwise unreachable error conditions, and the very last cases where done through code analysis (we needed 100% coverage).

h0l0cube• 8 years ago

I'm a huge fan of the following essay. It provides a set of very rational, well considered and compelling arguments for less unit tests:

https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...

actionowl• 8 years ago

Unit tests are better than no tests.

karmakaze• 8 years ago

The highezt cost of unit tests is when they are refs tired with code. The value of a unit test is greatest when written before the implemantation. When refactored it is usually just made to pass by any means necessary to stay green without much/any thought. They are also brittle in these cases where refactoring the test can take meant times longer than the code it tests.

My advice is usually to do tdd for new dev have good coverage of full function tests and delete unit tests if the get in the way/lose value on refactoring. No one likes deleting tests but they must add value or its just more debt.

peterwwillis• 8 years ago

There are two kinds of end-to-end tests: 1) the software works, 2) the software works in production. The difficulty usually lies in implementing the latter.

shinylane• 8 years ago

For me this depends on what I'm working on, and these days my unit tests are generally for API regression. When I'm writing my APIs, generally my "integration" test would simply involve an external request and assertions of the response. This is fantastic for testing what a client expects, not as good at specific units of business logic where regression can be rather expensive. I would agree many folks take unit testing too far, but in some cases it's saved my neck at a pretty small cost of creation.

This is probably not the same story for something like a react or swift application. Good read, thanks for the solid points.

mdip2• 8 years ago

Great article and I'm somewhat surprised at how often I have to repeat the things mentioned in it -- heck, sometimes it's difficult to even convince developers of the value of unit testing, in general.

I'm a huge advocate of automated testing and with the available tools, like docker, it's relatively painless to get the pieces together to sort out automated testing. Often the tooling you use to enable automated testing is tooling you end up needing anyway -- it's dual purpose. Before the "first run" of a set of code, I'll create Dockerfiles that make up a complete, local, development instance of an application along with some boilerplate tooling that I include to make debugging easier on me. When setting up the production build, the final version is usually this same environment with fewer lines in the file. Because my environments tend to be similar, I have a zsh script that strips out lines in the Dockerfile to get me 90% of the way to a production container.

For me, it's always worth it. I came to this conclusion after spending a few months forcing myself to test rigorously[0], starting with unit tests written often and early and ending with a small number of integration tests and a much smaller number of end-to-end tests. I don't find any of these particularly difficult to write.

The benefits, however, are vast: (1) Avoiding the debugger time-sink: The #1 thing that I always come back to is that I generally end up never having to fire up a debugger. I noticed that every time I encountered a bug in code that was poorly covered, the first instinct was to attach a debugger and peek at locals to see what was going on. This rarely resulted in spending less than an hour troubleshooting. Sometimes you get lucky and you find more than one issue in that debugging session, but often it landed in at burning an hour on ever bug and way too often it was an hour spent debugging production code and the bug was customer impacting[1]. At the same time, it's rare for an automated test to have a time cost that high. (2) Refactoring - Since "premature optimization is the root of all evil", that necessarily means that a performance bug is going to involve injecting complexity into a running codebase and this often comes with high-impact refactoring. Unit tests, specifically, are incredibly helpful here. This is often an argument against integration/end-to-end test automation since refactoring regularly breaks these brittle tests, however, I've found in practice that this isn't the case at least half the time. Of the times that it does affect those tests, the practice of refactoring can surface subtle bugs (on a few occasions it surfaced a subtle race condition that might have been missed if a few of the integration tests covering a subset of the functionality hadn't broken). (3) Design - more for integration and unit tests, thinking about testing while writing code can result in a less brittle design[2]. On integration tests, it means writing SQL scripts and migrations to ensure that a fresh environment can be spun up on-demand instead of using GUI tooling (or, using the GUI tooling to generate said scripts/migrations). (4) Build automation - I'm somewhat surprised at how often I encounter a customer project where I have to follow a 20 step process to get things functional in a development environment. It seems like if CI isn't involved, people figure a README.md with a mess of shell commands and button clicking is OK. Scripting out environment configuration and build was already one of the first things I did when I began running the code I'd written, however, I find I no longer have to argue in favor of this when testing is involved -- everyone wants a single command to execute tests and once integration and end-to-end tests are involved, it just makes sense to add standing up the docker parts, too[3].

I get why there's resistance to doing these things -- getting people to simply write any automated tests seems to be the most difficult hurdle. Throw in "learn docker" and other technologies to make automated end-to-end testing easier and the barrier is even higher. And hey, there are some times that the time spent writing tests doesn't pan out to a time savings. For the unconvinced, I can only strongly recommend: try it on your next big project. There's no need to change the way you think -- skip TDD if it doesn't work for you -- but write unit tests over your, public facing surface area. Write integration tests over the most important parts of your codebase -- those which if a bug were to be encountered would have the greatest impact on reliability. Write a few end-to-end tests of major functionality. Keep track of the time spent from the first line of code to the final, released, product. If your experience is like mine and the 4 different teams I've done this exercise with, you'll end up doing things this way from that point forward. If not, you have a gift that I lack -- you write incredibly bug-free code "the first time", every time. :) Then try switching to a single 1080P monitor[4].

[0] I tend to code first, test second. Though, on paper, TDD looks like a good idea in that it forces you to think about the desired outcome and write methods in a manner that guarantees the ability to test, I don't find it difficult to write things in this manner from the beginning and I find it more natural to write the actual code first and I'll often write a large footprint of code before writing the first test -- I don't find as much value as others in frequent, instant, feedback but I recognize the value that others find in that.

[1] And bugs resolved during an outage are duct-tape "there, I fixed it" kind of repairs.

[2] Provided you don't like having to figure out new and creative ways to mock complex god-objects/routines. Maybe you dig that sort of thing?

[3] I reload enough that I have a script that automates installing and configuring docker in the most common manner if it's missing or the configuration isn't complete.

[4] https://news.ycombinator.com/item?id=14482587

pletnes• 8 years ago

The article argues that you should test larger units (integration tests) rather than unit tests, but also that you should use a statically typed language. This leads to less flexibility and more «ossification» in my experience, much like the overly detailed unit tests. Seems to be a contradiction, in my opinion.

swaggyBoatswain• 8 years ago

I started unit testing for the first time several months back. I tried incorporating it into my general workflow

Here's an isolated overkill example of how unit tests should not be done on one of my codepen files (its a javascript calculator). All tests passed but the calculator is partially broken still. I made about 20 unit test case scenarios when in reality it should have been significantly less (~ about 5 given the complexity of the problem)

https://codepen.io/vincentntang/pen/XqNGqv?editors=1100

I ended up getting very little useful feedback on codereview / stackexchange, but this is how I felt about unit-testing in general.

https://codereview.stackexchange.com/questions/193128/shunty...

I think unit testing is great in some applications (e.g. working with teams as more things can be missed, where a bug can be exponentially cause issues, systems with larger complexity). Doing it in isolated solo projects is such a waste of time IMO, good commenting / structurally organizing code/ naming conventions suffices here. Most things don't work in a functional way anyways that makes unit-testing easier. E.g hardware calculators don't have logic the way I wrote it in my code above.

Unit testing is OPPURTUNITY COST. For a business to implement this the risks of exponential failure and its demand for risk-mitigation should far outweigh the time and developmental practice of implementing it in your training / management / workflow / toolchain.

...disclaimer: I might have no idea what I'm talking about I'm a fairly inexperienced dev

marcosdumay5• 8 years ago

You won't get very far without tests. You will notice it, once you start to create larger programs your productivity will decline. That happens even if you use safe languages and verify your code.

This article is to be about the kind of test you should use. It's just that unit tests are often not powerful, but always expensive. So, you should use other kinds of tests.

billsmithaustin2• 8 years ago

"This is because an end-to-end test covers a much greater area of the code base." I disagree. A single unit test will cover less area than an end-to-end test, but unit tests in aggregate will cover a greater area.

marcosdumay5• 8 years ago

but unit tests in aggregate will cover a greater area.

If integration tests can't cover some part of your code, it's because that part does not matter. Get rid of it.

billsmithaustin2• 8 years ago

I agree that some scenarios are more important than others, but that seems to be orthogonal to the claim that I was objecting to.

I also disagree that whether an integration test can cover some part of your code is the litmus test for whether the code is relevant. There are types of errors that are relevant but complicated to simulate in an integration test, or at least in an automated integration test.

An example would be a network partition. It's involved to code a test to cause the partition, wait for the various timeouts to take place, then heal the partition, then wait some more for things to start talking and recovering. And depending on who owns the test environment and how your organization is managed, you may need to negotiate with a DevOps / Operations / Security person to get permission to run iptables in the appropriate machines.

On the other hand, unit-testing what happens when a connection fails is probably pretty easy.

Jtsummers2• 8 years ago

It isn't always feasible to introduce certain error conditions with integration tests (depending on how much of your system you're integrating, whether faults can be injected at that level or not). You often need unit testing to ensure that your code handles those failure modes correctly. That code can be critical to a system, but hard to exercise directly.

crazygringo2• 8 years ago

Unit tests also help ensure both the coder and readers understand what the function's defined behavior is supposed to be, 100%. They help with clear thinking to ensure you write the code correctly in the first place.

dalore• 8 years ago

This is making the assumption that writing tests takes longer to produce code then test without. But it has been shown that TDD can and usually does produce code faster with it. TDD makes you write code that's easier to test, the tests are fast (mere ms). And it gives you confidence in your code, plus locality as you know the error since the tests either pass or fail each step.

Not advocating writing 100% test coverage, but just some code that shows how you will be using the new feature as you add it is enough to help you structure your code.

mamcx• 8 years ago

I write some test when starting, with the full acceptance most, if not all, will be abandoned. Maybe comment out all the ones I don't need anymore or just are too broken (because are invalid). So tests are just another way to do exploratory programming...

Most are related to DBs. I NEVER mockup. NEVER.

NEVER.

I just rebuild the DB each time the tests start (and is PostgreSQL. I found that DROP/REBUILD the SCHEMA is fast than drop the database and rebuild, so is ok).

This always leave me with all the data in the DB, no matter it fail or not. This is the main reason to never mock. Mock data get lost after the tests end. The db instead persist the data and allow to inspect after the fact.

Plus, fill me with the data necessary to let the UI run without re-create again here.

This simple rule is what help me most. If I wanna the test to run faster, I can't do anything else that write a good database layer and not call the DB when not needed...

marcosdumay5• 8 years ago

Once you abandon the idea of unit testing everything, plenty of possibilities open up.

I have a mail server that I test by cloning a root (that's in git) and chrooting there. I also have created a language for testing network protocols once.

thecleaner• 8 years ago

Perhaps this approach of end to end testing is great for startups. Although e2e tests can be hard to write, they can be a significant boost when moving with high velocity.

rbongers• 8 years ago

Your testing strategy should be situational.

Unit tests can efficiently handle, for example, libraries, error handling which cannot normally be reached, complex algorithms which are not AI based and require a large number of test cases, fragile pieces code and code patterns, and program design patterns which can be tested using inexpensive common tests. Consider the business value of quick tests in each of these situations. It is high.

While your units have to come together in a way that allows your users to actually use your software, your business logic has to be represented in whole or in part by one or more unit. Testing this logic earlier on in the process and faster than other types of tests is better.

emodendroket• 8 years ago

I'm not sure if I agree with having a lot of DB tests, but at the same time I definitely have seen the thing where overly specific unit tests make it so painful to change anything that nobody wants to do it.

I think "Why Most Unit Testing is Waste" is a better article on this subject. https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...

guru4consulting• 8 years ago

I guess it also depends on the nature of tech stack and business domain. For e.g., if you develop a standard CRUD application without much complexity, and use a static lang like Java, you probably need less unit tests. If you use framework features, annotations, validation expressions, then most of the simple data validations (like numbers, dates, future date check, etc) are automatically handled at framework level. In those scenarios, you would generally need a very little unit test case coverage.

On the other side, if you use dynamic lang (Ruby, Javascript, etc), or if you are developing a library or framework, then a good coverage for unit tests is a must. I guess the developer's context (whether he/she is developing a framework, library or a simple CRUD app) plays a bigger role.

theptip• 8 years ago

unit tests increase maintenance liabilities because they are less resilient against code changes. Coupling between modules and their tests is introduced! Tests are system modules as well.

Every time I hear this kind of statement, I think of Bob Martin's article: https://blog.cleancoder.com/uncle-bob/2017/05/05/TestDefinit...

The author tells of how his unit tests are fragile because he mocks out all the collaborators. (sigh). Every time a collaborator changes, the mocks have to be changed. And this, of course, makes the unit tests fragile... the author of the above blog has given up on badly written micro-tests, in favor of badly written functional tests. Why badly written? Because in both cases the author describes these tests as coupled to things that they should not be coupled to. In the case of his micro-tests, he was using too many mocks, and deeply coupling his tests to the implementation of the production code. That, of course, leads to fragile tests.
What the blog author does not seem to recognize is that first class citizens of the system should not be coupled. Someone who treats their tests like first class citizens will take great pains to ensure that those tests are not strongly coupled to the production code.

Testing is hard for a number of reasons, not least because given five engineers you'll get six definitions of Unit vs. Integration test. I think the term "micro-test" is useful as a contrast; most beginners sit down and write tests for every significant function, be it public or private (i.e. they write micro-tests), and this produces tests that are tightly coupled to the shape of the code at the time of writing.

However if your code is well-encapsulated, and just tested through the public API of your class, then you should be able to make fairly significant changes to the structure of that class without actually changing the tests. (Of course if you change the API then the tests must change; it must always be so).

kazinator• 8 years ago

I think I can articulate what I find distasteful about approaches like TDD: it's basically a "how to program" recipe for people who have absolutely no talent. Like "paint by numbers". It's tone-deaf toward the creative aspects of development: the Art, if you will.

TDD for music composition: add some notes which jar the ear (red test). Then massage them until they sound pleasing (green test). Make some adjustments for better flow with surrounding notes (refactor). Now repeat. Gee whiz, you're gonna be the next Mozart or Bach!

crazygringo2• 8 years ago

I may never have disagreed more with a comment on HN than this one.

TDD does absolutely nothing to hinder your creativity. All it does is help you to think about your code's behavior more thoroughly and catch inadvertent bugs. Nobody has so much talent they don't write bugs.

For music composition: composers generally start out with an idea of what they want to achieve, whether at the scale of the symphony, movement, section, passage, phrase, or bar. Then they try different solutions until they satisfy the idea they want to achieve. And ensure it flows with the overall structure. And repeat.

Beethoven didn't write out symphonies fully formed. He spent months and years experimenting with ideas until they satisfied what he wanted, e.g. the "test".

throwaway5752• 8 years ago

This is ... not even basic test engineering 101. This is remedial: understand the testing pyramid.

If you can take a 5ms unit test and verify with 80% confidence for a given feature, run that first before running at 5s system integration test that verifies it with 100% confidence.

You run both, because you run unit tests with a build/linting stage (something that can run synchronously with every commit, or close to it) and you run the integration test less frequently because of time/cost constraints (stable branch merges).

Even though they test they same thing, they serve different needs.

lucisferre3• 8 years ago

I kind of lost it at this point:

Kent C. Dodds claims that integration tests provide the best balance of cost, speed and confidence. I subscribe to that claim. We don’t have empirical evidence showing that this is actually true

And

There is the claim that making your code unit-testable will improve its quality. Many arguments and some empirical evidence in favor of that claim exist

Why rely on evidence when conjecture an opinion are so much easier?

So here's my own limited evidence to toss on the other pile. Our team has started to unit test everything, 100% coverage. We don't take this approach to testing lightly, it is a holistic approach that includes the code itself, focusing on architectures that enable testing and testability. Doing so has vastly improved code quality in measurable ways.

Code reviews are faster and easier as a starting point. It is easier for another person to understand quickly what each component does and what it is expected to do. Code is composed using small autonomous functional components that are uncompleted and easy to reason about and test. These make debugging easier because they can be isolated to simple components and tests. Bug fixes are faster to implement and test since the test infrastructure is already in place and does not need to be remediated in later.

Yes, we do also write integration tests. They tests the end-to-end scenarios we are designing the system to support. But unit testing is still the primary approach to how we design, while integration provides support by verifying complete scenarios.

However our approach admittedly doesn't have "Lean" in the name, so that's one downside I guess.

newgame2• 8 years ago

Software quality can be measured in different ways. Software quality is not the only dimension to care about when developing products.

lucisferre3• 8 years ago

So what dimensions do you propose to care about? Do the dimensions I mention not matter at all? If they do matter then I think there is still a point here.

You are engaging in a strawman. I don't claim anywhere that these are the only dimensions to care about, I simply claim that these are some things we've noted that testing leads to improvements.

Ultimately though my primary claim is simply that I see less value in opinions vs evidence.

Jtsummers2• 8 years ago

Imagine you have three components, A, B and C. You have written an extensive unit test suite to test them. Later on you decide to refactor the architecture so that functionality of B will be split among A and C. You now have two new components with different interfaces. All the unit tests are suddenly rendered useless. Some test code may be reused but all in all the entire test suite has to be rewritten.

That's not a refactor. That's a rewrite or rearchitecturing. Of course you have to redo unit tests, you've eliminated the units that were being tested.

What if you decouple two components that are always used together.

If they're always used together, then they were a unit. Why were they decoupled? See the other front page post about Dijkstra's parable for a rather apropos example of where coupling (perhaps not his intended moral) makes sense.

miga• 8 years ago

I have also been advocating economic approach to testing, but I find this article to be misleading at best.

Long-time experience of debugging proves that untested code is worthless. And indeed most people start programming by testing some test case or use case.

I would rather recommend starting with use case testing. Data driven testing.

Never believe that you can get a code without testing. I guess I finally need to write the article about my diamond testing routine, so that people have something better than unit test swarms ;-)

Lean Testing or Why Unit Tests Are Worse Than You Think

Related Stories