It's OK Not to Write Unit Tests

http://blogs.msdn.com/b/cashto/archive/2009/03/31/it-s-ok-not-to-write-unit-tests.aspx

by raganwald • 16 years ago

105 47 16 years ago

blogs.msdn.com

johnswamps• 16 years ago

Original discussion: http://news.ycombinator.com/item?id=922715

nopassrecover• 16 years ago

Favourite paragraph:

"If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes in and replaces "<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does."

fleitz• 16 years ago

Everyone knows that the best way to program java is via a DSL called XML. The .java files are just an illusion.

njharman2• 16 years ago

I don't understand at all. Why would the tests still pass?

z92• 16 years ago

That's the horror.

metamemetics• 16 years ago

Because they are arbitrary and not a one-to-one correspondence with what they purport to be testing. One false positive and everything could still be broken.

scorpioxy• 16 years ago

Never tried it. But its probably because unit tests can only tests things you explicitly test for and if you don't have your corner cases covered, then these changes wouldn't blow up your tests. Someone correct me if I'm wrong.

Not a fan of unit testing my self. I feel the approach lacking.

baddox• 16 years ago

It shows that green bars on all your tests don't necessarily mean anything. All too often, developers don't test for the cases that may actually fail (because, obviously, their actual code is only correct for the cases they're cognizant of).

WorkerBee3• 16 years ago

Because the author wrote poor tests and thinks that you did too.

jacquesm2• 16 years ago

Normally they really shouldn't, but if you have a problem writing test cases (which probably means you have a problem programming to begin with) then you may not be testing with enough variation in your input to detect corner cases, tests that should come up false etc.

An example:

  int fun(int a, int b) {
    if (a >= b) {
      return 1;
    }
    return 0;
  }

  now you could test this by saying:

  fun(4,3) should give '1' as the output

  and 

  fun(3,4) should give '0' as the output

But you'd miss to test for equality, with

  fun(3,3) which should *also* give 1 as output

So you'd need an extra test for the '=' part of the condition. Replacing '>=' with '>' will still pass the first two tests, so if you forgot about the third you will not even realize something is wrong until your code starts breaking in mysterious ways.

Unit tests have to be just as good as the code you wrote.

aarongough• 16 years ago

If you're a Ruby developer you should check out Heckle:

http://glu.ttono.us/articles/2006/12/19/tormenting-your-test...

Heckle is a mutation testing framework for Ruby. I don't use it all the time, but my tests are definitely better now that I've tried it and seen what some of the edge cases are...

bh42• 16 years ago

So basically what he's saying is that super naive unit tests which barely even test the happy path are hardly better then nothing.

I fully agree, but I am getting tired of all of this slam down style, flame-bait headlined, writing.

gruseom• 16 years ago

I don't see why a fanatical anti-rant is any better than a fanatical rant.

There's this theory out there that Agile projects can refactor fearlessly because there's this immaculate suite of tests that can sound the alarm the second the smallest regression gets introduced. But anyone who's actually tried it knows that it's mostly just a fantasy.

Poppycock. My project has an extensive test suite that we rely on to make major design changes with alacrity, just exactly the way that unit-test advocates advocate. We couldn't dream of doing what we're doing without it.

It's true that not every module of every system responds equally well to this style of testing. It's true that putzing around revising test infrastructure for the nth time really sucks. But these are points that belong in a serious discussion, the kind that recognizes tradeoffs.

douglasputnam• 16 years ago

cashto's post was a "serious" effort to disperse the cloud of dogma surrounding unit testing and TDD. I hardly think that lashing back with "poppycock" is driving the conversation in the direction of "serious discussion".

cashto emphasized that unit testing is useful for teams with non-expert programmers (viz. his comment about the Dreyfus model of skill acquisition). Many of us are non-experts who benefit from working with the TDD training wheels on; it's O.K. to be less a Jedi All-Star hacker. But the phenomenal programmers, the men and women who were 10 times better than me, could not be bothered with TDD, even through the endorsed it for mere mortals.

To paraphrase the TDD credo of “no test is worse than having no tests”, we can assert that blind faith is not better than no faith at all.

pmccool• 16 years ago

The training wheels analogy was the weakest part of the article to me. TDD isn't a teaching tool, and the fact that they useless to people in a very specialized niche doesn't really say anything about their general value.

I also notice reference to writing tests after the fact. I understand TDD to involve writing tests first. That's how I practice it anyway.

ZeroGravitas• 16 years ago

That's just poisoning the well. Yes, X is fine if you're an idiot is exactly what you would say if you didn't want to have a conversation about something and would rather shut down discussion.

Vitaly• 16 years ago

this is one of his worst arguments. I find it almost exactly the opposite. I've been programming for almost 20 years and I am considering myself an expert. and many people around seem to agree ;). most of the beginning was spent w/o any kind of unit testing. But I find it that I do more and more testing with time, and I'm sure as hell I'm not becoming a weaker programmer that suddenly needs the training wheels ;)

cabalamat4• 16 years ago

> My project has an extensive test suite that we rely on to make major design changes with alacrity, just exactly the way that unit-test advocates advocate. We couldn't dream of doing what we're doing without it.

My experiences are the same as yours. I've refactored code where the unit tests caught that the new structure introduced subtle logical inconsistencies in the program. The unit tests meant I caught this immediately; without them I would have had to wait until subtle and hard-to-fix bugs emerged.

On a different project, I've been able to play around with the structure of the code, secure in the knowledge that my regression test suite will tell me when I fuck things up.

Tests, when done well, are a lifesaver, and essential to any large complex codebase.

> But these are points that belong in a serious discussion, the kind that recognizes tradeoffs.

Exactly.

MartinCron• 16 years ago

There's this theory out there that there are creatures that aren't birds but somehow fly through the air using wings. Anyone who has looked for these mythical "bats" knows that they are just fantasy.

Here we see the risk of asserting something isn't real just because we haven't ourselves seen it. I've held a bat in my hands, and I've had my butt saved countless times by tests I've written.

tetha• 16 years ago

Ah, when I first began to read this, I expected one of those rants against unit tests which don't leave any kinda good thing about tests and I mentally got ready to comment about that.

However, I am quire pleased that this is a very sensible and mature position about unit testing (I think a short version would be: well, testing can help, it won't carry you and you have to stand on your own feet, but it can help), which is pretty close to my own position. Good Article.

jwr• 16 years ago

As with all things, truth is somewhere in the middle. I've found that unit tests do help me write better code and find corner cases which I would otherwise miss. But I don't treat them as a religion, and I don't write them for every function.

Just don't overdo either way and you'll be fine.

arethuza2• 16 years ago

But people really don't want that kind of sensible mature argument and would rather have set of "best practices" that they can blindly follow.

abp• 16 years ago

Yes, but only if one got zero or not much experience in the field. So actually you are saying something that is true for everything.

crististm• 16 years ago

I've tried TDD for one project and it gave me a nice feeling of having a safety net. It was a good feeling. The problems started to appear when the project increased in complexity and needed a redesign in some of it's parts. I had to do _again_ twice the work to have the tests match the code behavior.

The article mentions this problem too. The benefit of the tests disappear when your code changes beyond refactoring.

allend• 16 years ago

Then why didn't you drop the tests if you think there was no benefit?

philjackson• 16 years ago

This whole article could be boiled down to: "bad tests are useless".

cawhitworth• 16 years ago

SQLite would like to disagree with this rant, I think.

alextgordon• 16 years ago

The article doesn't seem to be claiming that unit testing is bad, rather that religious promotion of it is bad, and the benefits and drawbacks should be weighted up like any other decision.

Obviously SQLite has got a tremendous amount of value out of unit testing. But SQLite is quite abnormal in many respects. It's used in aviation, so the standards of testing are ridiculously high. It's also a library, and libraries tend to be very easy to test. It's also a long-term project with no deadline to speak of.

humblepatience2• 16 years ago

The way I read the article it sounded like he was downplaying the usefulness of even healthy amount of testing i.e. writing tests along with the code your writing.

Certainly, all his arguments downplay TDD in general

regularfry• 16 years ago

"Many of them will not even survive the process at all."

That's the point, and I think everything else in this article stems from it. Unit tests are supposed to be transient; you're supposed to be able to just ditch them and replace them when you ditch and replace the code they're testing.

Functional tests, on the other hand, should be longer-lived.

chmike• 16 years ago

With my students and used the analogy of unit tests (or asserts) with a fisher net to catch bugs. As a fisher net, it is made of a mesh that won't be able to catch small fishes. It may also have big holes in it here and there. So there is always a limit to its efficiency.

It requires a significant effort to make unit tests more efficient. This effort must then be balanced with the potential costs of the bugs and across the application, because some part of the program are more critical than others.

A good programmer should thus balance his effort in unit tests writing.

I also suggest to write unit tests as needed to be more efficient. If the software is a data processing pipeline kind, simply write a very tight data validity checks at the end of the processing pipe and inject test data (including bogus one). If no problem ever show up with the test data, there is no point in writing unit tests for all intermediate steps of the data processing pipeline. If a problem is later discovered, then write the test, but only at the middle of the processing pipeline. You'll know if the problem is before or after. Then continue as needed. With this approach, you'll write unit tests only where it is needed and won't waste you time.

Suggesting as the author that we don't need to write unit tests is like saying we don't need to put the car safety belt because people still die in car accident with it, and most of the time we don't have any accident even if we don't wear it. Is this a smart advise ? I don't think so.

BTW, I use asserts wherever I can by reflex in my code because it saved me valuable time on many occasions.

10ren• 16 years ago

Aside on testing: I was surprised in re-reading Fred Brooks how cloose he sounds to unit-testing, in Mythical Man Month, 2ed, pp.147-148. He suggests testing components separately, before integrating, even though this requires lots of "scaffolding".

Scaffolding includes dummy components, miniature file (minimal eg of a file format), "generators of test data, special analysis printouts, cross-reference table analyzers" (jigs and fixtures).

"It is not unreasonable for there to be half as much code in scaffolding as there is in product". I end up writing a lot of the above kind of code - not as automated unit tests, but just as a way to make sure my code is doing what I think it's doing - just plain ol' testing. It can feel wasteful, so it was reassuring to hear Fred's take.

JoeAltmaier• 16 years ago

How about "buckshot testing"? I wrote a file system test that made every single API call with null arguments, good arguments, illegal arguments and then logged the error code. We ran this against every release of the OS and simply dif'd the result against the previous release run. It not only aided in bugfinding, it identified necessary release notes about API differences. The bigger point is, you have a computer at your disposal. It can make LOTS of calls in a short time. Try calling that function with every conceivable argument. Once you have a baseline, a diff tool can reveal everything.

projectileboy• 16 years ago

I'm not sure how valuable these articles - or threads - really are for anyone. Don't we have to balance cost vs. value for _anything_ we do? Find and replace "unit testing" with "documentation" in this article and we could be having the same conversation.

j_baker• 16 years ago

Personally, if anyone asks me if they should unit test, I'll play the fanatic and take a hard line. Mostly because the way to learn the usefulness of unit tests is to write them. You quickly learn what they're good at and what they are bad at.

Once you've written enough of them, you kind of seem to develop an instinct as to what unit tests are useful for.

humblepatience2• 16 years ago

I wrote a response to this on my blog

http://www.scottschulthess.com/coding/?p=143

Basically, there are other benefits to TDD than just preventing regression.

Documentation and ease of debugging while your writing code being the biggest ones.

In a non-compiled language it can serve to verify that your at least runs without gross failure a little like the compiler does.

It's OK Not to Write Unit Tests

Related Stories