Unit testing is not (generally) useful

by axod • 17 years ago

21 53 17 years ago

The author rejects unit tests because they don't help find bugs. I don't think his reasoning adds up to much, more like rationalizations for already-held beliefs.

But if you want to take the argument seriously, I would say that "finding bugs" is not the primary reason I write unit tests anyway. It's to prevent regressions, and make me feel safe that I haven't broken anything after a refactoring. If not for my unit tests, I'd be scared to change much of anything, for fear of stirring up a whole bunch of trouble that I'll have to fix later on.

jgrahamc• 17 years ago

I agree. Unit tests are what make refactoring safe.

Adding known bugs to the unit test suite is a good way to prevent regressions.

abstractbill11• 17 years ago

"preventing regressions" is a subset of "finding bugs", and I've found the other methods I listed to be more effective (from an ROI perspective) at doing that too.

allenbrunson2• 17 years ago

Apparently you work at Justin.TV. I don't know how big your individual projects are, but personally, I didn't get unit-testing religion until I started working on really big stuff, close to the limits of what I'm capable of. I worked for a year on a 60,000 line program, and it started getting painful. I worked for another year on a 100,000 line program, and that's when I finally had to relent and start writing unit tests.

Here's a common scenario that made me want to write tests. I want to implement feature A, but to make it work, I'm first going to have to write library B and C. So I write library B, but there is no way to test it in the program yet, because feature A doesn't exist. Right after I've finished the code, I have a pretty good idea of where things might go wrong, and what corner cases I want to exercise. But by the time I finish library C and then get around to implementing feature A, I've forgotten that stuff. Pain points go unaddressed.

In summary, unit tests really start to show their value when you're working on a program that's too big to fit into your head all at once.

chuckm• 17 years ago

It seems like you're still using some form of automation to find bugs, regardless of whether these would be classified as unit tests, correct?

axod5• 17 years ago

>> "If not for my unit tests, I'd be scared to change much of anything, for fear of stirring up a whole bunch of trouble that I'll have to fix later on."

For me, if I had unit tests everywhere, I'd be scared to change much of anything, knowing that after the refactoring, and usual testing/checking things work, I'd then have to rewrite/fix unit tests.

darkxanthos• 17 years ago

That's why I stopped for a while. Then I discovered BDD style testing (or context/specification, or...).

Whatever it's called, it's not quite as fine grained as a unit test and it's not as large in scope as integration testing. It's at the level of your business specs. So the only time you'll break your tests is if you've jacked up a spec.

Honestly, if my changes cause new behavior in the system but the people paying me won't care, then I don't care either.

jon_dahl2• 17 years ago

Unit testing can be this way, but it doesn't have to be. At a really low level, unit testing can be "testing the implementation", which may be useful while you're writing code, but isn't useful during a refactoring.

But slightly higher level tests can be written to almost never have this problem. Let's say you've written code to ensure that a CC transaction doesn't go through with an invalid number. No matter what the implementation looks like - no matter how much you refactor - this test should always work. The same is true for a wide range of useful tests. (These aren't strictly "unit" tests, but they fall into a similar category.)

My personal experience is that a good unit test suite is 99% useful when refactoring, and 1% painful.

swombat4• 17 years ago

Also, TDD/BDD are most useful as design practices - not as bug-finding practices. They help you think more carefully about your design and make it more modular and clear.

tow21• 17 years ago

I agree entirely. Whenever I leave out writing unit tests, I end up with a web of messy interdependencies which ends up far too hard to extend, and will always need refactored as soon as I need to add the smallest feature.

Writing unit tests up front forces me to write the code in a way where inputs & outputs can easily be stubbed out, which in turn makes writing messy interdependencies simply too hard!

Basically, if I write with unit tests, it becomes easier to write maintainable code, and harder to write unmaintainable code.

pmarsh• 17 years ago

It all depends on how your development works.

Let's look at a scenario where you are using an outside framework.

Not writing tests for your code? Framework v2.0 comes out and you want to use a great new feature in that version. Will your app work if you upgrade? Sure you'll have to test/QA your app regardless, but with tests you might have a better idea of where to look for problems.

This is what I'm facing at the moment as a single developer. The app has no test coverage so updating is a laborious process of testing everything by hand.

But getting into testing has it's own problems. The biggest I have with testing is when I read discussions about bugs within the test framework. How much time will you spend working around bugs in the testing framework?

bjclark2• 17 years ago

Judging by his analogy and depth of insight, I'm guessing he's never even tried to write unit tests on a daily basis and certainly never done TDD. It's the same old argument that everyone makes when they've never actually learned to do something properly.

abstractbill11• 17 years ago

Actually I wrote unit tests daily at my last company, for about two years, and in the last six months of that I gave TDD a try.

My claim isn't really that they don't work, it's that (compared to the other things I listed) they have a tiny ROI.

bjclark2• 17 years ago

Unit tests have nothing to do with, and aren't in any way mutually exclusive to load testing and logging and user reports. Frankly, it just sounds naive to say that you replaced unit testing with logging, or load testing, or letting your users find the bugs for you.

People that do unit tests do all that stuff too!

Your lack of specs also has nothing to do with unit testing. It might dictate the difficultly in writing acceptance tests, but that's a completely different subject.

I don't feel the need to proselytize unit tests since you've already written them off, but what's the point to posting about why you don't like something only to back it up with some sort of rambling, incoherent analogy that you probably don't even believe?

cytzol• 17 years ago

I'm currently writing a personal project for university, and I've made sure to test it all the way through.

When I was part-way into coding this thing, I took a day to go through all the code and write tests for anything I could. I found a few bugs, so I could at least say that it helped somehow. But the tests have helped a lot more when I added to it - it feels -good- to see a big green bar light up whenever I add something. It's -nice- to run all tests, leave it a minute, and confirm that it's not going to blow up before I have to demonstrate to anybody. Of course, finding bugs is nice, and I set up a better testing rig to make sure it doesn't fail on anything I haven't thought of, but the benefits of unit testing go further than bug detection.

When you're all crestfallen with no desire to code, finding bugs is only secondary.

abstractbill11• 17 years ago

If passing unit tests give you the motivation you need to work, that's great!

Personally, just making something people want is enough motivation for me ;-)

axod5• 17 years ago

It's a false sense of security though. It's like putting a big badge on your website saying it 'validates' - even if it's totally broken in real world browsers.

In my experience, the more common bugs are integration bugs, scaling bugs, weird memory usage issues, not to mention interfacing to external systems - for example some weird browser comes along and does idiotic requests you'd never expect a sane person to do.

Those things are very hard to test for, and don't really produce a good ROI as abstractbill says... Best just to log things, graph things where possible, spot weirdness, investigate and fix.

biohacker424• 17 years ago

Everything you mentioned I count as part of TDD or BDD or what ever automated testing acronym you like.

What I suspect is that all of us have basically the same idea in mind, and what we're disagreeing over is mostly semantics and context.

Btw, all of my unit tests include memory usage validation. It's easy in c and c++, but I'm not sure how it would be done in a web development environment.

Hates_• 17 years ago

To me Unit Testing is more about testing the expected outcome and making sure of the status quo rather then finding bugs and testing all the extreme small edge cases. Then when an abnormal and unexpected situation occurs, writing a test that induces it and fixing the code so that all the related tests pass.

To use the doctor analogy, I would say unit testing is more about testing patient outcomes when they are put through varying courses of drugs or treatments (fixing bugs and adding new features) rather then trying to make them die before declaring them healthy. Is the patients blood pressure normal? Check. Now the patient wants to start some treatment for something else. Oh wait, I'm getting an abnormal blood pressure reading.

jcromartie• 17 years ago

I didn't appreciate unit testing until I started doing development test-first in Smalltalk. You just sketch out the skeleton of your classes with instance variables and accessors. Then you write tests the way you want your objects to work (design your API by pretending to use it). Then run your tests, and marvel as the debugger stops on missing methods and lets you create them at that point and continue the test. It's a terrific way to get your code right the first time.

I don't know what he's using there at Justin.tv, but most languages that are more rigid (than Smalltalk or Ruby for example) are certainly not quite as nice for unit testing.

abstractbill11• 17 years ago

Sounds great! But in the meantime, I've put a couple of prototypes online, got a bunch of feedback, iterated, and figured out that nobody wanted the thing we designed in the first place. I've moved on to an evolution of the original idea, while you're still focused on getting the first idea to work "perfectly".

My point is that unit tests aren't cost-free, and their ROI is awful for most things.

fizx• 17 years ago

That's funny, cause I practice something I'll start calling "cost-free unit testing".

Step 1: If you're thinking about how a code api should act, sketch it out inside a test.

Step 2: If you're about to jump into a repl and poke at your code, open a unit test instead.

Now you've simply replaced other activites at no net cost to you. My core takeaway is that everyone tests their code somehow. You're best off if you can automate that process. Don't obsess over it (I almost never test my html/views), but try to find opportunity.

swombat4• 17 years ago

Unit tests don't take that long. If it takes you that much longer to write your unit tests, you must be doing something fundamentally wrong - like perhaps testing the wrong thing or focusing on the wrong kind of testing.

biohacker424• 17 years ago

When (or should I say if) JustinTV grows much bigger and hires a lot more programmers this whole unit tests are not useful thing will come back to bite them.

But I can see how right now it seems their time is better spent on other things.

abstractbill11• 17 years ago

Out of interest, are you speaking from personal experience?

biohacker424• 17 years ago

Yep, I got TDD religion when I started working for a huge company.

I did some testing, but not much when I worked for a startup.

dfragnito• 17 years ago

It's a money make. We needed some python work done. It was a small project 42 hours 6 hours was unit testing. Unit testing increased the invoice by almost 17%. I can see why consultants like it.

swombat4• 17 years ago

I know many people who do unit testing (myself included), some of them consultants. Not a single one of them does it "to buff up the invoice". In fact, every single one of them will argue (sincerely) that good unit testing practices help reduce the time they need to deliver code and increase the quality of that code. They'd also argue that good unit testing practices make it easier to then hand over the code to someone else.

jon_dahl2• 17 years ago

This is only valid if the code is not going to be maintained. If it is, and especially if someone other than the original developers are going to maintain it, then unit tests are worth their wait in gold.

vicaya• 17 years ago

The article has a rather narrow perspective on unit testing, given his product is not mission critical. He's basically treating live production as a giant integration test.

Some of my perspectives:

class/function level unit testing is only needed for common library code.

component level unit testing is critical if you want to do any significant refactoring.

non trivial integration tests (especially timing sensitive multithreaded code) can find many subtle bugs that would be too late to find in production.

I've written a few tests (for a database engine) that paid themselves many times over.

abstractbill11• 17 years ago

The article has a rather narrow perspective on unit testing, given his product is not mission critical.

I do have a habit of assuming every other hacker is doing some kind of a startup, so this is a fair point.

thenduks• 17 years ago

For me the primary use of unit testing is in the prototype stage. I write the tests instead of the application code to make sure everything is sane. I find lots of architectural/assumption problems this way.

After you've got real app code the tests serve more or less as regression tests to make sure I haven't broken anything major deep in the guts of my models and such.

In short, like some of the other comments here: Unit tests aren't for finding bugs, they're for helping you not write the bugs in the first place.

nradov• 17 years ago

This is another case of overgeneralizing based on limited experience. Automated unit testing may well be of limited value for building the applications that run a video sharing web site (where failures are no big deal). But now try building a complex library used as a core component of several packaged applications. The library developer doesn't really have a "program" to put through user testing. The only solution to keeping the library stable while refactoring and adding new features is to build a comprehensive suite of unit tests that exercise the library's public API.

lnguyen2• 17 years ago

Whenever someone comments on the usefulness of unit testing (or in the extreme, testing in general), I point them to Feynman's contribution to the Challenger shuttle report:

Appendix F - Personal Observations on Reliability of Shuttle http://www.ralentz.com/old/space/feynman-report.html

A good summary is at http://duartes.org/gustavo/blog/post/richard-feynman-challen...

axod5• 17 years ago

How many people are writing mission critical software? In startups, hardly any.

lnguyen2• 17 years ago

If you rely on your software to be up and running for your livelihood (aka any web startup), it is mission critical.

akeefer• 17 years ago

I'd say that it's also important once the team gets beyond a certain size; codebase and team size tend to scale together, of course, but not always. The value of tests in catching regressions in code you didn't write, didn't realize was there, and don't understand is immense. It's one kind of difficult to not introduce regressions in 100k lines of code that you were deeply involved with; it's a whole different deal to walk into 100k lines of someone else's code and try to change it without breaking anything.

Unit testing is like vaccination: it's good for me that I'm vaccinated against smallpox, but it's also really good for you that I won't be getting sick and giving it to you, and your ability to avoid vaccination (or testing) and still function is often directly correlated to how well everyone else is vaccinated (or how well they've tested their code so you know when you're breaking it).

Those don't always have to be unit tests in the strictest sense, so you could perhaps argue that a test that breaks as a result of unrelated changes isn't really a "unit" test because it wasn't properly isolated, but I really hate those sorts of language arguments. I'll call something a "unit" test if it's not an "end-to-end" test, even if I didn't mock or stub out every possible dependency. And it certainly sounds like the author of this post is arguing against unit tests in that sense rather than in the stricter sense.

Allocator2008• 17 years ago

Really it's like that bridge in 'Apocalypse Now'. The sergeant is telling Martin Sheen how the bridge is always bombed and they have to come in and re-build it, "just so the generals can say the bridge is open." Well, I have done a lot of unit testing (CPPUnit), and it is like that. We do unit testing so management can say the product has been unit tested. Coals to Newcastle, to be sure, but hey, it's a job!

thewileyone• 17 years ago

If your unit testing isn't useful, then find a way to make it useful. Author misses the point completely.

Not unit testing is like building a car door and not check that it fits until all parts are to be assembled, and then reviewing what's wrong. Dumb.

Unit testing is not (generally) useful

Related Stories