Unit Testing: Is There Really Any Debate Any Longer?

http://www.drdobbs.com/testing/unit-testing-is-there-really-any-debate/240007176

by codereview12414 • 14 years ago

30 69 14 years ago

drdobbs.com

markmm2• 14 years ago

Yes come on, does anyone still think they are useful and simply not a complete waste of developer time?

lucian1900• 14 years ago

I can't decide if you're serious and misguided or just joking.

batista2• 14 years ago

You know, actual luminaries in our field are also against them. I've seen UNIX programming gods among men dismiss them. It's not like it's a clear cut issue.

For some things like libraries and such, there are absolutely needed, but unit testing everything and especially TDD are stupid.

To be frank, they mostly sound like some Java EE-like monstrosity that for some bizarro reason caught on with the hip crowd too. I'd put them out there with FactoryOfFactory and XML documentation...

taligent• 14 years ago

Of course there is debate.

The reason is that automated tests costs 2-3x as much time upfront. The assumption is then you will recoup that investment in the future. The problem is that automated testing doesn't cover everything so you have to augment it with human testers. Which then begs the question are the human testers being effectively utilised.

And for those of that do develop have a think about all the time spent on buggy web UI testing, screenshot comparison testing, tests being flakey due to software choices etc.

markmm2• 14 years ago

Agreed, I would focus on hiring shit hot testers and automating end to end testing if possible. Unit testing is useful for a small number of examples like a method that does complicated calculations,but I see TDD zealots writing them to test a method that simply forwards data to another layer, all designed around dependency injection.

rwallace3• 14 years ago

Automated _unit_ tests cost two or three times as much time, sure. That doesn't mean don't use automated tests, it means don't use unit tests. Use integration tests instead. Not only do they provide much better coverage (you're testing all the code from top to bottom, each module in the context in which it is actually used), but they are also far cheaper.

k2xl• 14 years ago

Unit testing is a very very expensive operation. Consider the following: The developers add unit testing at beginning for specific results. Later on, after product is released a new design or feature change happens. Often times unit tests will have to be rewritten else they will fail.

This can be very expensive and take developers and QA engineers extra time when they try and figure out what's wrong... "Is the test failing because the test is out of date? Or is our program causing a bug?" I know for a fact HP deals with this issue constantly. Not saying unit testing is bad (I think it's usually good) but I don't think it's always good.

antrix• 14 years ago

> "Is the test failing because the test is out of date? Or is our program causing a bug?"

IMO, when you change code (extensively), you'd still have to invariably answer those questions. But in the absence of tests, it becomes much harder to find the answers.

justincormack• 14 years ago

Indeed. The tests tell you that you are breaking a contract that you once thought was important. It might not be any more, but good tests reflect good contracts. Removing the test requires you to think whether any other code expects this contract.

jtreminio• 14 years ago

Lowly PHP dev here, so take my words with a grain of sakt, but if you go with TDD, any future refactoring should also follow TDD - write the test first (or modify existing tests) to what the expected behavior should be, then modify the actual app code.

ollysb• 14 years ago

Unit tests take a little time to write, sure. But look what you get for your investment:

(for the sake of this discussion I'll avoid subjective issues such as improved design)

1) Faster feedback cycle - if you're developing new code you can run a single test and check inputs/outputs very quickly. If you're testing by refreshing a page, or shudder, restarting your java server, you're going to have to wait far longer to find out if your code changes work as expected.

2) Reduced risk when making changes to the code base - the risk increases the closer you get to your data store as the amount of dependent code increases. The only way to reduce risk without unit tests in place is to not change any code i.e. you have to keep building on top (see the open/closed principle). This inevitably means that your code is going to introduce duplication as you can't refactor it out where appropriate. The other option of course is that you just risk it and hope to god that your manual QA testing will catch any problems before your users do.

It's also worth noting that the cost of adding unit tests isn't fixed, it's dependent on

a) experience

b) the tooling available for your language

From personal experience I find adding unit tests in ruby takes very little time. This means that benefit 1 in particular kicks in far sooner. Currently I feel it's faster for me to write a piece of code with a unit test than without.

bryanlarsen3• 14 years ago

That's the major problem with system tests, but it shouldn't be a big problem for unit tests. In my experience system tests are so fragile that the cost of upkeep is larger than the benefit gained except in a few cases. Unit tests on the other hand are much more stable.

rwallace3• 14 years ago

My experience is precisely the reverse. System tests are stable because they only need to change when the observable behavior of the entire system changes. Unit tests are intimately coupled to the internal structure, so if you use them, you have to change them every time you refactor so much as a single function.

breckenedge2• 14 years ago

In practice, I find I write code faster and with more confidence after the tests (usually specs) have been written. So, my productivity increases. I also sleep better at night.

mhd2• 14 years ago

For me TDD is doing the exact opposite. It bores me to tears, which really brings down my edge.

Granted, this is one of the reasons why I'm recently thinking about whether I've got a slight case of ADHD, but I think a lot of it just stems from the fact that I like to brainstorm code and afterwards use something like tests or just good documentation to reason whether that was the most proper approach.

Never mind that things like cucumber make me cringe big time.

batista2• 14 years ago

>For me TDD is doing the exact opposite. It bores me to tears, which really brings down my edge.

That.

It takes all the fun out of using a dynamic language. You might as well go back to a static typing and write LESS tests for obvious things...

breckenedge2• 14 years ago

I agree that experimentation and brainstorming are not the time to be messing with unit tests. Specs maybe. I've been frustrated enough with failing tests to occasionally lose them.

But, when it comes to something I expect someone else to use, there is no question that I will start with a test.

bluedanieru• 14 years ago

Ah, the old 'Argument by Dismissal'. Still just as bold, and still just as unconvincing.

damoncali• 14 years ago

A rare Anti-Betteridge. http://en.wikipedia.org/wiki/Betteridges_Law_of_Headlines

kmfrk• 14 years ago

With CI, it's become a little harder to make the excuse. :)

arctangent4• 14 years ago

I see unit testing (and especially TDD) as a barrier to agile development.

If I'm cranking out a prototype to show a customer then I don't want to spend precious time writing tests for code that stands a good chance of being thrown away at the start of the next iteration.

Tests often do need to be written, but the best time to do this seems like an unsolved problem to me.

agscala• 14 years ago

Not only is the best time to write tests an unsolved problem, but also the method in which you write tests. I think TDD is a joke, but unit tests are still valuable.

morphyn• 14 years ago

Maybe you shouldn't write tests for a "prototype" that you're going to "throw away". Tests are (in my opinion) necessary, but TDD is a style of coding and you don't have to use it.

johnkchow• 14 years ago

I definitely agree with the statement about prototyping and testing. Testing is another signal of quality, a guarantee if you will, to your stakeholders that your software provided is dependable and trustworthy. That doesn't matter if you have no customers to begin with!

I'd also like to throw it out there that testing can be especially helpful when writing code against known problems. Some benefits are:

- Since you already know the problem domain, your test writing will be extremely quick and effortless, so the benefit of having that test in the first place far outweighs the time/effort invested in writing them.

- In dynamic languages especially, typos are what kills you in this situation. Testing will quickly catch 'em as you program.

- Finally, testing will force you to think more about the edge cases, since implementation is a straight-forward process. Thus, you'll spend less time later doing QA and have an overall more polished feature.

spdy• 14 years ago

I discovered for my self that the best time to add tests to a project is when you cant overlook everything anymore and you ran into problems when you add one feature on one side and on the other a whole house breaks down.

There is no need to write tests for fast moving objects and for this kind of work TDD is a waste of time in my opinion. But for core infrastructure this makes sense.

But driving blind all the time is not a good idea on the long run.

roel_v2• 14 years ago

Your argument can only lead to the logical conclusion that you consider agile development to be only for making prototypes, which I assume you don't mean. But if you don't, your objection becomes unsupported. Maybe the problem is in your seemingly equating unit testing with TDD.

arctangent4• 14 years ago

Perhaps my choice of words could have been better... swap out "prototype" (a hyperbolic example) in favour of "iteration" in my second paragraph and see if it makes more sense.

The central point I am trying to make is that it seems wasteful to spend time writing tests for code that may soon be deleted.

bryanlarsen3• 14 years ago

In my experience, a prototype is never just a prototype. You crank out a prototype in an hour to demonstrate something. Then somebody says "add this and then we'll show it to the boss.". Your boss says "that's cool, add this and this and this and then we'll demo it to a potential customer". You do so, then play test the demo a few times, uncovering scenarios that need hardening. You demo to the customer, and they ask a few questions meaning that you add a few more features to the prototype for the next demo. Rinse and repeat.

After a few successful demos you're given an tight deadline to "finish things", so you have no choice but to polish up the prototype and ship it.

The moral: code that was written to be thrown away often isn't. You can still write code quickly, and take shortcuts (such as skipping tests), but recognize that you'll probably be working on that code for a very long time and may regret your decisions later.

arctangent4• 14 years ago

This is what I was alluding to when I mentioned that it's not obvious when tests should be written. It simply isn't practical to write them as part of a quick prototype (or any other rapid iteration). But writing them at the end of a project may also be difficult due to the need to move on to something else. The same argument holds for things like documentation...

ehosca• 14 years ago

why do i get the feeling that you probably haven't spent much time "debugging" unit-tests...

unit testing give you the false confidence that your stuff is working, it has little to no bearing on stuff actually working when pushed to production (unless you have 100% coverage and actually thought of and defended againt every possible use scenario before hand)

dasil003• 14 years ago

Yes, unit tests are code and all code can have bugs, but you're missing the point. Automated tests are a reflection on the code just like code reviews or documentation or formal specifications. If they are well written (ie. orthogonally structured to the code they are testing and with good readability) then they provide a strong window into the intent of the original developer, and this intent can be continually tested for almost no marginal cost for the lifetime of the code.

Yes there is a maintenance cost, but I find most people's objection to testing is based on poor testing skills. Once you get good at testing with a decent framework, you find that it's often about the same time to write formal tests as it is to do the manual testing which we all know is necessary to have any certainty that code is working.

koski• 14 years ago

Unit tests are good. System wide functional tests are awesome.

lukev3• 14 years ago

At work, we're definitely turning away from traditional unit tests, which is not to say we're turning away from testing.

Instead, for complex or algorithmic code we're using generative testing, barraging the code with random inputs instead of a few cherry-picked test cases.

Then, we're structuring our systems and operations in such a way that we can spin up test instances of the entire system and run full, end-to-end integration tests at various levels of load.

We haven't figured it all out yet, but the results are promising compared to unit tests, especially tests which contain a lot of mocks and stubs.

DanWaterworth• 14 years ago

This is the approach that I take with my personal projects; I firmly believe quickcheck style testing is a huge improvement over manually writing out individual test cases. Do you tend to find, as I do, that when you write you code to be testable using randomized testing that it tends to look more functional? (I'm referring to the paradigm rather than the opposite of dysfunctional)

lukev3• 14 years ago

Well, considering the code I'm talking about is mostly in Clojure, yeah, definitely functional. It's really cool to see how all the pieces fit together once you buy into the functional paradigm.

kernel_sanders• 14 years ago

I find that the same quickcheck style testing is the fastest way to get out quality code for a project that I am the sole devleoper of.

I usually code in a terminal running screen with many shells going, one of which is running ipython so I can interactively import and test snippets, classes, or small functions that I'm writing in a larger module. It works great.

However - those tests disappear! Once a recent project of mine was handed off to a team for maintenance and more features, I found that, in particular places, where the code was subtle, it was subject to breaking by others who were just trying to fix something unrelated. This was when I, personally, realized the value of unit testing.

Yes, its slower and takes more time up front, but if your code may someday be worked on by others, it protects the sections of your code that have test coverage against breaking - which over a longer time scale can save time.

dmlorenzetti• 14 years ago

For my algorithmic code, the unit tests do the random input selection.

Actually, it's often a sort of guided random testing. For example, if the algorithm has a straightforward linear regime with an exact solution, and a nonlinear regime that has to be solved iteratively, the unit test will target, say, 2% of inputs at the linear regime, 1% at the breakpoint, and the rest at the nonlinear regime.

Mockups and stubs are another matter. My algorithmic code is numerical, so stubs are rarely an issue. If an algorithm needs, say, a matrix solver, you pretty much just have to use a real (unit-tested) matrix solver in there.

scott_w• 14 years ago

The issue with using random test data is the it doesn't expose your edge cases.

There was a post on HN a short while back that showed why - in essence your edge cases sit at the edge of the normal distribution ergo they will rarely be hit by the unit tests.

This leads to you doing more work, as you need to log what data caused the test to fail. It also gives you a false sense of security, since the tests aren't really doing anything 99% of the time.

jerf• 14 years ago

I'm sitting here significantly refactoring a HTML-generating class hierarchy that I made a couple of foundational mistakes on a couple of years ago, causing excessively slow rendering. It has about 100 classes, and is used in over 1000 screens. Most of them trivial, but the non-trivial ones are definitely non-trivial. Fortunately, it happens that there is a very clean input and output that is, by design and by network structure, isolated, making it feasible to test. I've got decent automated test coverage on the existing set, and I'm in the process of adding an automated test that will verify that both the old and the new system emit byte-for-byte identical output. Without this test, this would be a hopeless exercise and I'd have to just live with the outcome forever.

I find myself wondering how many people complaining about unit tests work on leaf-node sorts of programming projects, because for me, many of the tasks I routinely take on in my environment where about a dozen projects are building on the code I am writing mean that the question isn't whether unit tests are "better" or "worse". It's a question of whether the task is possible or not.

programminggeek3• 14 years ago

One thing to consider is that most code that gets written is not built to be highly testable. This makes testing harder to the point that "it's not worth it". If you optimize your coding style for testability, writing tests is faster and easier.

The problem is, writing testable code requires more thought up front not just about individual objects and methods, but also the overall architecture of your project. If your code is highly tied to an ORM or framework, then it's going to suck to test.

Also, not every language is equally easy to test. For example, Ruby has an obsessive culture around testing, so they have a lot of fantastic testing tools and the language itself makes things like mocking pretty easy. PHP by comparison doesn't value testing as much as Ruby does. The testing tools are worse and there are fewer of them. Also, due to the lack of monkey patching, you have to write your PHP code with dependency injection and testability in mind from day one for it to be highly testable.

In short, testing is a useful thing, but most code isn't written to support it and many languages and frameworks and patterns encourage you to write unpleasant to test code. If code is hard to test, programmers won't write tests unless their job requires it.

cheez3• 14 years ago

In short: by keeping testing in mind, you make your code better.

kleiba• 14 years ago

...for testing.

Daniel_Newby2• 14 years ago

> If your code is highly tied to an ORM or framework, then it's going to suck to test.

Unit test, yes. System integration test, no.

programminggeek3• 14 years ago

Actually, if it's tied to an ORM, then you have to make DB calls for each tests, which is to say running the full test suite could take minutes, hours, or even days? If it takes a long time to run your tests, how often will you run them?

I'm not trying to imply that integration tests that hit the DB are always a bad thing, just that slow tests are another reason people don't like or use tests.

languagehacker• 14 years ago

There's no debate about testing stateless functions. I think the debate is in whether the ledger reflects a cost benefit of spending all the time mocking up different functionalities in order to test stateful components or components with external dependencies. Since these are hard to calculate ahead of time, it really does become a question of engineering culture and time to market.

cbs• 14 years ago

That debate exists because the people selling the idea of unit-tests were so goddamn gung-ho for unit testing everywhere and all the time.

I have friends in the medical industry, their tests must achieve 100% code coverage on multiple coverage metrics, and code coverage is only an afterthought to even more rigorous test requirements. Even they were pointing and laughing at the TDD ideologues.

Unit testing is beneficial, if you do the right amount of it and in the right places at the right time. You might have a totally sensible test plan. But like it or not, for the time being, plenty of people are unwilling to listen to you because some other assholes already burned them on the idea of unit testing.

Its going to take years and years to shake loose all the cobwebs in our industry caused by half-cocked advocacy. Then we can maybe get to the sensible discussion about how to analyze a project and develop a sensible test strategy that fits the project's needs.

nicholassmith• 14 years ago

Unit testing, like many other good and healthy coding practises is great. In theory.

In the big mean real world frequently you're shipping things in such narrow deadlines that if you say "Hey boss, I'm going to take some time to write some wicked cool unit tests, it'll make my life easier in future because we'll know this code works!" then Boss' head will pop as it's a non-tangible improvement.

Developer focused companies are great as they'll let you get away with this, but the large portion aren't.

columbo• 14 years ago

Purposeless unit tests are a waste of time. These are tests against your compiler or external frameworks. For example:

    @Test
    testAddUser() {assertNotNull(new User("columbo").save())}

Wahoo! the ORM works!

The biggest problem I have with unit tests is they tend to be

a) just testing frameworks (crud) and not much else

b) so heavily built on mocking that changing them is like redesigning a wool sweater. These tests tend to examine how the code was written, not necessarily what the expected output is (oh shoot, I need to mock up five more objects to make this test pass... hey look at that, this test doesn't even check what should be returned. Nice)

c) there is no context into or out of the unit test. The assertions are vague, it isn't clear how I got there, it isn't clear where I'm going (testXMLOutWithInProgressUsers ... what is the xml for, who are the in progress users... what is next in this chain of events)

d) the things that REALLY need unit tests don't have them because they are too complex to write (ah, that whole system that runs a nightly bash script, takes in a stack of csv files from the client and converts it in our database... yeah, that's not tested).

With all that said, tests are invaluable, but they need to actually, you know, TEST something.

I like nodejs in that respect. It takes me seconds to fire up the webserver and test the rest api. I don't mock objects, I pass values directly through the api and check the expected results. I like this because it shows the next user what the api actually looks like and I'm testing the full stack (In my experience controllers are notoriously undertested, mainly because you need the full stack running). Since the application is really just an api this means that I'm able to test everything equally, from batch processes to CRUD.

JoeAltmaier• 14 years ago

Agreed.

API testing has that wonderful property, its appearance is the whole deal, if it looks right then its working.

Anything but app code isn't so easily tested. When frameworks are written to support testing, they often are sub-useful - crippled versions of real programming languages etc.

Anecdote: I recall a QA guy asking me how to test our OS's file system. I said, make every filespec-based call with a good spec, a non-existant spec, an illegal spec, a null spec; log every result. After each release, diff the logs and explain every deviation.

He said "That's too hard, I'll just shotgun it." and left. Why ? Because his test-scripting framework wouldn't allow anything as sophisticated as lambdas etc.

So I spent half an hour writing a table and a loop, and sent it to him. He used it for the next 5 years to test every release.

My moral: do an exhaustive test when possible. The computer is fast, it doesn't feel any pain, and you will find more fencepost and boundary issues than you ever thought existed.

TianCaiBenBen• 14 years ago

1. Unit Testing is useful ONLY for complex API method/function, which should not change too often and too frequently;

2. Ideally, you should only test API not API's implementation but most of the mock framework will tie your test code directly against your API implementation. Once your API implementation changed, your test code fails, you have to rewrite test code, which is waste of time.

Someone may argue that it because you write bad test code; Talking is easier than doing;

So, usually, I prefer to doing the functional integration test than the unit test, which is productive for me. (I know people will throw out thousands of reasons about the benefit of unit test and "correct" way of writing unit test.)

Anyway, I will avoid unit test as possible as I can and ONLY do my designed integration test, which works for me.

3. And writing unit test for code coverage is stupid; Some manager even brag about the unit test coverage, which is totally a false confidence about the quality of the code underline; This is even stupid;

4. Writing unit test for those requirement-always-change area is waste of time;

5. TDD is a BIGGEST JOKE, which weights too much on writing auto testable code over writing working functional productive code;

jhrobert• 14 years ago

Unit Test, sure, but at what "unit level"?

I usually prefer "automatized system testing" where fewer "high level" tests involve a lot of code, preferably in a test environment that is close the actual production environment.

Of course, when a test fails, some more analysis is required to determine what went wrong. But it's usually related to recently modified code, hence easy to spot.

That way I get the benefits of automatized tests without the burden of tons of "micro" tests at the class/method level.

Unit Testing: Is There Really Any Debate Any Longer?

Related Stories