When TDD Fails

http://bitroar.posterous.com/when-tdd-fails

by gambler • 15 years ago

93 101 15 years ago

bitroar.posterous.com

pyre• 15 years ago

  > Okay, maybe I'm going abit overboard with fake qotes.

In other words, a straw man?

Retric2• 15 years ago

Or a joke.

mdonahoe• 15 years ago

Why am I not surprised that there are a bunch of typos in this article?

cpeterso• 15 years ago

No tests?

jasonwatkinspdx• 15 years ago

Unit testing is not the only form of testing.

shadowfiend5• 15 years ago

Which, of course, is part of the point of the article (“I'm not saying TDD is a bad thing, but there are more tests than unit tests, and there are more ways to verify software than testing.“)

petercooper4• 15 years ago

I think there's some confusion in the definitions which makes things a bit muddier.

TDD is traditionally based around unit tests, but nowadays can pragmatically include integration and functional testing practices. The former definition seems to be the one accepted by the article, but the latter definition somewhat solves most of the problems raised.

Corrado• 15 years ago

I've actually come to this conclusion on several of my Rails projects. After struggling with mocking up yet another set of I/O objects I realized that it's not really doing anything useful. I agree that there should be testing, its just that TDD on MVC is very difficult to do properly. :/

blackhole3• 15 years ago

I'm currently building a test framework for my graphics engine, and have run into the exact problems described in this post. A graphics engine, by design, must be able to do millions of different things that are created by the interaction between relatively few methods, none of which can go wrong at any time. One of my new features is probably buggy, but attempting to brute force test border conditions requires tens of thousands of tests because of all the interacting elements, any two of which could interact in just the wrong way to blow something up. It gets even more ridiculous when you have precision bugs, where only certain numbers in certain ranges in certain cases will explode. Testing that using inputs is impossible.

This occurs so often and with such regularity I am now convinced that everything I write is riddled with bugs that I will probably never find without beta-testing in real-world scenarios, plus many more bugs I will simply never find because they never come up.

A much better design would be a test platform that analyzes the assembly structure to pinpoint actual edge cases in the code itself, which could then be used as a guide for finding bugs instead of relying on hundreds of thousands of test cases.

gambler5• 15 years ago

Depending on the language you're using, there might be some tools to help you.

For example, MS recently released a library for Design by Contract (aka Contract Programming) in .NET 4.0. It allows you to specify constraints on your methods, such as things you expect to be true before the method is called, after, and between all method calls in a class. The library is capable of static verification, but it's partly an experimental feature and you need to pay for expensive version of VS. (Runtime checking is available for free.)

But, here is the cool part: MS also released an automatic test generator called PEX. It can do exactly what you've described - go into your code and automatically find edge cases, and generate tests that cover each of them. And it's free.

So, you can write contracts, run PEX, and if something goes wrong, you will see which inputs generated exceptions.

D also has DbC functionality. I don't think it runs any static verification on it yet, but you can use it to detect abnormalities during functional testing.

blackhole3• 15 years ago

That would be fantastic if I didn't write everything in C++

xentronium2• 15 years ago

Genuine question: doesn't such thing as "design by contract" make for the same bloat as checked exceptions in java?

gregors2• 15 years ago

pex link http://research.microsoft.com/en-us/projects/pex/

alexatkeplar• 15 years ago

It sounds like you might be better with something like QuickCheck for your testing: http://en.wikipedia.org/wiki/QuickCheck

This was featured recently in Bryan O'Sullivan's "Running a startup on Haskell" presentation: "[QuickCheck] generates random data, then tells me if all properties that I expect to be true of my functions and data hold. QuickCheck is shockingly more effective at finding bugs than unit tests."

There are ports of QuickCheck to tons of other languages...

blackhole3• 15 years ago

This gets complicated in a graphics engine where the only place you can tell something has gone wrong is after the image shows up in the wrong place. It would help a lot for the majority of cases, though.

osivertsson• 15 years ago

If you have interactions between relatively few methods, then unit-testing those interactions shouldn't be too much of a problem. You wouldn't test all possible inputs, but focus on making sure that you have a test-case for each input that needs to be handled differently. Bugs will still creep through, but they will be substantially fewer.

If you really do need tens of thousands of tests because you have so much interaction with unique states going on in your engine then you will probably never ship a product with any reasonable quality.

And that floating point operations lacks precision is just the way things are, and you need to understand how to write stable code despite it if you want a stable graphics engine.

fizx4• 15 years ago

The author argues that TTD fails in code that's largely wiring. I think the opposite is true. I'm writing an application that's largely wiring, and for the first time in a long time, using TDD. It's refreshing.

Most of my tests are two-liners, like:

    it "should do x when y" do
        obj_under_test.should_receive(:consequence_method).with(some_args)
        obj_under_test.do_something
    end

There are a number of reasons why this has helped:

1. It ensures I'm using dependency injection, etc to write testable, well-factored code. There's huge correlation between testability and maintainability. 2. I don't have to boot the thing all the time to confirm that I didn't mess up in some obvious way. Covering the code paths prevents typos. 3. My test suite runs in under three seconds. I can sanity check what I'm doing, without being tempted to browse HN/reddit/twitter/etc.

I like TDD more in wiring-only code. If I'm writing wiring, I know what my test case will look like ahead of time, and I'll write it first. If I'm writing experimental algorithms, I have no idea what will happen, and I'd rather write code and poke it.

shadowfiend5• 15 years ago

I know I'm pulling out raganwald's argument from one of his posts, but I'm honestly curious: is there anything other than anecdotal evidence that “there's huge correlation between testability and maintainability”? In particular, here we're talking about unit testability. I don't agree or disagree—I've done both TDD and not, but I'm still not sure I've created more maintainable code when testing than when not.

fizx4• 15 years ago

Testable code is loosely coupled, because it has to interact with mocks. Untestable code tends to have things like hard-wired http clients, because it was "easier" to just inline the constructor. When you go back to upgrade the current version of your http client later, you'll appreciate that you didn't hardwire it.

phzbOx3• 15 years ago

Maintainability is not just about you changing your code later on; it's about someone else trying to understand what you were thinking when you write it.. and then change it. Tests make that process way easier. You can add your new feature without wasting time to understand everything else and still make sure it's working. If it's your own project, meh, you know your stuff. You even know all the hack you did to gain time. I see tests as a documentation that shows me what's working. Reading tests (high level ones) is usually the first thing I do when starting on a new project. Comments change over time.. documents aren't updated.. people leave.. but tests remain. If the test passes, it doesn't mean everything is perfect, but at least you know that these things work.

shadowfiend5• 15 years ago

Replying to both of you so replying to myself -- I know all of the arguments for why testing improves maintainability; that has nothing to do with what I'm asking. It was my mistake for muddying my question with my own anecdote.

I'm asking: is there scientific proof (i.e., in a study) that testing and maintainability are at least positively correlated in a reasonably universal fashion?

xentronium2• 15 years ago

What happens when you want to rename and/or split your consequence_method? How many tests do you have to rewrite?

fizx4• 15 years ago

Conversely, what happens if someone else renames/splits consequence_method? Without this unit test, how would they have known they broke something?

Unit tests help refactorability in dynamicly typed languages. Whether this is an argument for static typing is left as an exercise for the reader.

jasonlotito11• 15 years ago

What happens when you refactor your code? It's fairly obvious what happens: you change the name of the method being called. However, the benefit here with TDD is you still retain your testing ability. More importantly, anywhere in the code also using the method can be easily discovered. Finally, by doing this, you've automatically updated your documentation as well.

So, let's see, you've updated tests, code, and documentation all in one. Whereas if you didn't have tests, you'd have to update the code through manually searching and testing everything, and still have to remember to update the documentation.

texel• 15 years ago

I don't like this kind of mocking because usually, I don't care what methods are called internally, I care what the outputs and side effects are. There's almost always a better approach than expectations.

aduric• 15 years ago

Agreed. I'm currently writing a package that contains a few sophisticated algorithmic parts and a ton of wiring. I find it much easier to utilize TDD in the wiring sections than in the algorithmic parts. I still use TDD for everything, it's just that I find myself to be more easily dogmatic about TDD in the wiring code.

LVB• 15 years ago

I generally agree with the final sentence: "If you choose a methodology without comparing it against alternatives, if you claim that it works all the time without evaluating the results, than you're not doing engineering - you're practicing a religion."

I've definitely seen some negative effects when a team is forced to create a huge volume of low-level tests because they perceive that to be the only acceptable solution. They get bored with the work, and worse, think less about the larger integration issues.

I'm not arguing that TDD fails, but you'd better monitor the efficacy of whatever testing regimen you employ lest you suffer process and quality rot.

P.S. Dude needs to make friends with a spell checker... wow.

ctide4• 15 years ago

I got as far as: Oh, and good luck mocking your database and HTTP request objects. and just stopped. If you don't have any experience with TDD, then don't write about it. Mocking HTTP request objects is incredibly trivial and tools are written in every language to do just that.

I don't know why these articles keep getting written. No one's forcing you to write your tests first, and if you can't wrap your head around the benefits (which the author clearly can't) then don't do it. It's fine. I'll still continue to utilize TDD, and continue to pump out significantly more code (with significantly less bugs) than the code I produced without TDD. Yes, there's definitely code I write where I don't write the tests first. It happens, sometimes it's just easier to bang something out because you aren't sure what you're building. That doesn't mean TDD is a failure, it's a tool, like anything else. Use it where it's appropriate, because the benefits are massive.

Retric2• 15 years ago

IMO, TDD is a crutch. It can be useful but don't think just because they make you faster they they will continue to be necessary. I know plenty of people that found TDD extremely helpful but most of them have moved on, not because they had not found it useful but because they slowly got faster without it. I think it works like training wheals because you notice problems sooner you can start to recognize they approaches that lead to those problems and avoid making those mistakes in the first place.

PS: Having a great tests can really help maintain code, but I would much rather have cleaner code than more tests.

ctide4• 15 years ago

That's not why it's faster. It's faster because I never have to manually run through whatever I'm developing. On a large feature that I can completely write using TDD, I don't ever 'test' the feature until I'm completely finished. There's a MASSIVE time savings in not ever having to walk through anything in a web browser, and never creating any regressions while fixing anything that breaks later in development.

pbz2• 15 years ago

You should have continued. His point is that while TDD is good for some things it misses huge areas and has very little ROI in most simplistic cases.

Yes, you can mock HTTP requests, but there are numerous bugs you wouldn't catch unless you went and hit your application with an actual browser. Similarly, you can mock the database, but you're not testing all the "magic" the database does and all the myriad combinations it could fail.

ctide4• 15 years ago

I have no intention of ever mocking the database, and I mock HTTP requests solely so I can work on my app without an internet connection. Look, a better approach is to use TDD for things it makes sense for. Spending a ton of time to mock out your database does nothing for you. Is mysql going to fail for you? Is the mysql driver that a million people are using going to fail for you? No? So, don't mock it out.

regularfry2• 15 years ago

For those cases, tests have value in telling you where the bug isn't.

For me, one of the biggest benefits of code that has been written to be testable is that it's been designed such that it's possible to reason about fine-grained details. If I find a problem, I can generally start adding tests as part of the debugging procedure to make assertions about the specific part of the code that I think might be failing.

That's much harder to do at a sufficiently high level of detail if the code wasn't structured to make it easy from the start.

jasonlotito11• 15 years ago

> Similarly, you can mock the database, but you're not testing all the "magic" the database does and all the myriad combinations it could fail.

You are correct. And this is why you unit test the DB and all the stored procs and queries you perform there.

And, this is also why you can add in tests that do use the database connection, and test the more complete stack. TDD does not limit you to only use the tests at the application level. It does not limit you to building other forms of tests. To imply that is dishonest. It would be akin to me saying "Because you do not use TDD, you do not test."

SoftwareMaven• 15 years ago

No one's forcing you to write your tests first

That is very likely not true. If TDD is your "process" at work, then you are being forced to write your tests first.

If TDD was chosen to be your process because of erroneous assumptions about the values of TDD, its strengths and its weaknesses, and those erroneous assumptions are a result of religious zealotry squashing the heretics rather than reasoned debate, then you have a problem.

Religion belongs in the church, not in software. It frustrates me how poor the technical debates in software can be because people get so attached to their technology. It is directly analogous to an artist demanding a specific type of paint for all their work who freak out when somebody says "you know, a gallon of latex paint might be better than that tube of acrylic for painting the walls in here."

philwelch2• 15 years ago

Unless someone is hovering over your shoulder, how is your employer going to tell the difference if you just commit tests and code together?

wnight3• 15 years ago

> If TDD is your "process" at work, then you are being forced to write your tests first.

Not necessarily. Usually you're only required to write a test before you start working on a specific feature. If you just want to research something I've never seen anyone who'd get in the way of you pulling up a REPL.

But even so, saying you're forced to is like saying your company forces you to check your work into a VCS before they'll give you credit for having written it. Or forces you to fix bugs/add features from a list instead of just ad-hoc.

mml• 15 years ago

More accurately, nobody's forcing you to take a job where you disagree with the methodology.

I've had to deal with agile whining for nearly a decade now. The only thing to do is ask them to find work elsewhere: they won't be happy on an agile project, and the agile project won't be happy with them.

sathish316• 15 years ago

I'm surprised not many people have talked about BDD. BDD lets you focus on the behavior of a method, class, or a set of methods/classes that interact with each other without worrying about the internal implementation details, unlike TDD where you test every method and may end up with tests for dead code. BDD lets you paraphrase what you're trying to do and tells you when you're done. Having said that, i have to agree that TDD/BDD is just a tool and no one should be forced to using it as the only method.

As far as the problem of mocking http requests is concerned, most frameworks like Rails/Spring come with http mocks. But a much better solution is to make your controller code as trivial as possible and move as much of your business logic to models. RESTful design helps you solve this problem. This is not just useful for web applications. Moving your logic from controllers to models is a good MVC principle too. It gives you the flexibility to use a different GUI toolkit for the same code.

As far as the problem of mocking database calls, it depends on whether you prefer your test suites to be real or fast. For simple database calls, I'd rather hit the database and end up with tests that assert behavior than tests which redundantly repeat what the code is trying to do using mocks. Mocking is still a useful technique for things you can't control like an External API.

TDD/BDD is useful for most of the model-centric work we do in Web based software. I'm not sure how useful it would be in areas like language design, framework design, mathematical modeling, algorithm design etc. QuickCheck or some other form of program verification will be a much better choice if your work involves designing/verifying complex algorithms. Djikstra's quote:"Testing shows the presence, not the absence of bugs" points out the problems with TDD. TDD ensures your system satisfies the specs and the addition of new features don't break existing behavior. But it does not protect your system from behaviors you did not anticipate when you wrote the code/tests in the first place. TDD/BDD will not make you a smarter programmer. Genius programmers like Rich Hickey don't need TDD. Hey, But we are mere mortals who work with other mere mortals in building non trivial systems. And TDD is just one of the ways to ensure fewer bugs in the system.

recursive2• 15 years ago

No one at my work place, including me, understand unit testing or TDD. I was recently asked to add a test suite to a service I wrote that is basically a simple wrapper that returns the result of a stored procedure call. The only test I could think of was to call it with all null parameters, in which case there should be no output. Other than that, the results depend on the state of the database. I'm familiar with the concept of dependency injection, but I couldn't add it to this very simple service in good conscience, since I knew that adding the necessary complexity would only increase the likelihood of a defect.

shadowfiend5• 15 years ago

Traditionally, in these cases, you use a test database with sample data (fixtures) to test that the output depends on the db data in the right way.

petercooper4• 15 years ago

As shadowfiend notes, fixtures are a common approach here.

However, you could also use a mock. What you're really testing here isn't the connection to the database or even that the database contains certain data, so you can rule that out of the equation. What you're testing is that some service (let's just say a 'method' for simplicity's sake) turns an input into an output in some particular way.

What you do, then, is mock the database connection for a particular test case so it returns a guaranteed result to whatever's doing the request in your method. You can then test that the method converts that input into the correct result. You've now unit tested the method rather than the entire service (in a nutshell - it can be more complex than that).

j-kidd• 15 years ago

I don't get it. Why would one mock the database connection when the thing to be tested here is a stored procedure?

jrockway3• 15 years ago

I very rarely use TDD, but I am a fan of it. First off, absolutes like "always write tests" are for people that are bad at programming but are still employed as programmers and can't be fired. They haven't developed the judgement for when to write a test or when not to, so in the interest of getting some reasonably useful test suite, you say "you must write a test for everything".

Secondly, I don't really agree that these action methods are untestable. Sure, "print hello world" is untestable, because it's so simple that you're not going to fuck it up, and because there's only one execution path that can possibly occur. But most methods are not like this; they need to reject invalid data or state, they need to craft database queries, and so on. In that case, you very well can write good tests for this sort of thing.

Say I have some code that needs to accept an HTTP request that has a "foo" parameter:

    def do_something(self, context, foo=None):
        if foo is None:
            raise context.UserError( 400, 'You must supply foo.' )

        context.get_model('Something').apply_foo_to( user=context.current_user(), foo=foo)
        return context.get_view('Something').render()

This is easy and valuable to test:

    container = DependencyInjectionThingie()
    context = container.get_fake_context( current_user='jrockway' )
    controller = container.get_fake_controller( DoSomethingController )
    something  = container.get_fake_model( Something )

    # ensure that empty foo is rejected
    raises_ok( lambda: controller.do_something( context ), UserError )

    # ensure that something is mutated correctly
    controller.do_something( context, foo='OH HAI' )
    compare( something.applied_foo, '==', ('jrockway', 'OH HAI' )

(Who would have predicted the day where I started writing my HN examples in Python!)

In just a few lines of code, we get a little bit of extra security around our do_something action. We are sure that a UserError is thrown when foo is not provided (which our imaginary framework turns into a friendly error message for the person at the other end of the HTTP connection), and we're sure that the model is mutated correctly when foo is valid. In three lines of code.

I find that people that have the hardest time writing tests have poorly-architected applications that don't lend themselves to easy testing. The key point to remember is: if you don't pass something to an instance's constructor or to a method, don't touch it. Then everything is easy to test, because you can isolate your function (or class) from the rest of the application, and then test only what that function is supposed to do. (In this case, the fact that a UserError exception becomes an error screen is something you test in your framework's tests. Same goes for the fact that view.render() renders a web page; test that in your view tests.)

This style of development is also good for more than just testing. A few months ago, I wrote an application that monitored a network service. Not wanting to rewrite the service or mock it, I pointed my tests against a dev instance of this service. Everything was great until the service blew up on a Friday night and nobody was around to fix it. Faced with not being able to write any more code until Monday morning, I knew I had to fake that service somehow. 20 minutes later, I had a class with the same API as my "connect to that service" class. I changed one line of code in my dependency injection container (to create an instance of my connection-to-fake-in-memory-server instead of connection-to-networked-dev-server), and then I was back in business. That's the beauty of writing code to be flexible: you don't have to get everything right on the first day.

(People will argue that tests should never depend on external services, because they can blow up and then you're fucked. Yes they can, and yes you are! But while I didn't do everything right on the first day, my design allowed me to recover from this mistake without any code changes. And now I just run the test suite twice before a release; once against the fake connection and once against the real server, just to make sure that whatever assumptions I made in the mock server also hold when connected to a real server. I like releasing code that I know works in real life in addition to my fantasy mock world, but that's just me, I guess.)

Edit: and oh yeah, it's easy to mock databases and HTTP requests. We've seen the second one already; you let your framework translate between HTTP and method calls, and you write the tests for that when you write your framework. This frees up your application developers to Not Care about that sort of thing, allowing them to write great tests with minimal effort. The first one is also easy. You write code like:

    class UserModel(Model):
       def __init__(self, database_schema): ...
       def last_login(self, user):
           self.database_schema.get_resultset('User', user=user).related('LoginTime').order_by('time DESC').get_column('time').first()

Then when you're testing your controller, you pass in a fake UserModel that just defines last_login as something like:

    class FakeUserModel(Model):
        def __init__(self, database_schema): pass # don't care
        @memoize
        def last_login(self, user): return datetime.now()

The code to ensure that last_login generates the right sequence of operations on your ORM is somewhere else. The test that your ORM generates the right SQL AST is somewhere else. And the test that tests that ASTs are converted to correct SQLite SQL is somewhere else. You already wrote and tested that code. Assume it works!!!

Yes, sometimes you will write a few end-to-end tests to ensure that when an HTTP request that looks like foo arrives on the socket, you write a HTTP response that looks like bar to that socket and your database's baz table now contains a record for gorch. But that's not how you test every single tiny thing your application does; it takes too long, it's hard to get right, and it buys you nearly nothing.

So I guess I add: testing is hard if you write your tests wrong.

irahul• 15 years ago

> I find that people that have the hardest time writing tests have poorly-architected applications that don't lend themselves to easy testing.

It's generally good to start with some framework which provides capabilities to easily mock most of the objects. You sure can architecture your application as such, and have your dependency_injection_thingie to mock objects, but I don't think it's a worthy investment of time.

> (Who would have predicted the day where I started writing my HN examples in Python!)

So why is that? Working on a Python application? While you are there, check out decorators and co-routines/generators. You already have checked out decorators(which are basically function composition - same in Perl other than the syntactical sugar) - I see your @memoize example.

Decorators along with Python introspection are super cool. I used decorators to implement a small contract system - https://github.com/thoughtnirvana/augment

I recommend this talk on co-routines http://www.dabeaz.com/coroutines/

EDIT: Perl has coro. But the language integration(generator expressions, convenient yield) makes it a bit more natural in Python(YMMV). And Python has gevent if you are looking for a threading equivalent.

igouy• 15 years ago

> Sure, "print hello world" is untestable

Redirect stdout to a file and diff against a file containing "hello world" :-)

LeafStorm• 15 years ago

While testing code is important (and I have tried to do better at writing proper tests for my code), the reason that I have not adopted TDD is because when I write a test and it fails, about half the time it's an actual bug in the code, and half the time it's a bug in the test. I view testing as more akin to going back and checking your work after solving a math problem - not as the definition of what your code is supposed to do, but as a verification that you did it right (and that it continues to work right in the future).

wnight3• 15 years ago

> when I write a test and it fails, about half the time it's an actual bug in the code, and half the time it's a bug in the test.

Well yeah... You'd expect otherwise?

Do you mean when the test fails that it's some sort of not-bug like you changed your mind about what the method should do since writing the test?

jarin• 15 years ago

I don't know, it's not that hard.

Write acceptance tests (with Cucumber or something), use a good unit test helper like Shoulda for your standard unit tests, and write unit tests for any complicated methods.

The acceptance tests will cover pretty much everything, and the unit tests will cover anything that's hard to test from a high-level point of view.

plinkplonk• 15 years ago

The real "problems" with TDD are

(a) in the "driven" part, not the "test" part. Tests are (in general) a good thing. However, using a series of tests to drive your design (aka "TDD is not a testing method,it is a design method" idea) often gives you an illusion of progress as you hill climb using conformance to an increasing number of tests as a progress heuristic and end up on top of a local maximum (as for example in the TDD sudoku episode).

(b)in conflating TDD with one or more of (1) testing, (2) automated testing (3) automated regression test suites (4) developers adding more tests to the automated regression test suite as they develop more features, refactor, debug etc.

You can have (1) to (4) without either (5)writing tests first (aka "don't write a line of code without having written a test covering it") or (6)driving your design with tests. The last two ideas are the real distinguishing features of TDD and are of debatable merit. None of (1) through (4) are novel ideas. (5) and (6) are where differences of opinion happen.

Even if you choose to use TDD, it is good to be aware it is just one tool in your toolbox and not necessarily the default tool to reach for.

(c) in the zealotry of some of its evangelists who insist that TDD is some kind of moral imperative and is the only "correct" way of developing software and anyone who doesn't follow that path or make respectful obeisance to it is "unprofessional","dodgy" etc. This is often accompanied by conflating TDD with more generic notions like "automated tests" etc as above.

For example, Rich Hickey, the author of Clojure, said recently at the Strange Loop conference "We say, “I can make a change because I have tests.” Who does that? Who drives their car around banging into the guard rails!?"

(and that is all he said. One sentence in a keynote presentation)

For this Hickey was taken to task by a TDD advocate, Brian Marick, for not being "respectful" enough to TDD and for his "tone" in daring to mildly criticize it as a development practice. After some tweets complaining about Rich Hickey's tone driving away people from Clojure etc he wrote

http://www.exampler.com/blog/2011/09/29/my-clojure-problem

"The dodgy attitudes come from the Clojure core, especially Rich Hickey himself, I’m sad to say."

This kind of repeated whining and harassment over a few days made the normally unflappable Hickey (who asked for references to his "disrespect" etc, to the sound of crickets) lose his temper and say (on his twitter stream)

"If people get offended when their tools/methods are criticized as being inadequate for certain purposes, they need to relax.",

and "Testing is not a strawman. It's an activity, it has benefits, but is not a panacea. No 'community' should be threatened by that fact"

and later "Accusing people who merely disagree with you of being snarky, intolerant, dismissive etc is both wrong and destructive."

and much later after being subjected to a barrage of tweets criticizing his tone and 'lack of respect' for TDD, "If launching an ad hominem attack is the product of a lot of thought, it is time for you to move on. Good riddance."

postscript: the best criticism of TDD I've seen is at http://www.dalkescientific.com/writings/diary/archive/2009/1... . The responses at http://dalkescientific.blogspot.com/2009/12/problems-with-td... are (mildly) interesting as well.

phzbOx3• 15 years ago

I don't get why the fact that TDD drives good design is a bad thing. When you're not testing your code, it's easy to start taking shortcuts and coupling modules that should have been kept separated. TDD forces you to think about "modular design", which is just great. Of course, if you're trying lots of solutions to find "the right one", don't write tests. Just write hacky prototypes as fast as you can.

  "(b)in conflating TDD with one or more of (1) testing, (2) automated testing (3) automated regression test suites (4) developers adding more tests to the automated regression test suite as they develop more features, refactor, debug etc.

You can have (1) to (4) without either (5)writing tests first (aka "don't write a line..."

I get here that you're being sarcastic; but there's a difference between using TDD effectively to get the job done and using it "because it's cool" or "because people say it's nice". You don't have to write tests for every trivial things, or write tests for your tests of your tests. Lots of time, writing high level tests testing for the interfaces is a good time/quality ratio.. there's nothing wrong with that.

About c). I do agree. However, have you ever jumped into a project without tests? With tests? Which one do you prefer? Tests don't prove that the library is working perfectly; they only prove that what's tested is working. Personally, jumping into a new project that have tests give the confidence to change things and know I won't break anything. Of course, that's not always the case but I think you know what I mean.

And about Hickeys.. I haven't heard the speech and don't know about the context. But changing stuff in your language is something; but again, starting to change stuff in someone else' is different.

samfoo• 15 years ago

Just because you're not doing TDD doesn't mean you're not testing your code. I don't need to be forced to think about modular code, because I (try to) think that way voluntarily.

One of the most frustrating things (to me) about BDD/TDD is that I write a test knowing it will fail (a good thing), then I implement just enough to pass that test. Even though I know that I'll be rewriting that code again very soon to implement just enough more to pass some other test. It's needless context switching when I have a general idea of how I want the code structured before I start writing anyway.

With more experience, you should learn to avoid many of the pitfalls of not writing/designing modularly enough in the first place and at that point writing your tests first becomes a productivity crutch.

0x123• 15 years ago

> I don't get why the fact that TDD drives good design is a bad thing.

TDD does not in and of itself lead to good design.

JoachimSchipper2• 15 years ago

This comment should not be [dead] (if you don't want to give/dock me karma for irahul's comment, down-/upvote my reply to this comment):

irahul 7 hours ago | link [dead]

> For example, Rich Hickey, the author of Clojure, said recently at the Strange Loop conference "We say, “I can make a change because I have tests.” Who does that? Who drives their car around banging into the guard rails!?"

Rich has spoken about it other time with an interview with Fogus:

http://www.codequarterly.com/2011/rich-hickey/

Hickey: I never spoke out ‘against’ TDD. What I have said is, life is short and there are only a finite number of hours in a day. So, we have to make choices about how we spend our time. If we spend it writing tests, that is time we are not spending doing something else. Each of us needs to assess how best to spend our time in order to maximize our results, both in quantity and quality. If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem. I’m certain that, for me, this produces better solutions, with fewer defects, than any other use of my time. A bad design with a complete test suite is still a bad design.

He said something on the similar lines about development on CLR:

Fogus: Clojure was once in parallel development on both the JVM and the CLR, why did you eventually decide to focus in on the former? Hickey: I got tired of doing everything twice, and wanted instead to do twice as much.

His explanation on both fronts boil down to he doesn't find it(TDD and CLR/JVM parallel development) a worthy investment of time, given there are only so many hours in a day.

I don't understand why TDD advocates get all worked up when someone says TDD doesn't work from them. Well, if TDD is silver bullet of software development, they should be delighted that the ignorant singletons fail to see it and they have an edge over the fools.

These reactions remind of this:

“You are never dedicated to something you have complete confidence in. (No one is fanatically shouting that the sun is going to rise tomorrow. They know it's going to rise tomorrow.) When people are fanatically dedicated to political or religious faiths or any other kinds of dogmas or goals, it's always because these dogmas or goals are in doubt.” ― Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values

JoachimSchipper2• 15 years ago

(karma sink, see above)

DrJokepu• 15 years ago

It looks like irahul was hell banned yesterday for the following comment, that's why his comments show up as dead:

http://news.ycombinator.com/item?id=3059454

wnight3• 15 years ago

> the zealotry of some of its evangelists who insist that TDD is some kind of moral imperative and is the only "correct" way of developing software and anyone who doesn't follow that path or make respectful obeisance to it is "unprofessional","dodgy" etc.

Well, it is (the only correct way of developing software). If you're writing software without a test, even if only a mental one, you're just goofing off. And if it is mental, write it down and let the computer check it for you.

It's like not writing specs. (Not necessarily the executable type.) If you don't know what you're trying to do how do you know when you're done?

As for what people who confuse specs with acceptance tests think, I wouldn't judge a product/methodology by those who don't use it well.

ocivelek• 15 years ago

IMHO, there're two points here; 1. Unit testing is not TDD 2. TDD has nothing to do with software testing

TDD is good for developing rapid changing codebases with short development cycles with known requirements and a harsh time pressure, but it's a front loaded way as well.

By which I mean, you have to invest some hours beforehand to make sure you're not shipping crap because you didn't have time to see what would break or to make sure you didn't skipped the controller your GUI developer needed.

TDD tries to ensure one thing only: "Awareness". You accept the initial costs of TDD if awareness is a big issue for you. Otherwise, use something else which serves you to solve it.

If you're aware of what you will be implementing and if there's a mechanism you can check your code against, it'd be efficient, right? TDD uses unit tests for documenting that "Awareness", aiming to utilize the benefits of unit testing as well.

You're right, that it is not feasible trying to test every aspect of your solution via unit tests and that's why there are integration tests, systems tests and acceptance tests. So if you're planning to find defects via the unit tests you're writing for your TDD cycle, think again. Unit tests are good for checking "completeness" and a great tool helping with regressions. Nothing more and nothing less.

As a good engineering practice, we adopt the method of working, based on how we planned to work. If we find TDD fruitful to implement, we avoid putting any logic to controllers. We implement our business logic behind the controller, which also cleans it off of implementation specific crap. Besides us using a completely detached GUI layer helps us write controller unit tests running via HTTP.

We simplified our working environment and make it suitable for TDD, by ways we found to provide more capabilities to us. If we're not to use TDD, we employ other logics and structures, which would fit well with the method we'd use instead.

Long story short, TDD gives what TDD is intended for and as long as you cooperate. If you expect more than it can provide, you'd be disappointed. Regardless of your development method, you have to make sure your development/architecture models and your tool set complies with your method of choice. The rest relies in the question "What do you need your development model to solve for you?"

All the best

mojo852• 15 years ago

The author is complaining that when requirements change that his tests need to be updated/rewritten.

Is he joking? I mean if the updated requirement changes the behavior of the code then the test better freaking fail and require the test to be updated, otherwise the test (if it even exists) is terrible.

yason• 15 years ago

IMHO, good testing is hard. I think that something that's probably as hard as writing programs in the first place shouldn't be commoditized into a methodology. I don't particularly dislike TDD, but I certainly don't like it either.

The best phase to write tests is when you've locked down a part of your program. A part such as one distinct submodule or function or the sort.c or nodegraph.c of your latest project--a part that is relatively orthogonal to the rest of your code. That sort of ensures that the basic blocks, once finished, won't fail surprisingly. However, this can only be applied to basic building blocks.

Testing the bigger parts of your non-trivial medium-size program is likely to be so hard and complex that you have no chance at planning testing beforehand. I think that a good programmer or tester can come up with a relatively comprehensive testing suite that triggers execution paths up to 80-90% code coverage if given a sufficiently finished program, i.e. the program structure has mostly stabilized. Good programmer can also make changes to the same program without decreasing the quality of code. Bad programmers are as able to write good testing as modifying the code itself without letting entropy to the driver's seat.

jamieb• 15 years ago

IMHO, good testing is hard. I think that something that's probably as hard as writing programs in the first place shouldn't be commoditized into a methodology. I don't particularly dislike TDD, but I certainly don't like it either.

Completely agree.

The best phase to write tests is when you've locked down a part of your program.

Completely disagree.

RodgerTheGreat• 15 years ago

To elaborate, writing tests can very quickly make you aware of shortcomings or clumsy aspects of your APIs, since it should be the first time they're actually used.

jannes• 15 years ago

Using TDD everywhere is stupid. I think, tests are most useful between layers of abstraction, especially if you are the provider of abstraction. If you are merely a user of abstraction, it is stupid to unit test that code.

If you have an interface to the outside, then that should be unit tested.

Volpe• 15 years ago

This blog and some of the comments below are just crap. If you haven't TDDed before (for a non trivial project) you simply don't know what you are talking about, and it's apparent in your ignorant opinions.

Sorry if that's a bit harsh but I'm tired of seeing these posts by people who don't get it.

I don't get quantum physics, that doesn't mean I'm going to write an blog "when quantum physics fails"

If you don't understand something... Learn about it, practice it, then criticize its actual flaws.

Don't just get frustrated at something and claim its crap. Because its clear to those of us that do understand it, that you are talking crap

ludflu• 15 years ago

"TDD is nearly useless when your code is the opposite of what I've described above: specific and mostly trivial, with complexity coming from the sheer amount of methods and their interactions. "

In other words, badly factored code is hard to test and maintain. Its just that with TDD its more obvious why badly factored code is bad.

romaniv• 15 years ago

More like wiring or integration code. Not everything is a framework. At some point you have to write code that works with and inside of the framework.

gregors2• 15 years ago

To the dev's who take the time to write tests, when I inherit your code base I feel like buying you a beer!

For the dev's who don't - no matter how 'clean' the code. FFFFUUUUUU

todojunkie• 15 years ago

This is a difficult topic. I've found that adhering to TDD is just not realistic in some cases.

For example, I'm following the Steve Blank approach of Customer Discovery, building my MVP. If I take the TDD approach (which takes more time to code a finished piece of functionality), and successfully iterate enough times during the Customer Discovery process, I throw out all of the tests that I built and am now moving onto something different that I've discovered a customer is actually willing to pay for.

When TDD Fails

Related Stories