Is the development of automated tests a waste of time for a startup?

https://news.ycombinator.com/item?id=27875

by richcollins • 19 years ago

13 45 19 years ago

news.ycombinator.com

richcollins11• 19 years ago

I always go back and forth on this one. It always feels great when the tests catch an introduced bug, but it is really worth the time you could be spending on testing out new features with users?

I wonder if the solution is to be careful to write very simple, well designed code.

nostrademons• 19 years ago

I've gone back and forth on this issue too. I don't really have a firm opinion on the subject, so all I can do is share my experiences.

At inAsphere.com (2000 teen content dot-com), we had no tests and didn't really miss them. Our software was dead-simple though: a bunch of 3rd-party forum and link directory stuff. We got killed for other reasons (i.e. there wasn't really a market and we kinda sucked content-wise).

At Traxit (2000-2001 venture backed startup I worked at, selling remote access software to get through firewalls) we had no unit tests, and it sank the company. We got into a QA death spiral, where every bug that QA found made the engineers introduce 5 more, and the product quality suffered so much that we cancelled our launch. Cancelling the launch is the absolute worst thing you can do if you have venture funding, because the VCs will swoop in, throw out most of the management team, and try to pick up the pieces. Of course they'll fail, so the company folded about 6 months after I left.

At FictionAlley (2002-2005 nonprofit Harry Potter fanfiction archive), I started with comprehensive unit tests, determined not to repeat the mistakes at Traxit. I then got frustrated with how I was spending as much time writing test code as production code, and abandoned them. It eventually launched, a little buggy, and the users were reasonably happy despite some annoyance at the bugs. However, the lack of up-to-date tests has made it very difficult for new developers to come up to speed and change things, which means that imperfections in the architecture have kinda fossilized.

At my previous project at my current employer (2006 IDE plugin), I released without any unit tests. I then spent 2-3 weeks immediately after release writing a fairly comprehensive test suite for it. Overall it's been pretty successful: product quality is fairly high, and the tests have caught several insidious bugs since. I find they're most useful as a sanity check to show me I'm about to do something stupid. Often I'll do something that looks like a trivial refactoring and then roll back when I find out it's made a whole bunch of tests fail.

At my current project at my employer (2007 financial webapp), we have no unit tests. We are getting along okay without them, but that could be because we haven't really met reality yet. When my employer has released previous products (again, without any unit tests or QA), we've had to deal with some pretty pissed-off customers because of the product quality or lack thereof.

At my startup (2007, flash games website), I've had fairly comprehensive unit tests from the beginning. I haven't been running them lately, and they're a bit out of date, again because it takes as much time to maintain the tests as to write the code. While I was running them, they were useful in the same way as above: they keep me from doing stupid things and making silly mistakes. I plan to go back and make them pass 100% before or soon after launch.

If I had to summarize, I'd recommend that you skip them until after launch, and then any time you don't have an imminent feature request, spend that firming up the test suite. That way you don't delay your release too much but they'll still be there for you when you need them, and you also don't start writing them until the feature set has somewhat stabilized. They're a big investment, but they do pay you back.

palish• 19 years ago

For some reason, I really like the philosophy of "Don't write unit tests until right before you launch, then maintain them". Good observation.

sillydude• 19 years ago

It also depends on the size of your team and the quality of engineers and code. Having tests is usually a plus, but they don't always have to be a high priority.

tx4• 19 years ago

Without automatic testing dynamic languages are not very productive because they do not catch type mismatch errors during compilation time (there isn't one). With statically typed languages you get a lot of tiny issues taken care of once you hit magic "compile" button and compiler does some basic sanity checks.

Without solid unit tests backing you, activities like removing "obsolete" methods or renaming classes/modules turn into nightmare with duck typed languages.

richcollins11• 19 years ago

We didn't write tests for our last startup and we never had too many problems with bugs.

We do write tests for this one and we don't have too many problems with bugs.

Tests take almost as much time to develop as the features do. I just wonder if it is worth the time and if other people with startups write automated tests.

jsjenkins168• 19 years ago

I'd say try to plan for how much you will need to scale your code, and also how many others will be working on your code.

As your code gets larger, and as you start to have others working on it (who could break it), unit tests become more important. If you dont foresee either of these happening, then the time investment for tests may not be worth it..

tx4• 19 years ago

No difference? That's pretty impressive. I wish I was able to look at your process. Either your engineers are extremely smart and anal about their practicies or your domain is very simplistic (ta-da lists).

At my previous start up we had unit tests run on the entire system automatically after every check-in (with CruiseControl) and whenever someone broke a test suite, everybody got an email with name of the offender and diffs of his code against the trunk.

We also had a "beer rule" about violatros gettings drinks for everyone, to make the process a bit more exciting.

jimbokun• 19 years ago

Wil Shipley on unit tests:

http://wilshipley.com/blog/2005/09/unit-testing-is-teh-suck-urr.html

I think what he's saying is unit tests don't add much if you are doing UI type development. It is hard to write unit tests for UIs (web or GUI), so if most of your code is UI code, don't bother.

However, if you are writing a library that will be used by a lot of coders, unit tests are probably indispensable. Otherwise, you would never be able to make even simple architecture or design changes (refactoring).

SwellJoe2• 19 years ago

"Without automatic testing dynamic languages are not very productive because they do not catch type mismatch errors during compilation time (there isn't one). With statically typed languages you get a lot of tiny issues taken care of once you hit magic "compile" button and compiler does some basic sanity checks."

You're just exhibiting a basic dislike of dynamic languages. Type mismatch errors are mostly orthogonal to testing.

"Without solid unit tests backing you, activities like removing "obsolete" methods or renaming classes/modules turn into nightmare with duck typed languages."

This is a nightmare regardless of typing. If you remove a method that is still used somewhere, you've broken something, and you may not spot it until after it ships to customers. It doesn't matter whether you have type checks or not. A compiled language that links in those classes and methods at compile time does resolve this issue, but it's not related to types (and you have to wait on the bloody thing to compile every time you change one character in the file). Testing does resolve this problem--but it's not a problem specific to dynamic languages.

If you get a warm fuzzy feeling from type checking from your compiler, then by all means, enjoy it. But there are tools for most mature dynamic languages (Perl and Python for example) that will do that checking without having to write tests manually. But, having never found type errors to be a problem in my Perl or Python code, I've never used them. Some problem sets are more prone to type mismatch bugs than others, and I guess my areas of interest are among the ones that are rarely effected.

jey• 19 years ago

A compiled language that links in those classes and methods at compile time does resolve this issue, but it's not related to types (and you have to wait on the bloody thing to compile every time you change one character in the file).

How is it not related to the compile-time name resolution and type checking? In a compiled typed language, you would get a compile or link error if the symbol is not found. I regularly write and refactor C++ by inducing compile or link errors to identify the spots in the code where a specific method is called from.

weel• 19 years ago

You say that type mismatch errors are orthogonal to testing. This may be the case in languages like Java, which has a strong but not very expressive type system. In strongly typed functional programming languages like ML and Haskell, though, you can, with a bit of good design, capture a lot of information about what you want your program to do in the type system, to be checked automatically. Take, for example, the comment on this page by somebody who says that the only testing s/he does is XHTML validation. With strongly typed functional programming, it is possible, and not really even all that hard, to write an embedded combinator library for XHTML such that programs that generate improper XHTML don't type check in the first place.

nmeans3• 19 years ago

If you have humans writing code for you, tests are an imperative. Without tests, one seemingly small code error can wipe out a smatter of user-created data accidentally. Sure, it's worse case, but it could also mean lights out for your startup. You have to decide if it's worth the risk.

richcollins11• 19 years ago

Shouldn't you be backing this kind of data up? You can't guarantee that tests will catch this anyway.

nmeans3• 19 years ago

If you've got good coverage, you'll have a great chance that they will. You've got to actively monitor your coverage though.

As far as backups go, a backup is only as good as it's age. If a startup is struggling to decide if they've got the resources for writing tests, they probably don't have the resources for maintaining a real-time database mirror either. You can argue that you should always back up before you roll out a code update, but the real world's not always like that.

brlewis• 19 years ago

On ourdoings.com I've only ever had one error that could wipe out user-created data, and it would not have been caught by unit testing.

On the other hand, if I knew of a great tool for automated page tests I would use it. I don't think creation of such tests should be hard.

spejic• 19 years ago

I actually complete disagree with sillydude. Even though automated tests may make the team be a bit slower while developing, the suite sure as hell makes the code the team produces more reliable. So, while the speed of development is very important, so is quality of the product you get out there, even when it's something simple. The more small, simple bugs you can get out of the way before launch, the better. So, automated tests are, at least in my opinion, high priority.

richcollins11• 19 years ago

Is there evidence that time spent writing tests improves the quality of the code over time spent reading the code and trying to make sure it is simple and straightforward?

nmeans3• 19 years ago

When you've got 30,000 lines of code, you don't always have time to go back through and read all the code to make sure that the particular change you're making doesn't have drastic effects somewhere.

Writing good tests means you don't have to find and read the 200 relevant lines of code.

cwilbur3• 19 years ago

There's no evidence that reading the code and trying to make sure it is simple and straightforward improves the quality code, for that matter.

The quality of the code is a combination of the quality of the programmers and the quality of the processes they use to produce the code. If you have really good programmers, they can use a shoddy process (write, compile, ship) and still produce reasonable quality code; if you have average programmers with a good process (write test, write code, compile, fix warnings, compile, run test suite, fix errors, compile, run test suite, ship), you can produce good code.

What I've found is that whether a test will be useful or not depends on the nature of the code. If I'm writing logic code or building a domain-specific language, an automated test suite means the difference between getting it right quickly and taking a long time to get it hopefully probably mostly right.

Readability is only one aspect of code quality, and code that does something complex and convoluted will never be simple and straightforward. For that matter, complex, convoluted, and correct is preferable to simple, straightforward, and wrong, and that's the sort of thing that a test suite will tell you.

jcwentz• 19 years ago

Writing tests is just part of programming, whether it's for a startup or any other project.

richcollins11• 19 years ago

That is a pretty conformist thing to say

mdakin3• 19 years ago

Think of unit-testing as a tool and not a dogma.

Proper unit-testing roughly doubles the amount of time it takes me to write a "unit" of code. This is a high cost so I do not have 100% unit-test coverage.

Over time I have developed an intuition about whether I NEED a unit-test for a particular unit. When I think I need tests I write them.

Typically I'm correct about whether the units need tests but when I make a mistake (as evidenced by bugs in the code I didn't unit-test) I go back and write the tests.

This "middle way" works well for me.

richcollins11• 19 years ago

That sounds reasonable

felipe2• 19 years ago

Research has shown [1] that unit testing actually cuts development time by 20%-30%, because in the long run you will spend less time debugging code or fixing code broken by other changes.

[1] - I don't have time to look-up the specific link right now, so you'll need to trust me on this one! :) But I'm sure one can google it. I remember reading it on an IEEE magazine years ago.

richcollins11• 19 years ago

In what context does it cut development time?

mdakin3• 19 years ago

I don't want to come across as anti-unit-test or anti-studies but I suggest you run your own study and gauge your own performance rather than assuming you and your team perform as an average team does.

Thanks to the version control tools we use this is easy. Adopt a policy where you commit code for a SINGLE reason at a time. Enumerate a few standard reasons such as BUGFIX, FEATURE, FEATURE_AND_UNIT_TEST, REQUIREMENTS_CHANGE, UNIT_TEST_MAINTENANCE. Estimate the engineering time that was spent on making the change. Log that duration. Periodically analyze your performance and see for yourself how much time unit-testing is costing or saving you.

felipe2• 19 years ago

Another huge side advantage of automated tests is that it becomes a live documentation of how to use your code (unit tests as "sample code")

ricky_clarkson• 19 years ago

The larger your units the more important unit testing is. Now replace larger with smaller and more with less.

dhouston• 19 years ago

it depends, and in my experience can be reduced to a cost/benefit analysis. a sprinkle of different things has worked for me -- some things to think about:

what's the impact of a typical bug in your app: an error 500 page (annoying) or a BSOD or corrupted data (much worse) or a missile hitting a house? these will merit different levels of testing rigor.

if something is breaking, how do you even know? how much time will have elapsed? how long will it take you to find the cause, and is suitable debugging information available for your investigation? how quickly can you come up with a fix, and how expensive is it to deploy it? this is, in my experience, a much more effective place to optimize if you're a web app, especially if you're a small team.

for the non-core pieces of a web app, i'd rather release twice as fast and have logging and monitoring infrastructure in place to find and fix problems quickly (e.g. exception - logged and SMSed to our cell phones) than spend 2x to write unit tests (and maintaining them later.) a nice property is that you'll tend to find bugs roughly in order of decreasing impact, or at least prevalence. however, this approach won't do much against silent failures or bugs that are both rare and high impact.

there are lots of other effective tools: automated smoke tests (by which i mean end-to-end tests that quickly test large pieces of the app) deliver a lot of bang for buck, and root out unintended consequences of seemingly local changes that even unit tests sometimes miss. i also find generous use of the assert keyword finds failures fast/nearer to the cause.

tx4• 19 years ago

Lets not forget about quality of unit tests. I've seen people writing completely idiotic tests that really do not prevent them from anything (like calling methods and making sure they do not throw exceptions). Such "unit testing" is your time well wasted indeed. Another example is to go too low or too high, i.e. being unable to find a proper balance between unit/integration.

IMO proper unit testing is a skill and takes some time to acquire. Currently I am thinking about good interview questions for our candidates that help us find out how well people understand the reasons and goals of good UT.

Moreover, there are some areas of development that are notoriously hard to unit-test (with good results): Win32 UI is a good example, and even in those cases it's possible to write little, but highly effective UT-code (for instance if you have a unit-test-friendly design in place, like mediator pattern when you can "drive" your UI actions by sending commands to it).

tx4• 19 years ago

I find it really counterproductive whenever people bring the following arguments to the table (during discussions):

"I don't use it" or "Facebook does not use it". So? What does that prove? Who taught you gentleman, to use such useless arguments in a debate. It sounds like "I do not believe unit tests are useful because I do not use them". Your logic is reversed.

BTW Einstein did not unit test either.

wlievens• 19 years ago

My unit tests are mostly related to the core logic. The only UI testing I do is automated XHTML validation of pages generated for a test set.

dazzawazza• 19 years ago

I tend to write unit tests for core functions/modules as I develop them. It helps me create an API and to think about it's use/documentation. I also write tests whenever I come across a non trivial bug: write a test to create the bug, fix the bug, prove it by passing test. I haven't gone the whole hog and employed test driven development. I think all types of testing matter most once deployed. You need to prevent your customers 'testing' your app :-) Of course retro fitting tests is often a waste of time as your perspective isn't so clear.

There is always a balancing act between prefect text book engineering and just making some money!

Is the development of automated tests a waste of time for a startup?

Related Stories