Why are software development estimates regularly off by a factor of 2-3?

http://michaelrwolfe.com/2013/10/19/50/

by cviedmai • 13 years ago

311 168 13 years ago

nahname• 13 years ago

Imagine you are going on a hike to the top of a mountain. Countless paths exist between you and your destination. Most of these are dead ends. Every dead end forces you to backtrack, sometimes all the way to the beginning.

Software is a lot like that.

Selfcommit• 13 years ago

Then, on that Hike - Someone decides that the fastest way to the top is to hire 2-3 more guides. But before you can give those guides any meaningful work, you have to take them through all the previous bad paths...

tsotha• 13 years ago

Then whoever was planning the hike decides to go to a different peak when you're halfway up the mountain.

adambenayoun• 13 years ago

99% of the time you'll die before reaching the peak due to lack of water or food.

rubberband• 13 years ago

Your boss has also read about this hot new way to hike up mountains, and it is now mandated that you hike in this fashion.

jl6• 13 years ago

Kept waiting for the punchline where the friend has moved to Las Vegas and a total replan is required.

pc86• 13 years ago

"Oh it's just one state over, I didn't think it mattered. See you tomorrow!"

joshstrange• 13 years ago

That or the part where when he first calls his friend and says "We will be there in 10 days" and his friend says "10 days! really? I was expecting you here in 5 days"

chris_mahan2• 13 years ago

Because management looks at you funny when you say something simple will take 4 months. Then they lie and say that it couldn't possibly take that long. And the reason they lie is because there's no negative to them tying to you. Their job is to get you to go faster, and they are allowed to lie as part of their job, if it makes you go faster. The reality is that it doesn't make people go faster. It makes people leave faster. It makes people give up. Never give an estimate to management. Estimating is their job, as they control the inputs to production. If you give an estimate, they will pressure you to meet it, no matter how unreasonable it later turns out to be. That also makes people want to leave.

jerf• 13 years ago

While I agree with many of the other reasons and believe people in general can't estimate very well, I believe that for businesses this is indeed the fundamental reason. Even in a relatively understanding and engineering-centric environment, when I give what is probably still a bad under-estimate, I will get pressured on how we can reduce that number. I can't even imagine what it would be like in an environment that wasn't relatively understanding and engineering-centric.

If you don't want to hear the truth, you won't. Humans have known that since long before software engineering existed... or a subset of them, at least.

w0rd-driven2• 13 years ago

Management giving estimates? No thanks. I'd rather them take my input and do their job of understanding trends. When I say something takes a long time and it takes longer, I'm not bullshitting. When I think something takes a long time and doesn't, its generally because shortcuts are rarely understood up front. Will I find the same shortest path next time? If I'm doing the same exact work maybe.

I do work in a place where we are a software company but very few people, even some of the devs, understand this culture. I don't say something is complex just for the hell of it. I want to give good estimates but at the end of the day that is all it is, a somewhat educated guess before I've gotten in the weeds. I don't need people that absolutely don't understand this concept to mandate a timeline that is unrealistic. It does absolutely no one any favors. That is why I leave. I don't need to promote unrealistic expectations down the line to customers who think everything is extremely simple. Especially when all we do is create custom software where almost 0% is turn key or off the shelf. Its just insulting to continue that nonsense and the quicker I can facilitate a reality check for all parties, the better and more trust is given to my judgment (and theirs if they actually take the time to learn)

wpietri8• 13 years ago

Management giving estimates? No thanks.

This is actually achievable in a sane way.

I have done it using the XP planning practices. Basically, you make the suit break everything down into relatively small lumps and place them in priority order. Every week, the team completes a few lumps. Before you do them, engineers grade their relative complexity in arbitrary units. (The smallest substantial thing you do is 1 point; something twice as big is 2 points, and so on.) Every week, you count up the points completed. That's your "velocity".

From there, you let managers do all the estimating they want to. If they want the complexity of a unit of work measured, they ask the engineers. If they want to know when X will be done, they look at the team's recent velocity, what's in the queue before X, and do some basic math.

The nice part about this is the mental judo involved. Whenever they want to know when something will be done, it's their problem to trade features against time. Working like this, it's not geeks vs suits; the suits channel their schedule pressure into productive work: grooming the backlog.

chris_mahan2• 13 years ago

If you think about it, though, estimates (time, money, personnel) is really a management function.

You write "I'd rather them take my input and do their job of understanding trends." that's exactly what I'm talking about. They need to take your input, and do their job.

If a worker is told: "Here is what you (personnel) have to do (task), with these tools (environment), and we want you to do it in this amount of time (money)", and the worker fails to complete the work in the amount of time, then it is management's fault. They improperly estimated the amount of time it would take the worker to perform his task.

That management ask the worker how long the task will take means management does not understand the task, and if management does not understand the task, how can they know whether the worker's skill, experience, and knowledge is adequate for the task? Now, they also seem to not understand the worker's skills, experience, and knowledge, so the problem is compounded.

It is a management function to clearly define the tasks that must be performed, attract people whose skills, experience, and knowledge fit the tasks, and fund these people with the appropriate environment, tools, and salaries for them to perform the tasks.

That is the role of management. If they are unable to estimate what tasks must be performed, what people need to be retained, and how much money and time will be spent, they fail at their managerial duties.

That they then blame the workers themselves for improperly estimating the work is doubly wrong, a sign of managerial immaturity.

greenyoda3• 13 years ago

"Never give an estimate to management. Estimating is their job, as they control the inputs to production."

Unfortunately, it gets even worse if developers have no part in making estimates. If management doesn't know the details of your architecture and code-base (or even the other projects you may be working on, if you go high enough up the management chain), they have no way of coming up with a rational estimate on their own.

Where I work, VPs are constantly making "commitments" to customers to have certain features implemented by a certain date, and only notifying developers after the fact. Then they wonder why the development organization is "so bad at meeting deadlines". (Needless to say, I have no ethical requirement to honor commitments that weren't made by me and are frequently impossible to meet anyway.)

ssiddharth• 13 years ago

I admit it's funny but if I showed this to a non-programmer friend, he'd be asking why I didn't research my route better or start on an unknown trail with minimal planning.

avmich4• 13 years ago

Because the research will take a significant part of the project time. Once the thorough research is done - third-party libraries are verified (that they do what's needed, and not too buggy), prototypes are written (to confirm libraries and internal dependencies work as expected), proof-of-concepts are run (to actually see that it's possible for a reasonable effort), time is measured (to confirm the way to go), interfaces are clarified (as part of the research, specifically, in the architecture department), results are estimated (basing on all results above), specifications are formalized (and agreed upon) - there is an awful lot less of what to do.

Philadelphia• 13 years ago

Maybe part of the problem is that development organizations don't tend to charge for pre-estimate planning.

mpg332• 13 years ago

I just don't know why people won't arbitrarily add 2-3 more time on the estimate...

jwpeddle• 13 years ago

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.

flurdy• 13 years ago

True, and Parkinson's Law will ensure it will take at least that time

mpg332• 13 years ago

Yes but wouldn't it at least close the gap to make the time estimate a little more realistic?

dj-wonk3• 13 years ago

According to DJ Wonk's Law (shamelessly self-named), when discussing software estimation, it is inevitable (only a matter of time) that someone will mention Hofstadter's Law.

astrodust• 13 years ago

You need to calibrate your "optimism factor" as most of the time you'll estimate by the wrong amount. I chalk this up to people being optimists, that is, ignoring a lot of the junk involved in completing a task that they'd rather not do: Meetings, documentation, integration issues, testing and so on.

For most people it's around 2-5 but for some as high as 10 to 20.

You might feel like an idiot when you tell people "Hm, change the color of the logo to a different shade of green? Six hours." This is probably how long it will take, though, for that so-called "five minute fix".

ebiester• 13 years ago

It's simple. People don't like the honest answer.

As a developer, I have pressure to keep my estimate down. I can give you a guaranteed date with few problems, but the project is going to end up more time-consuming than what another developer is going to estimate.

The good news is that I will usually significantly beat my estimates under this system. However, I will then be accused of padding my estimates and will be given more aggressive targets, some of which I will miss.

The best way to combat this is to give me a team that has worked together on multiple projects within the domain, and account for changes and ambiguities as part of the process.

damon_c• 13 years ago

http://coding.abel.nu/2012/06/programmer-time-translation-ta...

I occasionally just paste this link with no explaination at the end of emails containing time estimates.

cavilling_elite• 13 years ago

The joke between us engineers at work is to estimate the project time, then multiply by pi.

antonwinter2• 13 years ago

You sir are the inspiration for my new workflow. pi based software dev

lahwf• 13 years ago

Egos.

_wwz42• 13 years ago

So the article tries to describe the hidden complexity of software with a hiking analogy but then it leaves a huge part missing. Let's extend the analogy somewhat... this isn't your first hiking trip. In fact, you've been on dozens and dozens of hiking trips. You're an experienced hiker. Yet, why do you continue to give meaningless as-the-crow-flies estimates of how long it will take you to get to your destination? The reason is that if you miss your estimate, it probably won't matter. Your sherpas will be there at every stop with your food, water, and tent. You have nowhere to be any time soon, so estimating your time just isn't important enough to worry about.

Developer estimates are regularly off because they seldom impact the developer directly. Experienced development managers will pad the hell out of the developers who give them the worst estimates. Most developers will explain all the complexity of what threw their estimate off without acknowledging the huge mistake of not anticipating extra complexity in the first place.

My estimates in my early career were no better than anyone else's, ie way off -- especially for more complex projects. I'd explain what happened to my managers and soldier on. The very next task that came up, I'd give my manager some best-case estimate of how long something would take and the cycle would begin again.

That all changed once I started to do consulting for myself using "not-to-exceed" pricing. The first multi-month project I did killed me. My effective hourly rate went down to sub McDonalds levels and took much longer to deliver than I had expected. After that project, I did a post-mortem on the project to figure out where I went wrong. I came up with several spreadsheet templates and checklists to run through before giving any more estimates.

Mostly I just concerned myself with getting a hell of a lot better at estimating project duration and difficulty. Like most things, when you really pay attention to it and practice it, you get better at it.

icelancer• 13 years ago

Spot on. NTE billing puts the onus on you and gets you actually thinking about it. Helped me a ton.

avmich4• 13 years ago

"Developer estimates are regularly off because they seldom impact the developer directly."

I think I most disagree with this. Developers, in my experience, are quite often asked to leave the company after such estimates.

It's good to have a template - it helps not to forget things - but it is inherent in the work to have unknowns. Large unknowns. A good consultant would probably refuse to work on a project which would require him to use several major technologies which aren't familiar to him, but developer in a company may not have such a luxury as selecting a kind of a project.

crusso5• 13 years ago

Developers, in my experience, are quite often asked to leave the company after such estimates.

I've seen developers let go because they weren't very good at writing software. I've seen developers let go because they were painfully anti-social to the point that they were negatively impacting the rest of the organization. I have never ever seen a developer let go because their estimates were crappy.

jroseattle• 13 years ago

"Quite often"? Really?

Frankly, I would expect such a company that employs those practices sees two things:

1) A very low retention rate among employees, with a lot of churn.

2) A risk-averse organization, heavily skewed to under-promise and just-enough over delivery.

kronholm2• 13 years ago

Been going at it for a couple years now with consulting and freelancing. I've finally gotten good at hitting my estimates. What worked for me was to keep strict time of everything (shoutout to toggl.com, love it!), so I could learn from my mistakes.

The trouble is now though, my competition seems to be underbidding me, but in reality they're providing those ~33% estimates they will never realistically keep, while I'm at ~100% estimates. Not really sure how to relay that to clients. One of the many reasons people like me need a salesperson in front, I guess.

avmich4• 13 years ago

If you're really good at estimates, try to play that you can keep not only the final delivery, but also the partial ones. Something like "you can be sure I'll get 100% done because you can check when 10% is going to be done". Sure it's not a guaranteed sale - and it's harder to keep both partial estimates and totals, comparing to only totals - but it's still something.

crusso5• 13 years ago

What worked for me was to keep strict time of everything (shoutout to toggl.com, love it!), so I could learn from my mistakes

Right, that was part of my method as well. I learned to keep really good notes of each part of the project. At the end of each project, I compared my "what I thought it would take" with "what it actually took". In subsequent projects, I tried to match up similar complexity items with "what it actually took" notes to remind myself of the pain.

RyanZAG• 13 years ago

You're probably losing more business than you think - customers have no idea who is accurate or who is just faster, etc. They will generally go with the lowest quote regardless of other factors unless they've been burned before multiple times. Most people haven't.

I don't have a solution here though other than noticing that underbidding and then getting skilled at convincing clients to do paid extensions later actually appears to make the most money at the cost of your ethics. I'd avoid that approach, but it does seem to work for a lot of companies.

My personal approach is to quote for very bare projects with only the bare essentials (eg, poor UI design, minimum possible feature for the client to see what they're asking for, etc). This can usually be done a lot cheaper than most people think as 90% of the work is in the last 20% of the features. Then once the client has something, you can give them a quote to touch up the parts they need. Basically you split the project up into many small projects each with their own quote which helps you to estimate tasks as they appear and helps your client to minimize costs by leaving off features that are more expensive than they initially appear.

jdlshore• 13 years ago

It's common in the fixed-bid world to deliberately underbid and charge exorbitant rates for the inevitable change orders. There's an opportunity to compete there, but maybe not an easy one.

wildgift3• 13 years ago

Good luck. I found myself in a similar situation with pretty good estimates, but unable to bid low. I'd be bidding double or triple the competition. One time, I was bidding at triple, and the previous dev had failed totally. They went with another dev who bid nearly the same price. He failed, too. By that time, I was like, I'm not even going to bother - they don't want to pay what it takes, and buy failure over and over.

thirdtruck2• 13 years ago

Have you shared these spreadsheets and lists online? They sound like something that could turn into an excellent, three-figure or more information project.

ataylor32• 13 years ago

I, too, would be interested in seeing these. Here are some guidelines from somebody else that I found helpful, though:

http://www.missionfamilybank.org/software-estimating/

pjungwir2• 13 years ago

Yep, I agree estimates are a lot more important when you work for yourself. I've had at least one client comment on how pleased he was that my estimates were usually accurate. I know another developer was giving him answers like, "That will take 5 minutes!", so perhaps his expectations were low. :-)

On the other hand, giving realistic estimates I often hear clients say, "I asked around, and other folks say this should take half what you're quoting." Usually this happens before the work, but I once had it happen in the middle of a long-term engagement after I delivered a two-week invoice, and they were disappointed I hadn't accomplished more. So if I'm not careful I can lose my clients' trust, which is my highest asset. I've gotten pretty good at explaining why a job isn't 5 minutes (or 1 day, or whatever), but I might save this article as something to share if a client doesn't have much experience doing software projects.

wpietri8• 13 years ago

I'm glad you got better at it, but I don't think software estimates can ever be very good. Nor should they be.

That a project is estimatable has to mean that novelty is low and predictability is high. But low novelty is a sign of duplication, which should be factored out into things like libraries, frameworks, and self-service tools. High predictability is either a sign of duplication or it's a sign that people have basically agreed not to learn anything during a project. If you are shipping early and often and study users and metrics as you go, then you will always learn things that affect the plan. Which means that the schedule isn't predictable even in theory.

crusso5• 13 years ago

First of all, don't overly aggrandize what we do. 99% of it is derivative in most ways. Although I've enjoyed doing the work on many of my projects, I'm not under any illusions that what I was doing was ground-breaking.

Besides, most of the things you mention can be accounted for. If part of your development process involves iterating through the design with the customer every couple of weeks then you build that into your estimates.

Likewise, I always build documentation and code handover support into any serious project rather than acting surprised that customers will want such a thing and expect it to be part of what I'm delivering.

knappador• 13 years ago

I do flat-rate projects like this using a library that I have to fill in gaps myself. Even in the context of very high unpredictability, I have to make good predictions else suffer low compensation exactly like the parent poster did.

I like crow-flies vs hiking analogy. I'd like to add upon it for so-called "green field" programming as it relates to getting cost predictions correct and understanding programmer time growth in relation to program size. Some parts are linear. Some parts are exponential.

Before I go on talking about "hello-world" space, note that I do architecture and really like architecture, big-picture solutions etc but I try only to solve what is apparent and then iterate on it whenever I notice two pieces tangling. There are excellent wins to be made here, but a working program itself constitutes part of the information necessary to arrive at the final architectural decisions. This is why even on FOSS code, I try not to sweat the architecture on the first pass -- when I do, dead code results. It calls out to me that I have solved something not relevant to the emergent implementation.

Once I've decided what the API might look like, what data structures have to be accessible to what, and what the minimum program states are, I try to stay in what I call "hello world" space. The idea is that the code I'm writing is never doing anything more than one problem at a time, line by line if I don't know exactly what a line does.

This is incredibly efficient because I deliberately break my problem into something testable and well understandable at every step. However, it's impossible to write production code this way. Not only is test feedback, (printlining and frequently more complex testing) not part of the final behavior, but more importantly, the elimination of concurrence of many states, the overlapping of problem workflows in execution sequence, implicitly says that "hello world" space problems are not production code. They have eliminated some functionality or consideration so as to make "hello world" solutions utterly oblivious to concurrent states or processes.

However, two "hello world" problems that need some coordination themselves creates a third "hello world" problem to implement their coordination. The third problem isn't apparent until the first two are explicitly solved. This implies hiking from the article.

Solving "hello world" problems brings more of the problem domain into "hello world" space. The concurrence problem itself will become apparent as a "hello world" problem when it has manifested itself. If there are two states that have been independently implemented but can overlap in execution, there is at least one set of logical statements for dealing with the maximum concurrence of the two states. Growth for N states is exactly an NxN truth table unless some of the states are sparse. In the worst-case, for N "hello world" solutions, one layer of abstraction where all states depend on all other states will result in N^2 logical blocks. While in practice this concurrence is usually much lower, the growth in abstraction is exponential if one is to keep breaking each problem down into an independently writable, testable piece.

So my conclusion is that growth is pretty much linear in proportion to writing "hello world" problems and exponentially proportional how many layers of integration are necessary. Slamming code down ichi-geki style is linear for problems that don't have a potential for overlap and geometric for ones that do. Code that eliminates duplication of routines through logic creates them in implementation, but the payoff is a net gain in many ways, so of course it still makes sense to do it.

As for making predictions, while running my internal monte-carlo, the concurrence of states has a higher influence on the outcome sometimes than the raw code size I expect to come out in the end. Linear and exponential growth occur in different areas. Recognizing that each integration incurs smaller linear component, and that each abstraction layer involves N integrations of solutions, I think it's pretty achievable to have a damned good estimate. The exception is if scope -must- creep and this is only determined in the middle of a project. Scope creep gets paid. Bad estimations do not. Usually scope creep is itself revealed when a new facet of the problem comes up, one which represents a competitive opportunity, which the client is happy to pay for.

Saying that no estimate should ever be good is like saying programmers should make gigantic mistakes. This is only possible when working on W-2, and only when the company won't die. End of discussion. Independents have to get it right. Any thoughts to the contrary will soon land an independent programmer back in W-2. The fact is that clients like predictable cost, and programmers who can deliver it can ask for more because they don't have to start squirming around in client meetings later and making justifications that are inevitably open to self-serving padding and other temptations. Be brave and keep scope, or go W-2.

code_duck• 13 years ago

The hiking analogy works pretty well, extended to traveling in general.

I spent a couple of years traveling around to visit friends and family all over the US, and regularly my estimates were off in when I'd arrive somewhere. My software delivery estimates are similarly not perfect, in the later direction like most people. My conclusion was that my estimation of how long it takes to do anything is off a bit. For me, this is because I'm trying to say I'll do something sooner to meet the perceived needs and desires of others.

brudgers• 13 years ago

What makes the hiking analogy perfect is that people are so accepting of it as an analogy. My first thought: Why the fuck are you hiking to Los Angeles?

Plane, train, or for adventure, Greyhound. A sailboat makes more sense. A bicycle makes more sense. And if you insist on walking, at least follow a fucking road instead of cutting cross country.

If you decide the first step to driving to LA is mining iron ore to smelt for casting an engine block and building a Bessemer furnace for the steel for hand forged pistons, then there's lots of shit that can go wrong, EPA permits and zoning laws, even if you've done done it before.

The reason software production estimates are bad is because they aren't taken seriously. The guild of programmers can say, "we shall get from A to B by cross country hike."

Yes, the analogy is perfect because its absurdity is acceptable.

loceng• 13 years ago

What are you doing now?

sbov• 13 years ago

Part of the problem is good estimates take time. For many things you can't give a good estimate after 10 minutes of analysis much less the 10 seconds some people ask for.

Because of this I offer two estimates: one that might be off by a factor of 100x, and those that are fairly accurate but take more time to make. My boss usually goes for the former.

Edit: also there's two types of estimates. One measures how much of my time it will take. The other measures the date it will be done and/or live. The two are oftentimes different numbers, and as a developer I can't always control the 2nd.

gcv• 13 years ago

It's more like this: you've been on dozens and dozens of hiking trips. Except these trips generally go through seismically unstable territory and war zones. You just never know when you'll have to dig a tunnel underneath a battlefield to get to the next river you'd have to cross — a river unmarked on your map, or perhaps the result of some aggressive "engineering" by someone since your map was drawn.

Then you get an urgent message from your customer: you weren't supposed to hike from SF to LA after all. You actually need to go from New York to London, on a submarine. Specs require that you do this without surfacing or refueling. So you need to design a nuclear reactor, figure out a way to put it on the leaky old U-Boat your customer just happens to have lying around. All within the same timeline as before, and in the same budget.

You've had all this happen to you a few dozen times, so you factor in the time you know you need to build submarines (all just different enough, of course, that you can't reuse parts, even if the customers let you adapt, say, the propeller shafts to a new customer's design: I hear that in software, this is called "DRY" and "using libraries and frameworks" and "refactoring") — and you give an honest estimate: "it'll take eight months of full-time work, and will cost you about X hundred thousand dollars (a bargain for a nuclear submarine, if you ask me)" — and the customer will then go to someone else, leaving you wondering how you'll pay your bills.

_wwz42• 13 years ago

Meh, you're just contriving extreme circumstances for your analogy so that the software developer is both a super hero and has no blame in being late with deliverables.

There are simple methods to combat gross inaccuracies in the client specification. You document the requirements for what you'll be delivering in a carefully worded Statement of Work that the client signs off on, you ask to be paid in chunks as you go with a portion up front, and you implement a system of change control so that it's totally clear if the customer is sending you off in a completely new direction. At that point, you book what you've worked at your hourly rate and you submit new work and time estimates for the changes.

rquantz• 13 years ago

Would you be willing to share these checklists you've created? I would be curious to see them.

Iftheshoefits• 13 years ago

Let me fix that for you:

Developer estimates are regularly off because "experienced development managers" are all too eager and willing to allow organizational politics and external influences to add complexity in ways the developers cannot anticipate. Inexperienced developers' estimates are even further off because of their relative lack of experience.

I assure you, the first time a "development manager" tries to scapegoat his subordinates for his own failures to deliver to these external influences his subordinates realize that their estimates impact them eventually, and directly.

coldtea2• 13 years ago

So the article tries to describe the hidden complexity of software with a hiking analogy but then it leaves a huge part missing. Let's extend the analogy somewhat... this isn't your first hiking trip. In fact, you've been on dozens and dozens of hiking trips. You're an experienced hiker. Yet, why do you continue to give meaningless as-the-crow-flies estimates of how long it will take you to get to your destination? The reason is that if you miss your estimate, it probably won't matter.

Nice try, but no.

Developers who are DIRECTLY affected from missing estimates (e.g one man shops paying out of their own pockets, people building an app while having quit work, eating ramen and maxing their credit cards etc, developers in companies who will personally pay for any delays by attrocious, unpaid overtime crunch marches, etc) underestimate the time needed to get something done just the same. Just ask Allan Odgaard (of TextMate 2 infamy).

nnq• 13 years ago

100% agree on that. I did a lot of bad estimates and was directly affected by them (fixed rate projects that ended up driving my rate to "below mcdonalds worker" and some even with "lateness discounts" because I just wanted to keep the customer happy), and it took me a lot of time to get better at estimating even if I was paying from my own pocket for all the bad estimates.

Now if I am to hire a developer I will never think that "his estimates are bad because he's not directly affected by missing them"!

natmaster• 13 years ago

If you've been hiking on similar trails so much, why haven't you built a land rover or rail system that gets you there every time?

You're like the expert sherpa trying to convince people to use your message delivery service when the internet exists.

wildgift3• 13 years ago

The Sherpa are an ethnic group, not a job.

anupshinde• 13 years ago

Okay, I agreed (had no other choice) with management to not have less than 20% variance in my estimates.(read: negotiated-and-reduced estimates) as a part of our performance metrics.

dreamfactory• 13 years ago

Reduce your batch sizes to reduce uncertainty.

rdtsc• 13 years ago

Good points in the article.

You can think of it this way. A coast line is a fractal line. As you zoom in its length gets bigger and bigger seemingly. From the top level on the map it looks fairly straight.

Related to estimates. Programmers have and operate with idealized mental models. A FIFO queue works kind of like this ("/closes eyes and sees a line of people lining up at the store"), a tree traversal looks like this ("/closes eyes sees a red cursor move down a tree drawn with blue nodes on an white board"). Same with estimates. How long will this take? ("/closes eyes and imagines an idealized sequence of events that will lead from now till project completion -- use this library, develop that API, test a little bit here, done... 2 weeks").

The problem is, zooming in is hard. "Oh this library we are about to use actually doesn't implement these corner cases". So now we are spending 3 days patching it. Oh, while working on this project, a critical support ticket comes from a customer. And then testing reveals a fundamental flaw in our API design. Re-write a huge chunk and repeat.

Some developers and managers just use heuristics. They do the mental estimate then x2 or x3.

^ All the above is true if the developer is everyone is honest and trusts each other. As other posts wrote, that is not always the case.

Then it is a more complicated game. The main question to ask then is "What is the punishment for overestimating vs the punishment for underestimating?".

Have people been fired for underestimating? No, ok underestimate. Have people been fired for missing deadlines? Ok then overestimate.

You can even play this game with yourself. What kind of deadlines do you set for yourself and how to you handle setting a too short or too long of a deadline? Do you get better with time at setting deadlines? If no, why not?

mentos• 13 years ago

I think the fractal analogy nails it.

Haven't thought about it before but I realize I too use a heuristic of x2 or x3 when estimating my time. Nothing would stop me from getting a pen and paper out and stepping through the 'trail' in my head in pseudocode beforehand, but for most of my projects I'd benefit from just writing the real code from the start and apply the time I saved to the project.

pjungwir2• 13 years ago

I loved the article, but I was hoping we'd get a reference to the Coastline Paradox[1]. Basically, since a coast is a fractal, its distance is infinite. The "zoom" that caused the very first delay could keep going, and going, and going.

Anyway, nice article!

[1] http://en.wikipedia.org/wiki/Coastline_paradox

sz4kerto• 13 years ago

Because software developers, like most people, don't like conflicts.

mkramlich2• 13 years ago

also because we don't have Time Machines

swanson• 13 years ago

Most people would agree that practice makes perfect, right? I think a big issue with software estimation is that it is hard to get practice. I'd done full project estimates for 4-6 projects in 3 years of working professionally. Compared to the amount of practice I have writing code, this is nothing. How can I ever hope to get good at something if I only do it once every 3-6 months and it might take 2 years to get feedback on if my final number was close to the actual cost? And unlike other skills, I can't practice on my own or take a course (I've never seen an open source project that did round-trip estimates, maybe that's something to try out...)

Another issue is when you have an "indirect estimate" - basically I will estimate the work, but someone else is the one that ends up doing the project. If you aren't careful to consider who will be doing the work, you might estimate too low (if you are an expert and the work is done by a bunch of new hires).

And none of this even touches on misinterpreting client demands, scope creep, dev team turn-over, or often neglected timesinks like documentation and meetings.

pitchups• 13 years ago

This post was originally an answer on Quora to the question posed in the title [1]. The question has so far received 222 answers - some of them arguably better, though not as popular as this one.

[1] http://www.quora.com/Engineering-Management/Why-are-software...

mikegioia• 13 years ago

This blog article is more popular than the quora post because this blog article doesn't require people to sign up to read it.

greenyoda3• 13 years ago

There was a trick posted recently to HN for reading Quora articles without signing up:

Tell HN: Add ?share=1 to Quora URLs to display content without login

https://news.ycombinator.com/item?id=6300856

ChuckMcM• 13 years ago

I enjoyed reading this the first time around. I purposely illustrates the complexity of software and perhaps inadvertently illustrates the challenge of getting schedules from inexperienced teams. In rebuttal, had they walked from San Francisco to LA along any freeway they would have been done much closer to their predicted date. Which they would have known had they been experienced in walking back and forth to LA from the Bay Area.

The "big" message is that if you have never done something before, you have no idea how long it will take, even if you can describe what you are going to do pretty easily. Returning to the hike metaphor, this is why you need to listen to the cranky old guy saying "walk along the roads, not the beach" instead of the inexperienced folks saying "Hey its just walking, and walking on the beach rocks! Walking along a road is boring and lame!"

reticulated2• 13 years ago

The single biggest mistake in the example is giving an estimate for the whole project without knowing team velocity. He tried to plan everything up-front and committed to a completion date, strongly setting delivery expectations.

An Agile approach to this walk would have managed the expected target date much better and improved team morale by not overstretching each day ("sprint").

Lots of people think they know (and implement) Agile. I did, until a course I took about a year ago where I learnt skipping seemingly unimportant bits (measuring velocity, sprint retrospectives, etc.) bring the whole process crashing down.

cs02rm02• 13 years ago

The single biggest mistake in the example is giving an estimate for the whole project without knowing team velocity.

I'm inclined to agree. Often though a client will want an estimate (to a ridiculous accuracy) before agreeing to the work. Unless you churn out web page templates or something equally repetitive it's hard to have that relevant velocity.

I still come up against client managers that don't "believe" in Agile approaches too.

reticulated2• 13 years ago

Oh we still get regular "lively discussions" about exactly what features are going to be in a product with a release date 6 months away. I can completely understand though; it's very difficult to leave a conversation being told "we don't know yet, it's likely to be X & Y (but we're not 100% sure) and possibly Z too".

wpietri8• 13 years ago

I love this post; it conveys the feel of the experience so well.

One of my big aha moments about estimation was a bit in McConnell's Rapid Development. He pointed out that most estimates get made with executives pressuring for short numbers. When you iterate a few times with that, you end up with the smallest number that developers can't absolutely prove is impossible.

If you draw out a bell curve of probable completion dates, this is basically the same as picking one far to the left, so you have a 95-99% chance of being late. But somehow, executives are still surprised when their totally biased approach yields the numbers they liked, not the numbers they needed. All the estimation effort was pure waste; our time would have been just as well spent making cotton-candy raincoats.

I now have done a few projects with little to no estimation, and it has gone much better. I have a hard time now seeing why we ever bothered.

thinkstoomuch• 13 years ago

I wholly agree with this, and feel that all project plans should be constructed to mitigate the very high risk of initial estimates.

Too many project processes treat estimates as commitments with no backup plan when they (inevitably) go awry. I think a better approach would be to design your system around the idea that all estimates are suspect until proven otherwise, which of course leads to iterative approaches etc.

However, you can't run a business without any idea when anything will ship. Estimation is still, in my view, a very necessary step, so long as there is significant incentive to get it right rather than get it fast, AND the business is receptive to a little ambiguity through the project life cycle. This is a very difficult balance to strike.

In my experience you can have certainty in project scope or time, but not both.

wpietri8• 13 years ago

Your point about estimate risk is excellent.

I agree you can't run a business without any idea when anything will ship. But to get a good idea when things will ship, you don't need estimates.

For example, in the Lean world there's something I've heard called "Disneyland scheduling". You break the project down into lumps. You maintain a queue of the lumps. You measure the average time from position X in the queue to being released.

Now you have a reasonable early warning system for when things will ship. Which is generally much better than what people have with most projects, which is a fantasy date.

ScottBurson• 13 years ago

I once read an argument that went something like this: if you're actually going to go to the trouble to build something, its expected return to the business should be so large that how long it takes (within reason) shouldn't matter. Conversely, if a project is only economic to undertake if it can be done within a certain amount of time, it shouldn't be done at all.

I find that I often don't have a good feel for how long something is going to take until I'm about 1/4 the way through the project. If I absolutely had to produce accurate, high-confidence estimates, I would just do 1/4 of the work first, then estimate the rest.

As to why estimating is so hard, here's how I like to think about it. Tasks form a tree: each project has a number of steps, each of which itself breaks down into sub-steps, and so on. We tend to estimate by thinking about the tasks near the root of the tree; but the actual time required is proportional to the number of leaves -- tasks we can implement without further breakdown. Without planning the entire project out in detail it's hard to know what that number is. But to plan to that level of detail requires making many design decisions; and researching and making those decisions is much of the work of the project.

The bottom line is, good estimates are expensive to produce, and my experience aligns with yours: it's rarely worth the trouble.

wpietri8• 13 years ago

Agreed. Certainly, estimates of cost shouldn't be more precise than estimates of value, because ROI is the number that matters. And business is a black enough art that estimates of value are very rarely more precise than a factor of 4x.

What I think people try to get from estimates is a sense of control. But I think there are better ways of giving people not just a sense of control, but actual control.

triplesec• 13 years ago

This of course does beg the question as to how you know when you're 1/4 of the way though the project!

mkramlich2• 13 years ago

why? focus very closely on the following word:

estimate

logicallee• 13 years ago

Forgive my username, there is nothing logical about this.

It's because estimates are really just a test of your imagination and skill. They're a "different game" from delivering something. Here are some examples:

How long does it take you to add 1 + 1 together? The only correct amount of time is "0 - no amount of time." But clearly that's not true, is it?

How long would it take you to multiply 5 by 7? Well, zero. No amount of time - you know it 'by heart'. It doesn't take you any amount of time to multiply 5 by seven. Well, clearly, that's not true is it?

How long would it take you to test whether 31337 is prime or not, by any means? Like, 1 second because Wolfram Alpha can tell me I bet. But clearly it wouldn't take me ONE second, would it?

How long would it take you to write a function to test whether an integer less than 2^32 is prime or not, without having to be super-efficient about it? Well like 10 seconds. But, clearly that's not true, is it?

And so on. How long does it take to put a twitter bootstrap page up? Like, 5 minutes.

How long does it take to...

You should be able to build an app that doesn't crash and does what you're trying to code in like a day.

You should be able to build something that is software you can launch with in like a week, as an MVP.

You should be able to replace someone else's mature product with a reimplementation in like a month.

You should be able to build a scalable business that is ready for a series A in like a year.

These things are not meant to be accurate, so much as show that you have imagination.

jconley• 13 years ago

This McConnell book[1] is a must read on the subject. Generally developers just fail at estimating at a fine grained enough level and forget about all the ancillary tasks that take up a majority of a project's time.

[1] http://www.amazon.com/gp/aw/d/0735605351

pjmorris• 13 years ago

I just had an estimate/actual conversation that went like this: I think it will take me X time. Ok, but I want it done in .5 X time. Ok, I'll try to get it done in .5 X time. I finish it in X time.

Is it a missed estimate? That depends on who answers.

alextingle2• 13 years ago

Ok, I'll try to get it done in .5 X time.

Never, ever say this.

Personally, I always give my estimates as a range. If you break down your project into small tasks, and estimate each task as an optimistic/pessimistic range, then you can get a pretty good idea of the expected completion date, along with the deviations from that which you might reasonably expect.

However good your estimate, committing to ONE number is madness - it hides the complexity of the situation, making it impossible for you or your manager to see the real picture.

clord• 13 years ago

I try to do my estimates (team size, man-days) in Big-O notation. Haven't nailed how to do that yet, but it helps people understand that estimates are non-linear functions.

dlitz• 13 years ago

The problem is that people try to eliminate uncertainty instead of trying to manage it effectively. If you're trying to make estimates with more than order-of-magnitude precision, then you're probably doing something wrong.

If we drove cars like we develop software, we would pre-calculate a list of instructions, set a timer, and then execute those instructions without regard to what's actually happening on the road. With practice, you could probably get better at making those lists of instructions, but why bother? The general approach is just ridiculous.

Decades of experience have shown that there is enough inherent uncertainty in software development that trying to plan everything from the start just doesn't have a high success rate. Even if you know everything that has to be done today to get the job done, your environment will change, and your stakeholders will change their minds. What good is software that perfectly solves the problem you had two years ago?

Once you dig through all of the buzzwords, "Agile software development" is simply about applying closed-loop control theory to software development: You structure your project to provide high visibility and to allow frequent changes in direction, then you iterate and make adjustments until the result is satisfactory.

Everything else generates unnecessary risks that somebody ends up paying for.

DigitalJack• 13 years ago

http://en.m.wikipedia.org/wiki/Planning_fallacy

tmoertel2• 13 years ago

Software-development estimates are regularly off by a large margin for two reasons. First, the problem is inherently hard. Second, social and business pressures bias the estimates downward.

Why is the problem inherently hard? To see why, let's imagine that the All-Knowing Fairy Godmother of Software Estimation descends from the heavens and lends to us her magic estimating function F that, applied to any software project S, will tell us exactly how much time and money our preferred software team will consume to implement S. Don't worry about how F works, just believe that it does. (Okay, okay. Let's just say that F peers into a parallel universe in which our team has already implemented a software system line-for-line identical to S, and it just observes how much time and money were actually consumed in that universe. Anyway...)

Now that we have the magic F, our problem is easy, right? Nope. The real problem is that we don't know what S is. All we actually know is that our S, whatever it ends up being, must satisfy some set of fuzzy constraints C (commonly called "software requirements").

In truth, there are a countless number of possible values for S. In other words, S is a random variable, a mapping that ascribes a probability to each of the possible software systems that we could build to satisfy C. Therefore, our best estimate, even with the perfect estimating function F, is itself a random variable. In other words, it's not an estimate but a distribution of estimates.

See how fun this is getting?

But, wait, it gets funner. That's because people in the business world don't want a distribution of possibilities. They want a budget, a date on the calendar. So we must squeeze point estimates out of the true distribution. (And, remember, we don't even know what the true distribution is!)

That's where the second reason kicks in. Let's just think about the distribution of possible budgets for our distribution of possible values of S. Because S can range from "the simplest thing that could possibly satisfy C" to "the most insanely complex thing that a frighteningly gifted salesperson for an enterprise consulting firm could get our CIO to throw money at," that distribution is going to be w-i-d-e. From X to 10X wide.

So if we're the folks tasked with coming up with those point estimates, we could plausibly estimate X on the low end, or 10X on the high end, or anything in between. Guess which estimates are going to get us the most push-back from the higher-ups?

Now, we're swell guys and all, and we want to do a good job with our estimates. No question. But, still... We're human. If we think that giving a higher-end estimate is going to make us unpopular with the higher-ups, maybe we'll estimate a little lower. And when we get push-back on that estimate, maybe we'll "take another look at the numbers" to see if we "missed any opportunities."

And that's the old one-two. We start with a hard, fuzzy problem and then add to it the pressure to deliver only feel-good solutions. The result: estimates that are often way low.

wfunction• 13 years ago

For what it's worth, I haven't seen hardware engineers complaining about this issue as much as software engineers, which I find baffling. I think if you're going to justify why software development is such a "hard" problem, you should mention what makes it so different from other fields.

kwantam• 13 years ago

You've been talking to the wrong hardware engineers, I think.

Every engineer on my team has a characteristic "fudge factor" that represents his... optimism (say, from 1.5x to 2.5x). Once I apply those fudge factors, I add another 30% slack to the total schedule. And I usually still underestimate.

The team I'm on now has been doing a very good job compared to previous ones about getting good estimates for project duration, but the ability to underestimate is nonlinear in the project size, and most of our projects recently have been pretty small chips.

On our most recent tapeout, we did a new baseline based on a new feature request and underestimated the cost of that feature addition by a factor of 2. The total schedule was pushed out about 20% as a result. I still consider this a success in predicting the project's duration: I've been on projects where we taped out six months or more after our "commit" date.

pratik661• 13 years ago

Its much easier to justify changes in software requirements vs changes in hardware requirements because the perceived cost of adding a few more features on your application is low. Compare that to say.. adding another register to your chip..which might add millions to the whole production process. The changes can be made while the app is live. You can not do that for hardware.

gnarbarian2• 13 years ago

Hardware engineers have more clearly defined requirements, It's also well understood by everyone that you can't just change the design halfway through production with no additional cost. Hardware engineers clients are mostly other engineers who know their shit very well. Hardware engineering is also more formalized with a larger percentage of engineers actually possessing relevant degrees with common well established means of communicating and thoroughly clear phases of development for a project.

People hiring consultants to build them software are generally non-technical. On most projects the following problems may occur:

Problems With Clients:

1) They don't konw what they really want.

2) They are bad at communicating what they really want.

3) Not enough people from the organization are brought in to weigh in on their needs / actual duties.

4) Too many people are brought in, many with conflicting interests. (bike shedding). [1]

5) Critical omissions are made regarding idiosyncrasies of their infrastructure that are never brought to light until far too late in the project.

6) people trying to change the direction of a project late into development

7) complicated/redundant policies or procedures which are incompatible or inefficient to implement using the tools the client requires you to implement them in.

Problems with Consultants / Contractors:

1) Over confident estimates

2) They are bad at estimations / unfamiliar with the tools required to do the job

3) They are bad at communication / rooting out the the real problem early

4) Unable to detect confusion/bullshitting from a client or are unwilling to investigate it.

5) PMs who make promises they are unable to keep

6) PMs who do not understand the details and make decisions without consulting those who do

7) Developers who avoid asking the client for clarification

8) spineless developers/PMs who screw themselves to avoid confrontation

9) Procrastination

[1] http://en.wikipedia.org/wiki/Parkinson's_law_of_triviality

tmoertel2• 13 years ago

Here's the thing. Software is profitable at much lower quality levels than the hardware market will tolerate. As a result, most hardware is designed through a process that could accurately be called engineering, even though most software is not. So hardware engineers get to make estimates with the benefit of having fairly reliable engineering outputs to use as estimating inputs. Further, the business side of hardware is more culturally attuned to the realities of engineering and is therefore less likely to challenge estimates coming out of mature engineering processes.

In short, on both sides of the old one-two punch, the hardware guys have got advantages.

pcurve• 13 years ago

In hardware design, there is more up-front time making prototypes and running simulations, so that engineers have clearer ideas about what they will build and how they will solve problems.

We don't do the same with software, though I would argue that we should.

Based on my experience, the more time you spend coming up with application prototypes that closley resemble final products, more accurate estimates you will get from engineers.

venomsnake• 13 years ago

I have been in a company where we developed hardware and software. Well hardware - you have some hard stuff there - at the end the laws of physics give you a lower bound of stupidity that you can deal with. There are just times when you can write in reply to the manager with a cc to the CEO when answering the question - can we do it that way - with "Only if you send us to training in Hogwarts"

Because software is magic for a lot of people anyway that excuse just doesn't work.

michaelt• 13 years ago

Engineer with hardware and software experience here.

Software is held to much lower engineering standards because it can be made much more complicated, by most measures of complexity; and complexity in hardware drives up development and per-unit costs, whereas complexity in software drives up upfront costs only, so most people push complexity into software.

Consider an all-mechanical watch. Even a watch that just accurately displays time and day of week is going to be fairly complicated. If you want the day of month to account for the length of different months, that's more complicated again, and if you want to account for leap years, especially the mod-100 years that's more complicated again. To say nothing of products like the Calibre 89 [1].

A product like Pebble will do all that without batting an eyelid, and a hundred other things that would be impossible to do mechanically. Setting the time automatically. Programmable watch faces. Scheduling meetings and synchronizing with other watches, phones and computers. Correcting across time zones and daylight savings transitions. Accessing real time train schedules and weather.

[1] https://en.wikipedia.org/wiki/Calibre_89

bitcuration• 13 years ago

The challenge of software estimation is rooted in software engineering. As much as software wants to be called engineering, its current state of art is still craftsmanship, more of development than engineering.

Hardware engineering is possible as physics is the boundary and since has become categorically managed by human's mind of structure and logic. A jumbo jet can be decomposed to millions parts each confined by own specification therefore once design is done the production timeline is easy to estimated.

Software production to this day is still struggling between modular development and any attempt to standardize on a way to create software is overwhelmed by thousands of new frameworks or libraries even engineering methodologies every day. The reason software has such "joy" is because not like material or aerodynamic physics, software and way of making software are frequently derailed by its foundation, computer chips new advancement. Literally, every few years whoever can exploit the new capabilities of computer hardware has a chance to become next Steven Jobs.

Granted, most of corporate enterprise software projects don't have to aim high and certainly can be managed like any consumer production manufacture, so long as people are willing to settle on certain software framework, engineering approach, and treat pre-existed software component like real "physical" component meaning quit thinking the possibility of customize anything, simply focus on assembling. This is the engineering propaganda from school of IBM, Microsoft and SAP et al have been trying for decades but reality is no one but the labor cost sensitive offshore software firm would be interested to this. Everyone in corporate world pretend they're working in a creative campus and fancy about the software they make is the core competitive edge of their employer. The price to pay is uncertainty, but the question to ask is if their employer is in the business is in software business. The hard part is even this question is full of uncertainty, who'd imagine Walmart will be compete on the software it runs based on its past business model, well that might change.

The art to be able to estimate accurately (heck, even Boeing failed 787 estimate) is to turn craftsmanship to production assembling line. Stop thinking creatively, focus on manage a collection of proven engineering approaches of "manufacture" features meeting requirement. Everyone in software industry should realize there needs a distinct separation between the creative making of software world and the basic business operation needs. The line is not static though, as Google's engineers constantly made IBM's scalable/reliable enterprise software suite a joke. Still, not everyone is google and effectively manage the evolution of this distinction is not easy but necessary, because while google is looking for uncertainty, most of corporate enterprise hate uncertainty.

Once accepted the confined world, it will be possible to maintain a model of software composition by different combination of modular components, pre-existed or further decompose. The process can be fully automated and emulated as long as the time factor is added to each component. The upside is this can be estimate/managed down to minute level yet the downside is that the software created would be boring lack lust, but built with complexity to support operation reliably which is enterprise software all about anyway. Most importantly, once the estimate and cost become predictable, managing software system evolution will become a much easier and lucrative job. Imaging how joyful will it become if you go to your CFO asking for next round of upgrade (or even revamp) due to the "demand" of some fancy new capability the business users have experienced from their personal consumer digital gadget, while your commitment to the money and time has a solid track record.

The key to resolution to the challenge of corporate software engineering, including the majority of creative product builder in start ups, is the lack of an effective and efficient way to manage the software evolution and the chaos it brought. Sourcing to less cost labor or adding more bodies to the equation will not make it better, except short term psychological benefit.

In sum, software industry desperately needs automation. Ironically, software starts as the automation king to everything else, but itself has been largely remained in the hand of artist. The good news is cloud and DevOp are closing fast to bring an end to the nightmare of software engineering, at least an interim until the day software is no longer made by human.

sixothree• 13 years ago

And how often do hardware engineers have to deal with business logic and workflows.

dj-wonk3• 13 years ago

"First, the problem is inherently hard." If that was all it was, the estimates would still, on average, be correct. Your second point is really the key: others pressure create an incorrect bias. In an ideal world, there would be more conservative estimates coupled with pleasant surprises when delivery is ahead of schedule.

anigbrowl• 13 years ago

I don't think the problem is just top-down (though I agree with all of your factors identified above). On a few occasions managing projects, I've gotten an estimate and said 'OK, I'm going to assume it ends up taking us twice as long due to stuff we haven't discovered yet...' only to have this regarded as a personal insult.

marvin• 13 years ago

Great post. This sums up many of the problems the consulting industry have to (vainly) try to manage.

nollidge• 13 years ago

In my experience, the #1 problem by miles is that estimates are made far too early in the process, before enough detail is available to make an accurate judgment.

And then those crappy estimates are never revisited once more information becomes available.

divide_by_zero• 13 years ago

The other thing to note - time to completion often varies from the original estimate, when you finally get to figure out what you are actually building. But it doesn't vary equally, in terms of over-spend and under-spend.

You might end up taking twice as long. You might end up taking half the time estimated. Even with a perfectly honest estimate (which is rare) both scenarios are equally likely. So the overall distribution skews towards late delivery - you are as likely to require 2 more days, as you are to save 1.

csbowe• 13 years ago

For me, a lot of underestimation is about motivation. I work best with looming deadlines. I feel like I will be motivated to work harder with a deadline that's less realistic, even if that' not always true.

forgottenpaswrd• 13 years ago

I really love this article.

In practice a good strategy for companies is reduce unknown to a minimum, the Steve Jobs strategy, just focusing on very few products, with each having the less complexity possible(simplify and simplify).

Only focusing on very few products you could make perfection possible, estimates will be off, they always are when there is unknown, but at least completion will be finite (you will complete it!, which is number one issue with research work for example).

Sounds simple, but is really hard, leaving creative people on their own is dangerous for this reason, they could start new projects, not finishing anything.

greenyoda3• 13 years ago

Reducing the complexity of products is an enticing ideal, but in reality, some products are inherently complex. Let's take one of Steve Jobs' products, iOS, as an example. No matter how much you try to simplify it, it's going to be a huge piece of software (I'm guessing hundreds of developers and supporting staff). Apple's success has depended on keeping customers engaged by regularly coming out with improved versions of products, so they need to estimate fairly accurately what the release dates are going to be: being six months late could be disastrous for marketing efforts and the company's stock price (e.g., you want to get the next version of the iPhone in the stores by Christmas). So you can be sure that there are people at Apple whose job it is to make very careful estimates of how long it will take to develop stuff.

habosa• 13 years ago

It's the halting problem.

brosco45• 13 years ago

Because software estimate is impossible, if it's possible, then you are re-inventing the wheel.

analog313• 13 years ago

Imagine estimating the time to complete a hike, and then being told that a bunch of children will be coming along.

gesman• 13 years ago

Allowing for end-customer's bad habits, such as delayed payments, real-time changing of specs and other unknowns that are fact of life for developer.

praptak• 13 years ago

Expect to be told, halfway there, that the party has moved to Tijuana. After all you agreed to hike to the party and it would be unreasonable to expect of the host not to move the party when the weather changes.

stevewilhelm• 13 years ago

To extend the analogy further: add the fact that you need to meet up with several other groups that are providing essential supplies and they are hiking from several different locations in the Central valley.

Some of the groups are showing up on time and are waiting for you, some are late, all won't have all the supplies you need.

And to make it more realistic: half way through the hike, your friends call to say the lunch has been moved to San Diego and is now a formal dinner party at which you will need to wear tuxedos. And can you bring the Champagne as well?

anotherfadjs• 13 years ago

Because no one wants to admit they still have to learn more about the system they are going to build.

noarchy• 13 years ago

There's something to this, no doubt. I recently took part in a project where a firm was contracted due to their "expertise", which they fulfilled by hiring a team of contractors, most of whom were woefully unprepared, in terms of knowing the system on which they worked.

lignuist• 13 years ago

Another reason is, that there are the first 90 percent and then there are the other 90 percent.

smokey_the_bear• 13 years ago

We sometimes make white label geo apps for other companies. We've repeatedly had problems where our estimates were off by quite bit because we got data in incredibly bad formats from the customers. And not just once, every time they'd send us updated data it would be in a different format.

The first time or two it made us hate the customer, and feel like it wasn't our fault. But then we started writing it into the bid, and telling the customer they could save money if they delivered conforming data. Hopefully we'll be happier from here on out.

eftpotrm• 13 years ago

Well, yes, but also (stretching the analogy) how many times have we been asked for a schedule to an unnamed destination via unspecified waypoints and then told it's too long and we need to make it shorter?

Devs make mistakes, sure, but as often as not in my experience it's absurdly vague end user requirements married to unrealistic expectations.

antonwinter2• 13 years ago

when asked to estimate, i ask if the biz if they can also give me an estimate for meetings, UAT, changes, hiring resources, unscheduled downtime and scheduled downtime and historic data comparing previous project estimates to actuality. Its the only way to get closer to accurate.

The developers estimates generally are best case scenarios without any external or motivational factors included.

11thEarlOfMar• 13 years ago

If you can suspend the obvious question of why are you hiking from San Francisco to LA in the first place, the general analogy holds quite well: With software development, you can not know how long it will take until you know exactly what you are building. You can not know exactly what you are building until you have fully described it. The act of describing it fully, precisely and with sufficient quality, is the act of coding.

Hence the completion estimate has to be continuously refined as the details of implementation are continuously refined.

Just like the slow zoom-in on the hike.

puppetmaster3• 13 years ago

As opposed to what? Marketing estimates? Scrum and such can be used by Marketing and others.

codex• 13 years ago

To put it simply: software estimates are hard because nobody has made that particular bit software before. Why? Because software doesn't rust. There is little need to exactly duplicate software which has already been written. Thus, most software is charting unknown territory.

sytelus• 13 years ago

Dumbing down analogies aside, you can look at software development as a graph of connected tasks with each task having some variance around when it would complete. This variance accumulates as taskw are completed and you move on to next nodes in the graph. When you complete the graph, you have large probability distribution about average ETA.

Most decision makers have to choose some number for ETA however because world does not understand probability distributions or variances. Most likely this has to be lower end of the number to get project sponsored or fend off the competition.

This is not unique to software development. Any projects that requires large connected graph of small task nodes would show same characteristics. For example, fighter jets, Boeing Dreamliner, space station etc etc.

avenger123• 13 years ago

I have found that providing estimates within a range is the most reasonable approach.

The range is usually between "this is the likely amount of time with everything going as it should" and "this is the worst case scenario time with all that can go wrong".

It's a pretty broad definition between the two but I find it gives clients a sense of the scope of the project. The minimum time does include some padding to include all the usual likely overruns (testing, requirements gathering, etc.) but not too much.

The worst case time better really be worst case as I provide clients the expectation that it should not take longer than this.

I find that usually I will end up somewhere in the middle of the estimate and the client is still happy since they didn't hit the worst case scenario.

I find that this usually tends to be relatively effective even if as a rule of thumb the worst case time is 3-4x the minimum time.

Of course, there is still a lot of analysis done to come up with this range and there is a lot of discussion with the client to help them understand the details on why and how it could go.

At the end of the day, communication is king. I find that educating clients about the complexities of the process always helps to gain trust.

Zenst• 13 years ago

People are generaly optimists and with that anything in the future will be biased towards ideals, even planning for exceptions will bias towards a ideal.

The other factor is feature creep, if you find a project so well defined and set in stone then it would be fair to say it will have a better chance of finishing on time.

That is just the software side, then there is the hardware to run it and that opens up a whole new aspect of poetntual delays that will get leveridged at the project and when people say project behind involving software people will just focus on the software aspect and all other factors are buried.

Bugs and issues come from many factors and even suppliers of hardware can have bugs which can have a impact, design could have bugs and many aspects can have issues that impact the end result.

But it is impossible and silly to plan for every situation as whatever you plan for there will be an exception and you get a deminishing return, hence easier to have a area of time labeled contingency as a catch all. Best example of such exceptions would be say an earthquake killing your entire development team, that can and has happened and at the same time is not something you plan for. Partialy why we have insurance. Though you will find it very hard to impossible to get insurace again software development delays beyond life/death of parties involved, though that will not chance the delay into a non-delay, mearly compensate.

With all this Engineers have for many lifetimes gone with the engineering factor or whatever you think it will take, muliply that by 2-3 to counter optimisim. But that does not help feture creep and anything added to the design after code has started is creep.

relet• 13 years ago

For the same reason other estimates are off by a factor of 2-5, unless contractual penalties are agreed on. The cheapest contractor gets the job, and can adjust the estimates later.

coldtea2• 13 years ago

Here's another question: "Why are software feature requirement estimates regularly off and changing while in progress?"

i_am_dead• 13 years ago

Only 2-3? Please tell me your secret!

dj-wonk3• 13 years ago

Summary: The fractal dimension of a software project is roughly 2 or 3. :) https://en.wikipedia.org/wiki/Fractal_dimension

Now, if you think about that metaphor for a while you may find value in seeing: (a) the metaphor doesn't really fit intuitively because software projects are often charting new ground, not zooming in on a predefined complex problem (b) the metaphor is not too bad of an intuitive explanation -- you learn more about what is really needed as you get closer to the details.

judk3• 13 years ago

Because the estimates are always BS, but teams get 2-3x slack before they are force to declare victory and launch whatever they have. The more over budget they get, the more they cut features (including QA and security details)

paulhauggis• 13 years ago

My estimates are always off because management decides to constantly change requirements or didn't have all of the requirements in the first place.

fayyazkl• 13 years ago

That's a very interesting sort of analogy. Among most of the discussed issues,

a) For one, giving enough time to come up with a project estimate is often rare. Usually people try to ask for those upfront.

b) Usually we try to add a general buffer for any unseen issues. However, the deeper you go and list tasks against time, the closer to realistic is the total.

c) Often it is just a customer giving a requirement needed at date x, or a management guy setting goal for a date y. When all you can do is to cut features but have to deliver by a given date.

d) No matter what we say, but at the back of our minds, we are not enough prepared to defend a timeline when giving it in the first place. Implementing feature x needs a week? gosh boss would say it should take a day. Latter it turns out the assumption was based upon a library which for just internal implementation reason (that you only discover upon learning it enough through actual code) is completely useless and it should actually be planned for 3 weeks at least.

Shtirlic• 13 years ago

All estimates should be multiplied by Pi.

grownseed• 13 years ago

Here's my experience with this problem. First, a number of people go meet the clients, often several times, to see what the client needs. Generally however, none of the people involved with actually building the product are ever involved in the meeting (because, from the mouth of the Boss, "operations are too expensive"), and in the unlikely event that they are, they're essentially shut down because "nobody really cares about the technical details at this point"...

This is then followed by extremely loosely defined requirements being passed down to the technical people in hope of getting an estimate back. Technical people, especially experienced ones, know how this goes. First, you know you don't have half the details you should have, even though you've most likely asked for them numerous times. But you know, "all I need is an estimate" says the sales guy (who in this case also happens to share the board, made solely of other sales related people). You already know you can't win here, so you give a ballpark estimate considerably above what you imagine should be the real value, hoping it'll cover your ass, but highlighting the fact that until more details are given, this is still a ballpark figure.

It doesn't matter. The sales guy takes your estimate, shaves a third off of it, throws in a couple of bonus features in there (which are most likely of no use to the client), and makes it a hard estimate. When you point that out, you're told not to worry and that it's not your problem (spoiler alert: it very much is). So you write your specs down, or rather, you slap a few bullet point lists together with whatever little time you're given (you're expensive). This is given to the client as what both sides should abide by, despite you repeatedly saying that it's a bad idea.

From that point, you realize that the time you've been given to achieve the work is, roll drums, not nearly enough. You can already see poorly defined features growing arms and legs: clients, and I can't really blame them, will use any vague requirements to their advantage. You also realize that in order to do all the work and still meet the deadline (you still most likely won't), you have to half-ass a bunch of the work. You can already imagine the support issues pouring in once the project is in production, and you see all those legacy projects constantly getting in the way of your new, poorly estimated work, thus slowing it down even further. You will also get blamed for all these issues that are cropping up from past projects, and this ties to the next point.

The best one of all, that thing that wasn't your problem, suddenly is. You've somehow become responsible for the poor estimates you clearly said weren't to be relied on. So you know, you better make an effort and fix that by, say, working extra hours here and there, like, every single day. You burn out and grow frustrated, which slows the work down even further. Even better is that your crazy work hours become an expectation, if you happen to finish a project on time this way, it'll be used as "See, no problem here, you got it done!", pat on the back and condescending compliments ensue.

So you realize you're never given time to do decent work, that at the end of the day you're better off not meeting deadlines, and that whatever happens, you're always on the losing side.

I'm not saying this is the case in all companies, but I've also seen this far too often. On the very few occasions I had the chance to be the main point of contact with clients, projects were on time, clients were satisfied, and of course, some sales guy still slapped his name somewhere on there and got his bonus, but at least I got to do good work on reasonable deadlines.

I loathe most sales and marketing people, I often find they're a completely unnecessary layer in product and value creation. It's not that their position couldn't be of value, but I don't believe it can be if the money incentive is always more prevalent than the idea of good work done or happy people. As Henry Ford said, "A business that makes nothing but money is a poor business".

Note: I've resigned from this job to work in research.

nathan_long• 13 years ago

In this analogy, you have a single destination; anything short of that is useless.

What about the agile idea that you deliver something incrementally useful with every sprint?

Obviously, some projects can't be done that way. But if they can, it would mean 1) estimation can be done on a 2-week scale instead of maybe a 6-month one, and 2) they can start deriving benefit/making profit from what you've built while you build the next thing.

zupa-hu• 13 years ago

thanks, was a great read!

programminggeek• 13 years ago

One reason is as an industry we are terrible at planning. We do a very unthorough job of planning details or requirements. Then we allow for changes and new features mid project with no respect for budget or time.

aufreak3• 13 years ago

Kahnemann and Tversky call it the "planning fallacy".[1] the interesting bit is that the predictions are pessimistic when you get "non-involved" parties to estimate time to completion of the same task. Perhaps using a figure between self estimates and third party estimates may work?

[1] http://en.wikipedia.org/wiki/Planning_fallacy

shitgoose• 13 years ago

If only they used one of those frameworks, that lift you off the ground and glide you towards the destination effortlessly, graciously pulled by flying pigs, there would be no delay. They would even have time for daily scrums. Always pick the right technology stack, guys!

cuillevel3• 13 years ago

I think the calculation usually goes like that:

  * Take a day to come up with "estimates" for the project,
    can't bill that time of course.
  * Sell fantasy gant chart to the customer.
  * Take 10-20% of the projects time away for "project management".
  * Actually develop the software.
  * Fix "bugs" until customer goes silent.

If you want good estimates you need to take estimates and project management more serious and get away from that waterfall methodology.

systematical• 13 years ago

After getting a number of my estimates wrong recently I was thinking about this question. While walking to work I determined a few things.

It's hard to estimate something that is complex. If I am very confident the job can be completed in under an hour my assessment is very close to 100%. This is because something simple does not have a lot of variables and interacts with very few things. Things like fixing a display bug, a simple calculation, adding a print now button are all easy. Once you go beyond this it gets more difficult to determine how long something will take.

Business wants the estimate done quickly. Many businesses want estimates done fast. An accurate estimate for a complex job cannot be given fast. There has to be several rounds of back and fourth. Use-case scenarios need to be determined, which will open up even more questions. The underlying architecture to accomplish the feature also has to be scoped which will take time and usually we don't get it right the first time we think about it

Some more reasons are, businesses want the job done quicker than is realistic. Businesses don't know everything they want. They think they need stuff that they never end up using. Haven't thoroughly thought out the feature or problem themselves and that uncertainty gets passed into the development phase more often than not. Business is not solely to blame though. Sometimes developers think a particular portion of their code will be simple when its not or difficult when it isn't.

Basically the whole process is riddled with human error on both sides.

pasbesoin• 13 years ago

Years ago, I was introduced to the (simplistic) dual concepts of profit centers and cost centers.

To many in Management and accounting (and with varying degrees of pointy-haired-ness), profit centers "make money" while cost centers "spend/burn/"waste" money.

The sun shines on the former, while the inquisition perpetually stalks the latter.

Systems and systems software are often seen as a cost center. There is constant pressure to "control" these costs and to do more with less.

People are required to be over-optimistic in their communication while at the same time overly pessimistic ("frugal") in their resource allocation. Note the use of the word "pessimism". It is not so much a matter of "optimisim" thinking that "we can get it done with less"; rather, it is the pessimism of "we can't afford that".

Much of my professional life has been spent in big, "conservative" (cough), risk-averse organizations. I'm sure things are to some degree different elsewhere -- some degree.

Nonetheless, it's worth keeping in mind this (simplistic, and to varying degrees counter-productive) concept of "profit centers" and "cost centers". And if you are working for the latter, expect to be squeezed into ever more unrealistic positions.

It's something the seasoned professional stays attuned to. There will always be some squeezing; when it starts to worsen and become excessive, it's time to go. Your personal solution, as a "simple" employee, is to find another organization -- the sooner, the better.

judk3• 13 years ago

That's a great story for why the first project is slow. But next time, you say, OK, we can hike 10 miles a day, so it's 60 days, and it still comes out 2-3x of budget. Why?

sechastain• 13 years ago

The short answer is optimism and poor assumptions. The hiking analogy works well to illustrate this answer.

Everyone who has been a part of a software project knows about poor assumptions. From the start, you don't have well-defined requirements. You don't have a complete design. You don't understand the consequences of decisions you have made or will make. Any estimate is doomed to fail in the face of all these poor assumptions.

Staggering a software project into smaller chunks goes a long way to controlling these poor assumptions and the impact to schedule and, more importantly, customer expectations.

The principles of the agile manifesto go a long way for mitigating the too common pie-in-the-sky wishful thinking that dominates software engineering. They work even better when the customer realizes they are a part of the system under development and its success - if they can keep that balanced with giving the development team the space to do what it is they do.

Why are software development estimates regularly off by a factor of 2-3?

Related Stories