All estimations are wrong, but none are useful

https://newsletter.techworld-with-milan.com/p/all-estimations-are-wrong-but-none

by Kerrick • 1 year ago

49 71 1 year ago

newsletter.techworld-with-milan.com

brudgers3• 1 year ago

Construction projects get reasonably accurately estimated every day. Probably because:

1. People have been doing it long enough that estimator is a job description.

2. That it is a job description, means money is spent on estimation.

3. Money is spent on estimation because getting it wrong can cost money.

I think the problem with software estimation is that there is usually no culture of estimation, usually very little direct incentive to get it right, and no regime of formal training for those responsible.

To put it another way, software does not have standard documents for estimating.

blastro• 1 year ago

Well put!

HPsquared• 1 year ago

Never mind cost/time estimates, most software projects barely have anything resembling blueprints, bill of materials or even size estimates at the start of the project. It's hard to estimate something if you don't even know how big it'll be, or what shape.

9rx8• 1 year ago

most software projects barely have anything resembling blueprints, bill of materials or even size estimates at the start of the project

Producing that is the software project. Once you've delivered those things, the compiler will carry out the construction. I am sure you can estimate how long the compiler will take with reasonable accuracy.

So, you're right, construction isn't the greatest analogy.

bliteben• 1 year ago

The ease and seemingly no-cost nature of small changes in software development, causes scope creep that would have huge costs in construction.

fhennig• 1 year ago

Hmm.

What about the difference between building a house from a blueprint vs building software from tickets?

IMO the blueprint is a much clearer plan.

EDIT: In fact, I'd argue that the software is more like the blueprint itself than the finished construction project.

Ygg2• 1 year ago

Houses would be much different if they could be built on the inner side of the black hole.

SideburnsOfDoom2• 1 year ago

There is a better analogy between "compiling and installing software" and "building a house from a blueprint". The difference is that in software, anything that can be automated, now has automation. Which is easier with bits than it is with bricks, wires and pipes.

The process of writing software is more analogous to drawing up the house blueprints.

This is where the the whole question "If building houses is predictable, why can't writing software be?" falls apart. Because the unpredictable part of housebuilding is over when you have a bespoke or off-the-shelf blueprint. Source code is a blueprint. The software compile-test-deploy-install pipeline is actually fairly predictable.

t00ny• 1 year ago

Software development inherently involves more unknowns than fields like construction, making estimation harder. However, when tasks are broken down into simpler, well-defined components with clear requirements, estimation accuracy improves significantly. The challenge lies in the fact that software projects often begin with ambiguous goals, evolving technologies, and dynamic stakeholder needs—factors that are less prevalent in physical construction.

9rx8• 1 year ago

Software development inherently involves more unknowns than fields like construction, making estimation harder.

If by construction you mean only cookie cutter houses and garden sheds, maybe, but construction doesn't end there. As soon as you look to anything interesting, construction has all the unknowns software has and an environment that brings even more unknowns. Computing environments, in contrast, are highly predictable. Software has it easy.

The trouble here is that the original premise is flawed. Construction estimation is almost never accurate. Open today's newspaper and you're almost certainly going to read about yet another construction project that is over time and over budget by millions upon millions of dollars.

SideburnsOfDoom2• 1 year ago

4) A house, e.g. "3 beds, 2 baths, timber frame" is a much more known quantity with standard tools and techniques that have been executed many many times before, than a typical software project. The house builder can average the last 100 times that spec has been executed. The software project cannot. It is inherently much more variable and speculative.

It's not entirely to do with "blame the people" ideas such as "no culture of estimation" and "little direct incentive". The nature of the project and the way to go about is simply much more wide open. And doesn't have such easily-defined endpoints.

halayli• 1 year ago

This is more or less reverse reasoning starting with a false equivalence. Most construction projects that are a tad different than the typical go over budget and over time. Most construction projects are thesame. If you ask a software engineer to build the same project repetitively, they can give you a very reasonable estimation.

yen223• 1 year ago

The cool thing with software engineering is that we can drive the costs of building the same project repetitively down to zero. We don't have to write Postgres from scratch each time we spin up a CRUD app, we can just install it.

I think that's why estimation in software engineering is difficult. The predictable repetitive portion of the project takes up no time - all that's left are the novel unknown parts.

sjamaan• 1 year ago

I also had to do a double-take at the blatant falsity of this statement. Software estimation is only hard when it's the first time you're doing it. Unfortunately, most software is a first-time.

There are very few specialty shops which only do the same type of project over and over. This is probably also because such repetitive work would quickly be turned into a "framework" to do more of such work, leaving only the novel stuff for each project, which would then become difficult again.

In the real world, I guess that matches up with the "prefab" type of construction.

brudgers3• 1 year ago

Most construction projects are the same.

That is not the case.

It is just that there’s no drama in on-time and on-budget.

dkarl2• 1 year ago

"time estimates from software devs actually make sense if you think of them as 'this is how long it would take me the second time if i did this twice in a row'"

(Source: https://twitter.com/mechanical_monk/status/17468921975201382...)

esafak• 1 year ago

In software we're always doing new things, relying on new code internal and external, so we always have to learn new things and run into unexpected problems, neither of which can not be reliably accounted for. The more routine your task is, the easier it is to provide an estimate. Construction technology and blueprints are not constantly changing, right?

renewiltord• 1 year ago

Are we fucking around here? There are exactly two construction projects that got estimated on time and cost in the US. Cost overruns are so routine as to not be noticeable.

In fact, if this is not the case, then this makes a good argument for the cancelation of CA HSR since it must be run by notoriously incompetent people since it is delayed decades and over a hundred billion over budget.

9rx8• 1 year ago

Yeah. Construction projects arriving on time and on budget is almost unheard of, aside from maybe the construction of common structures, like residential homes, where the builders have already built the same building 100 times before.

roxolotl• 1 year ago

In addition to all of the other comments a good analogy I'd heard in the past is related to plumbing.

If 80% of the way into building a house you realize you want to swap the bathroom and the living room you'd be laughed at. The pipes are laid you and you'd have to drop back down to the 50% complete level to move them.

But this happens all the time in software. It's why so many projects get stuck at the 80/90% level. Because those defining requirements, and many times even engineers, don't have a concrete understanding of what is plumbing and what is finishing it's very difficult to know whether the ask is to hang wallpaper or move a toilet.

pjc50• 1 year ago

It happens remarkably often even in big construction projects, when requirements shift for political reasons or when something unexpected is discovered about the ground it's being built on.

bluGill2• 1 year ago

Such swaps happen all the time in construction. We call them "change orders" or "remodeling" depending on when in the building lifecycle they happen. In software we have not be good about figuring out which change orders are expensive.

I've been in construction, most of them are actually really cheap because the common/likely change orders are built in - want to add a fireplace: until the drywall is installed the cost doesn't change. Want to make a doorway bigger, no problem, it will cost you $20 + the bigger door. Want to switch the tub for a shower after the pumping is roughed in - $50 to move the drain and water pipes, but once the tub is installed you need to pay for the tub you don't use as well (it has to be broken to get it out). As you say moving the bathroom to a different location will be costly.

sholladay• 1 year ago

I think you’re right to a degree. Certainly there is a lazy approach to estimates in software compared to other fields. But I think that’s because the estimation is hard enough to feel hopeless at times. Unless the task is truly something mundane that’s been done 100x before, as in the case of construction where many things are standardized, then the buffer for uncertainty needs to be huge, meaning everything is estimated to take a long time, making the whole process pretty useless.

Even for mundane tasks, I frequently find bugs in libraries, software versions that are incompatible with each other unless I apply just the right combination of upgrades and downgrades. SaaS vendors change their pricing structure or deprecate the APIs I am familiar with. At each step, there’s a little bit of debugging, learning, and creativity in working around problems. Each task is a small piece of art, important or otherwise. And you wouldn’t bother asking a painter how long their landscape will take to finish, would you? It could take a day or it could take months. Timebox it if you want, but that won’t have the same result.

You can save time and make better estimates by never going off the beaten path and by reducing your reliance on 3rd parties, but that only gets you so far and results in boring, homogenous software. Kind of like modern architecture and construction practices, which plenty of people don’t like.

gilbetron2• 1 year ago

I don't think you understand software or construction. Construction runs over all the time for many reasons in common with software, but in general, construction is a horrible analogy to use for software development.

Tractor8626• 1 year ago

No, even construction projects don't get accurately estimated. Software development is R&D, not a construction - lot more unknowns. It is not skill issue by any means.

alistairSH2• 1 year ago

I'm not convinced "software development is R&D, not a construction" is really true. But, I agree there are more unknowns.

Maybe more accurately... - construction project estimates are done after architecture and some engineering analysis are completed. - software projects often include the architecture and analysis

datadrivenangel3• 1 year ago

One insight here is that there is a cost to estimation, and that your marginal improvement on estimate accuracy begins to level off, so too much estimation is waste.

dkarl2• 1 year ago

This is literally the FIRST bullshit I was fed when I started my software development career. Buildings are always on time because they're too important to be finished late! Buildings are expensive! Buildings are serious! If a house wasn't finished on time, a family might have to find somewhere else to live! If a sports stadium wasn't finished on time, the season would be ruined! If an office building wasn't finished on time, businesses with leases would have to find other office space! Hundreds of millions of dollars would be lost! Nobody could afford that, and it doesn't happen, because the people who build buildings understand How Important It Is! If programmers would just grow up and start acting like responsible adults like the people who build buildings, maybe start wearing collared shirts to work, then they would start shipping software on time!

Yeah, except all of those things do happen. Families spend months renting while their new house gets built. My city's new professional sports team played on the road the first several months of their first season, like the previous two expansion teams before them, because the stadium was behind schedule. Office buildings with 100% of their office space pre-leased have to push back their opening dates, leaving all their new tenants in the lurch and giving them the right to cancel their leases and go elsewhere. It turns out construction estimation is not reliable, either, despite the fact that much of construction is more analogous to a software build process than a software development process.

A lot of the reasons are pretty common-sense, and many will sound familiar to software developers:

- New building products that were expected to be drop-in replacements for older products turn out to have unexpected limitations.

- Available labor isn't as familiar with materials as expected.

- Economic conditions change, and it takes longer than expected to get materials.

- Economic conditions change, and the client decides to make changes to the project in response.

- Something about the site that was expected to be discovered during site engineering gets discovered at the beginning of construction instead.

- Another project going over schedule or over budget has a cascading effect.

- Labor becomes so scarce that construction has to be started late.

- Materials are accidentally ruined on site, and they can't be replaced quickly. (Imagine that a junior engineer arrived early one morning and ran a bunch of builds on a broken branch. Now you're scrambling to locate a supplier who can get you more build juice, and they all have a two week lead time.)

Unlike in software, money and time can often be traded off in construction. When labor and materials become scarce, you can pay extra to be the project that gets the best available, or gets them at all. This can create the appearance of more money buying better estimation, which is true to an extent, but sometimes it's simply a consequence of bigger projects have more on the line and more cash available. If you are building a $1m house, you are likely to discover that the team building a $5m house across town can afford to pay more to stay on schedule than you can. Likewise if you're building a $10m commercial building and something gets scarce, it's a good bet that the $100m project across town will offer more for it. They'll finish on time, you'll just have to wait.

gwbas1c• 1 year ago

In construction, if all you're doing is building (essentially) the same house, with minor variations, and little risk, you can make very accurate estimates.

In contrast, the construction projects that go over schedule and over budget typically involve a certain degree of novelty where you just can't say, "it typically takes me 5 months to build a house from these plans, so it'll probably take 5 months."

The thing with software is that no one writes the same program over and over again. It's more like the situation where someone is building a unique bridge and it goes over budget.

variaga2• 1 year ago

In my experience, a substantial portion of poor estimates comes from:

1. An initial estimate is made that is fairly accurate.

2. Someone in management says "that's too long, we've got to find a way to bring in that date"

3. The estimate is revised to assume the best case possible execution time in each task, that resources will be available as soon as they are needed, that everything is fully parallelizable.

4. The estimate now magically matches the target date.

5. Actual execution time is close to the original estimate.

People make bad estimates because (other) people don't want good estimates.

9rx8• 1 year ago

Early in my career, when I was young and naive, I used to spend an inordinate amount of time coming up with the perfect estimate. I could pretty well get it dead on nearly all the time, at the cost of wasting a stupid amount of time to determine it.

I soon noticed that it only ever lead to "okay, thanks" from whomever made the requested and that was the end of it. So now I just pick a random number. I still get "okay, thanks" and nothing changed.

Nobody really cares. It turns out it is just something people ask when they are uncomfortable with silence, but can't think of anything else to say.

variaga2• 1 year ago

That tracks. Frequently what people learn about estimating is to optimize out steps 1 and 2 and just make the estimate match the need-by date from the beginning.

scott_w• 1 year ago

My favourite joke whenever I tell people I make up estimates is “I’m completely wrong but I’d rather spend 30s and be wrong than 3h and still be just as wrong.”

Eddy_Viscosity2• 1 year ago

The managers are just putting that number into a spreadsheet which then gets combined with other spreadsheets and given to an upper level of management which does the same.

When I get asked for estimates I always ask what number would work best for them and then say that. You'd think I'd get a pushback on that approach, but I don't.

timmg• 1 year ago

I'm sure that happens a lot. In my experience, people tend to be bad at estimating things.

As a long-time SWE who eventually (also) managed teams: I found estimates particularly difficult. Some SWEs would give very optimistic estimates (and always miss). Others would "sandbag" their estimates, because they didn't want to have to explain why they were late.

I tried really hard to make sure the SWEs on my team didn't get "blamed" if they were late and asked them to generally give aggressive estimates. My rationale was that if you estimated more time than it really took, you would tend to fill up that time no matter what.

claudex• 1 year ago

It depends what you do with the estimate. If you have to deliver a product/a functionality where X people depends on it to continue. It's best, in the general case, to overestimate a little to not have the X people preparing to work when they can't. If you estimate to schedule the task for the week, it can be more useful to be more aggressive since missing the deadline is less impactful.

For example, there are a lot of task that I depends on which can be completed in one week or three month. I don't care which it will be, I just want to know when it will be done so I can schedule other tasks before.

Viliam1234• 1 year ago

Some SWEs would give very optimistic estimates (and always miss).

This seems to be the easiest case. Keep a spreadsheet with two columns: the time they estimated, and the actual time. Calculate the average coefficient between the columns. Next time they give you an estimate, don't comment on that, just privately multiply it by the coefficient.

thecupisblue• 1 year ago

I've had similar experiences but also reversed - saying estimates are too short, adding a buffer, saying they don't seem valid and we need to account for potential issues.

But also situations where a sales person would throw an estimate to the client which would end up being the official estimate due to no pushback. Or clients stretching the scope of a feature. Or underestimating/lack of understanding of a feature. Or changes from up top in the last days of building a feature.

Or the worst - overestimating due to previous experiences, where the dev team ended up on an 8 month estimate for 3 months worth of work tops which, if true, would have probably sank the company.

They can really be correct or close to truth - and useful to know what an ideal/okay/worst case is for planning, but are often then used as the holy grail and broken due to external influences.

The most useful thing about them is actual ritual of estimating, as it requires the team to mentally go through the items in the spec and align on things they usually would handwave away.

datadrivenangel3• 1 year ago

We value optimism, and telling ourselves that something won't be that much work makes it easier to start and commit ourselves to finding out that it's more work than we want later.

3vidence• 1 year ago

I've found in my career so far estimates are very useless but "key dates" can be very useful.

A key date could be an investor demo, marketing launch, actual physical launch if you are in hardware, etc, most importantly something with an audience outside of any of the core team. Find that important date and work backwards from it and find out what is important to finish by then.

This approach has defintiley saved me a lot of sanity compared to arbitrary "estimates" that are destined to be push forever until you run into one of these key dates.

mattferderer• 1 year ago

Risks, risks & risks.. That's my #1 priority on communicating estimates.

Overall this is a nice short summary on the topic. The one thing I would add that I found very helpful on larger projects is communicating the risks & unknowns. I suggest listing them out at the start of the project & update their status as you work on it.

I've worked on teams where it's done with a simple color (red, yellow or green) on how confident we are on the task estimate based on risks/unknowns. This is the simplest way in my opinion.

I also like Basecamp's Hill Charts - https://3.basecamp-help.com/article/412-hill-charts

ksec• 1 year ago

Skim read the article, why is it "But None are useful" and not "And none are useful"?

turnsout• 1 year ago

It's a reference to "All models are wrong, but some are useful"

PerilousBardom• 1 year ago

It's a play on the phrase "All models are wrong, but some are useful" in reference to statistical models (but often applied more generally to scientific or mental models).

donatj• 1 year ago

My first job I worked was for a small agency. I was kind of a young go getter and was able to very quickly pump out bad (in hindsight) yet mostly functional code very quickly.

Sales people figured out they could come to me specifically for estimates because they would be shorter than other developers estimates and more likely to get the sale. I was young and dumb and didn't connect that often times I wouldn't be the one developing the project. The other developers got angry with me after they caught on to why they were getting impossible timelines.

I started multiplying my personal estimates by three. The sales people were less than pleased and eventually started going elsewhere for their estimates as greener developers were hired.

j_maffe• 1 year ago

I hate it when people quote Hofstadter's "law" in a literal sense. It was never made to make any statement about timeline estimation. It was just an example of a recursively-defined law. Honestly I'm not even sure how many of the other quotes are also taken out of context.

teeray• 1 year ago

Software estimates also often fail to account for external factors. You may have a fairly good sense for how long some feature will take to get to code-complete. However, code reviews on many times are nobody’s job, so they happen whenever (and you end up with a Volunteer’s Dilemma of completing them). Then CI, linters, etc. may have various failures. Some of these may be legitimate, others may be flakiness your team has decided to ignore, or even infrastructure issues that prevent them from running at all. There’s enough X factor in external sources alone to make any estimate you provide total nonsense.

amelius• 1 year ago

If none are useful, then why does my manager keep asking me for them?

JackFr• 1 year ago

It's a low effort way to make you consider all the necessary steps and possible contingencies for delivering the work. Creating the DB, onboarding to the API, etc., etc.

hakaneskici2• 1 year ago

This. It's usually all the things besides writing code that takes most of the time.

gilbetron2• 1 year ago

This remains great reading for software estimation: https://www.researchgate.net/publication/247925262_Large_Lim...

I'm barely even interested in talking about the subject anymore - estimating software is only useful as an alternative viewpoint to help understand what you are building. Nobody accurately estimates complex software in any meaningful way. Even min/max/most-likely estimation isn't that useful, but ok to think about.

Hidden complexity and simplicity lurks everywhere. "3 month projects" turn into 3 day projects because of an unknown tool. Or a simple problem (think fermat) turns out to be insanely complicated.

The core issue is that when you create a function, or module, or whatever, you never need to do it again. So you are always creating new things. If you are writing the same code again and again, you are bad. Now, the past 10 years or so, that was less true as we wrangled with yaml configs and gluing together pipelines and modules and services. But I think AI is really good at that stuff, so I think we'll be back to always working on original things.

And that doesn't even take into consideration legacy codebases - good luck estimating changes to those ;)

hkpack• 1 year ago

You can only estimate job you’ve done before.

Everything else is just guessing.

Software engineering is not uniform: agencies working on similar projects may have good estimates.

R&D and novel approaches - usually takes whatever time it needs and then some more.

myrmidon• 1 year ago

My take is (and I somwhat agree with the article on this) that if you know requirements, interfaces and tasks precisely enough to give a reliable estimate, then that means that the majority of actual development was already done.

But thats just not the point where people typically want estimates, they want them much earlier.

gedy• 1 year ago

In 25 years in industry, I have never seen estimates proven to be valuable or important, mostly used for dumb purposes, and generally are a waste of time.

At this point I feel Kanban style priorities and limiting work in progress to be only useful approach.

When some product person dumps a huge task in the priorities, break it down into smaller deliverables. No estimates are really needed.

evklein• 1 year ago

ALL STORY POINTS ARE EQUAL BUT SOME STORY POINTS ARE MORE EQUAL THAN OTHERS

https://evklein.com/posts/agile-farm

didgetmaster• 1 year ago

Building software is often like building a house. When construction starts, progress appears to happen fast. You put up walls pretty fast so it starts looking like a real house in a short time. Later, construction progress appears to slow down significantly as work shifts to detail work (wiring, plumbing, etc.) which doesn't change the appearance from the street much.

With software, the basic UI can take shape quickly. Some rudimentary functionality sometimes comes along quickly as well. Then all the detail work (error handling, logging, performance enhancements, etc.) makes progress appear to slow significantly.

hakaneskici2• 1 year ago

The value is in the activity itself, like planning, rather than the final document.

zadkey• 1 year ago

I mean saying ALL estimations are wrong is exaggeration.

As they say, "Even a broken clock is right twice a day"

This has been my experience. I don't disagree with the content of the article, really just the exaggerated title.

Every now and again we do get a 100% correct estimation. It doesn't happen often. In fact it's quite rare. But its more than never.

All estimations are wrong, but none are useful

Related Stories