Responding to “Are bugs and slow delivery ok?”

https://uselessdevblog.wordpress.com/2023/07/03/responding-to-are-bugs-and-slow-delivery-ok-the-blog-post-that-ive-hated-the-most-ever/

by thunderbong • 3 years ago

84 124 3 years ago

uselessdevblog.wordpress.com

von_lohengramm• 3 years ago

This is how I feel about most "things I hate". Despite my strong feelings towards them, there's a reason why they are the way that they are. Almost always it comes down to my interests & priorities not aligning with reality/everyone else.

contravariant2• 3 years ago

To some extent hating stuff that frankly has no reason to exist is by definition temporal, it only lasts until the thing rightfully stops existing.

To really grind someone's gears they'd need to hate something that cannot simply go away.

int_19h3• 3 years ago

In this case I would argue that the interests of most people don't align with the "reality", but it is the way it is because it aligns with the interests of those people who have the most power to affect it - in this case, top executives and major shareholders.

ClosedPistachio• 3 years ago

roenxi• 3 years ago

Valentina's original blog post is easily misinterpreted. She is correct that quality is optional. Obviously so, many companies do fine with very poor quality software.

But something being mandatory or not doesn't influence the ROI. Something can be optional and high ROI or mandatory and low ROI. Low quality software is quite destructive over the long term and it is a horrible mistake to let coders go wild writing buggy software slowly.

That being said, the post is probably clickbait aimed at promoting her services. I doubt many people would make the mistake of thinking it a good idea to purposefully low quality written at slow speed.

userbinator• 3 years ago

Both this article and the one it references only seem to provide more evidence that the descent into mediocrity of the software industry is certainly happening. That said, there's a difference between actual quality, and "quality" as popularised by metrics-driven dogma.

H8crilA• 3 years ago

Example for the last point: people seem to have forgotten how much of a "quality" shitshow was Twitter. The thing was written in Ruby on Rails, downtime was normal and expected by everyone. Yet the platform thrived and the username grew. Or Facebook, the page didn't fully load like 30% of the time, meaning that at least one panel was broken, some picture was missing, etc.

contravariant2• 3 years ago

Always worth remembering the adage that you can't inspect quality into a product.

Which, interestingly, wasn't something a software engineer came up with.

DougBTX• 3 years ago

A blog with ads every few paragraphs isn’t where we’ll find writers with standards.

adamsmith143• 3 years ago

Maybe, just maybe, the best people to build complex software systems that serve millions of people aren't the ones who can do the most leetcode problems.

distcs2• 3 years ago

It really saddens me to know that the hacker culture I once so much loved to be part of is rapidly disappearing from the industry.

xupybd• 3 years ago

Is it? Or is a massive industry forming around a still small group of hackers?

Maybe the two are groups are not the same. Someone getting into software to make money is different to the kid who couldn't put the computer down. There are people that want to hack on software and others that want to solve business problems with computers for profit.

PragmaticPulp• 3 years ago

You know what I haven’t seen? not once in 15 years?
A company going under.

What a wild assertion: The OP hasn’t personally seen a company fail, and therefore software quality doesn’t matter? Bugs and slow delivery are fine?

It’s trivially easy to find counterexamples of companies failing because their software products were inferior to newcomers who delivered good results, fast development, and more stable experience. Startups fail all the time because their software isn’t good enough or isn’t delivered before the runway expires. The author is deliberately choosing to ignore this hard reality.

I think the author may have been swept up in big, slow companies that have so much money that they can afford to have terrible software development practices and massive internal bloat. Stay in this environment long enough and even the worst software development practices start to feel “normal” because you look around and nothing bad has happened yet.

ryandrake• 3 years ago

“My company didn’t go under” is pretty much the lowest bar you can shoot for.

This blog post is saying “Staying healthy doesn’t matter because neither I nor anyone I know died so far.”

jbreckmckye2• 3 years ago

"and I've never met any dead people either"

balaga01• 3 years ago

Well, what is your goal?

Sure, healthy and happy chickens are nice, but if you are into industrial farming and your goal is to make money, you will make their life as miserable as necessary to extract profit. You don't care if they are barely alive as long as the health standards are met.

It's not about you being happy as an individual, it's about the company making money. If your lawyers and sales are good enough to build a captive market of miserable customers, your company can still make a ton of money and be very successful.

jchw2• 3 years ago

This analogy is a bit tortured but it's honestly pretty good anecdata that something probably doesn't generally kill you if you've been around people doing it for 15 years and none of them have died yet. It can kill you, but that doesn't mean it generally does.

I'd love to come to the conclusion that it's wrong, but it's right. Most companies absolutely 100% can afford to ship bugs: the proof? They're doing it, they have been doing it, they will continue to do it. That doesn't mean that every company can, always, forever. A single really bad bug can tank a company. In the right conditions, simply being buggier to a superior competitor can tank a company. However, those are mostly just things developers fantasize about. The market is by and large not decided by how elegant and well-designed your software is, moreso in some verticals than others. In fact, this is basically true for almost anything that isn't explicitly developer oriented in the first place.

Just look at enterprise software. Jesus Christ.

icedchai• 3 years ago

From personal experience with early stage startups, it is quite a low bar. I've seen companies "acquired" for pennies on the dollar, after almost the entire staff is laid off. Somehow, this is considered a success even though you would've had an better outcome investing your time and money in almost anything else. Employees lost, investors lost. The founder often gets a cushy job they could've had anyway.

codetrotter• 3 years ago

Agree.

For example, pg said that ViaWeb was successful because they had put care into their code, which allowed them to iterate quickly and integrate new features that customers requested. Whereas competitors were held back by their cumbersome code and slow cadence of releasing features.

meepmorp• 3 years ago

pg would say that, though, because it was his company and it worked out for him.

But maybe he was actually wrong about quality being a differentiator, and his competitors could've shipped features with shitty code but had other problems.

fifticon• 3 years ago

I was in a successful company that nearly died in 2017, where our entire production system corrupted itself due to a sneaking scale bug we had ported the system into. The problem was, the system with data had been running migrated for 3 months with the bug in it, so it was no longer possible to revert to the earlier working design. We were down for a week where no clients could run, and we spent the next 12 months purely digging ourselves out of that hole, with all new development paused, and all hands on limping the ship along. I would say, that bug was very close to ending us. Luckily, we never disappointed in a similar way since.

tobr• 3 years ago

What I’m hearing here is that even a software problem so bad it forced you to focus on nothing except trying to fix it, for a full year, wasn’t bad enough to make the company go under.

the_gipsy• 3 years ago

It should have been "going under due to bugs / low quality and slow delivery".

Yoric• 3 years ago

For what it's worth, Mozilla was nearly killed at least twice by code quality.

Once upon startup. The code inherited from Netscape was... let's say charitably difficult to maintain. Turning it into something that could actually be improved upon and tested took years without any release of the Mozilla Suite.

Once when Chrome emerged. Not because the code was particularly bad, but because its architecture didn't reflect more modern requirements in terms of responsiveness or security. See https://yoric.github.io/post/why-did-mozilla-remove-xul-addo... for a few more details.

octokatt• 3 years ago

For what it's worth, Mozilla was nearly killed at least twice by code quality.
That’s not to say that I now think that “High quality good! Low quality better!”. As many commenters on Cupać’s post observed, while high quality isn’t necessary for success, it improves the chances of it.

_Nearly_ killed. But Mozilla is still kicking around. Maybe not doing the kind of internet-shaping work they once did (here's hoping they get back there), but they continue to exist.

drewcoo2• 3 years ago

It’s trivially easy to find counterexamples of companies failing because their software products were inferior to newcomers who delivered good results, fast development, and more stable experience.

Is it? What I've seen is the opposite.

Businesses can be terrible top to bottom, slow, inefficient, and painful for customers, and still keep going for years and years. It's more about income/funding than product.

I think the author may have been swept up in big, slow companies that have so much money that they can afford to . . .

That's what I'm talking about. They are legion! They could be companies that serve a niche that no one else does or with prohibitive switching costs (training is expensive). They could also be companies that somehow got enough market share that "no one gets fired for buying IBM."

Also, you know what those "big, slow companies" have in common? They are successful businesses. Unlike most startups.

olivierduval• 3 years ago

I think that it is (a little bit) more subtle: the importance of quality OF A PRODUCT (projects delivery is another beast) is relative to:

- the customer: either B2B or B2C

- the market share: minimal (< 1%... including all startups) or dominant (> 30%)

- B2C is really dynamic and few bad versions/products can make the customers fly away (except when strong dominance - like Windows - or no equivalent product) and shutdown a company. Price can be a strong factor and cost of migration/switching is usually not considered

- B2B is more conservative: hard to enter the market (so small market shares will need a lot of time to take off... if there's no competitor) but once you're in, the cost of change for a company is usually high enough to tolerate more bad versions (and more if there's few competitors, and incompatibilities between products, and legal requirements to keep records, and a lot of "configuration", and requiring a strong training for a lot of people...). Companies as customer dont see the switch of software as a technical problem (replacing on editor by another one) but as a management problem (training, cost to switch, data availability, availability of people already trained, cost of multi-year/multi-instance licences...)

KronisLV2• 3 years ago

It’s trivially easy to find counterexamples of companies failing because their software products were inferior to newcomers who delivered good results, fast development, and more stable experience. Startups fail all the time because their software isn’t good enough or isn’t delivered before the runway expires. The author is deliberately choosing to ignore this hard reality.

While I personally haven't seen a company going under due to bad code, one can also definitely make the argument that software that is buggy or doesn't scale will lead to lost profits, which could also eventually become a problem, or present risks to business continuity.

I still recall working on critical performance issues for a near-real-time auction system way past the end of my working day, because there was an auction scheduled the next day and a fix was needed. I also recall visiting a healthcare business which could not provide services, because a system of theirs kept crashing and there was a long queue of people just sitting around and being miserable.

Whether private or public sector, poor code has a bad impact on many different things.

However, once can also definitely make the distinction between keeping the lights on (KTLO) and everything else. If an auction system cannot do auctions for some needed amount of users, that's a KTLO issue. If an e-commerce system calculates prices/taxes/discounts wrong and this leads to direct monetary losses, that's possibly a KTLO issue. If users occasionally get errors or weird widget sizing in some CRUD app or blog site, nobody really cares that much, at least as far as existential threats go.

Outside of that, bugs and slow delivery can be an unfortunate reality, yet one that can mostly be coped with.

esafak2• 3 years ago

Friendster is memorable example. It did not scale well, and fed up users flocked to MySpace and Facebook, which came later.

More generally it depends on how competitive the space the product operates in, whether quality is something the buyer values and is able to evaluate! Enterprises for example infamously don't appreciate quality as much consumers because the economic buyer does not use the product.

sanitycheck3• 3 years ago

I've seen bad software and slow delivery lose companies important clients, causing job losses and severe financial problems.

I've also seen companies go out of business because they tried to make perfect software and never actually delivered it to anyone.

The right approach, I think, is to do the minimum necessary and do it very well. Easier said than done though.

cybrox• 3 years ago

I'd like to add: Do the minimum that is necessary. If it's well received, take the time to refine it. If it's not, scrap it.

aaronbrethorst• 3 years ago

This blog post is a response to an advertisement for coaching services. The article it responds to was so content-free and sales-focused that it actually got flagged here on HN.

https://news.ycombinator.com/item?id=36615325

I don’t think anyone needs to waste their time arguing with a strawman.

csydas2• 3 years ago

If I scribble a random time stamp on a piece of paper and frame it and call it a clock, it's still right at least once a day (24 hour notation or bust :) ) The original article was just a garbage native advertisement with the wrong conclusion, I agree, but the ideas it stated weren't really wrong; Microsoft is a pretty good example of a company that constantly leaves its products in very buggy states for long periods of time, and it expects you to use the products in that state. (read: pay Microsoft) They're still dominating. While I admire the ReFS team for what they pulled off, the first 2 years it was virtually unusable if you tried to use it like they told you to, either due to memory leak bugs or outright corrupting your data silently.

It still got used and was one of the common deployments for MS SQL and Exchange during that time.

The original article doesn't sell a solution, it sells some spin from the author, but the original author wasn't wrong that there is a ridiculously high tolerance for buggy, low quality software, and the reason isn't super clear. These aren't cheap services either, they're billion dollar companies that can barely keep the services running sometimes or are incredibly slow to react to missing features or features not working well/correctly.

I think this situation merits discussion.

thih9• 3 years ago

the original author wasn't wrong that there is a ridiculously high tolerance for buggy, low quality software, and the reason isn't super clear

I disagree. I’d say the reason is clear: the market decides; and the decision is that current amount of bugs is ok. Most software is fine for the end user.

We (devs) notice the bugs because we’re familiar with the process; but in the end we’re just people who know how the sausage is made.

kayodelycaon• 3 years ago

You'd be surprised how many businesses are run by completely incompetent people and still manage to survive.

A relative of mine who was an accountant for a small manufacturing plant and the systematic problems from top to bottom were shocking. They worked there for well over a decade and also managed a lot of other business aspects. Nothing got shipped, billed, ordered, or paid without them knowing. The company always had just enough income to keep going. Somehow, it was sustainable without any hidden external sources of money.

agentultra• 3 years ago

It really, really depends on the use-case. You certainly don't want bugs in your MMU, process scheduler, sensors, security libraries, etc.

But a piece of line-of-business software that sends out the wrong email once because someone made a bug in a batch script? Sure. Not the end of the world.

Users will tolerate some level of errors. Even if that level is sometimes expected to be 0.

bedobi• 3 years ago

The problem is that anyone with a semblance of professional pride will find it mentally quite difficult to knowingly ship bad software.

Like, sure, even if we make best efforts to catch both obvious and less obvious bugs, all software we ship will still be full of bugs. But knowingly shipping software full of obvious bugs... it feels unprofessional. And makes you really feel the weight of all the inevitable bug reports that come in.

The alternative often becomes to warn about it, have the organization respond (whether explicitly or otherwise) "we don't care", and then, to preserve your sanity, say to yourself "well, then all bets are off, and I'm not responsible" which also isn't healthy, because it leaves you jaded, disengaged and robs you of your sense of professionalism.

I don't have a solution here lol other than find an org that cares about quality to an acceptable degree.

jonnycomputer3• 3 years ago

yeah. and not just "not professional". It's basically giving the finger to a whole bunch of (generally) lower-paid support staff answering phone calls from irate customers.

JohnFen3• 3 years ago

Not to mention it's giving the finger to the customers.

nickelbob3• 3 years ago

Perhaps one solution is to divorce your own self worth from the quality of the software you happen to write and to realize that you are an employee first and a software engineer second - make decisions that increase the amount of money the company makes and look for self worth in that itself or in hobbies/family/friends/something else.

braza• 3 years ago

That’s was the main reason that I do my pet projects: to detach myself from some corporate persona that is a completely different from my professional self.

I still have a lot of professional pride, but it comes more from my personal projects and sharpening my craft than to some corporate version that could be dragged down by structural issues.

int_19h3• 3 years ago

If one is going to go there, why not make decisions that increase the amount of money you make? Why help companies succeed shipping crappy software?

tru1ock• 3 years ago

You are not professional if you are not aligning with the values that are being optimized for the context. There should be an explicit decision tree setup for your work context that informs you what those values are. If not have some conversations and align.

distcs2• 3 years ago

Original title of this HN post was:

"Responding to “Are bugs and slow delivery ok?”: The blog post that I’ve hated the most, ever"

But if you scroll down to the end of the post:

So thank you, Valentina, for your thought-provoking post. I loved to hate it.

So the author does not actually hate the post they say in the title they hated.

samwillis• 3 years ago

Ultimately it all comes down to business case. If there isn't one to invest in CI, developer time to build a test suite with 100% coverage, and then ongoing upkeep of that, then it's not going to happen.

We are too often religious about these things, and may not have visibility within the business to see why there isn't a case for it.

Having said that it can also be very hard to convince management of the business case. That may have had a bad experience of these tool and practices before and see them as a waste. Thay may also, in that specific case, be right.

commandlinefan7• 3 years ago

I've been coding now for about 30 years. I've never moved up into management and, now at almost 50, it's pretty clear I never will, and will spend the remainder of my career coding (which I'm perfectly fine with).

In the 30 years I've been coding, languages have come and gone, platforms have come and gone, frameworks have come and gone, methodologies have come and gone, but two things have never changed:

1) Software development is wildly unpredictable. I've never met anybody who could reliably predict how long something would take to develop, even if the requirements were "frozen" and even if they were the person delivering it. I have met people who could predict ahead of time how long it would take them to deliver something, but what they delivered always had "bugs" (either actual incorrect behavior, or edge cases that nobody thought of until they actually used the software) that themselves took an unpredictable amount of time to fix.

2) No organization can ever accept any amount of unpredictability. Every "advance" in software development I've ever seen has claimed to (implicitly or explicitly) remove the inherent unpredictability in software development, although none have actually delivered. They're fine with spending months in planning meetings while not actually developing anything tangible as long as those planning meetings provide the illusion of predictability.

jonnycomputer3• 3 years ago

We are all at the mercy of Hofstadter's Law.

KronisLV2• 3 years ago

https://en.wikipedia.org/wiki/Hofstadter%27s_law

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.

qingcharles2• 3 years ago

Thank you for (1). I was trying to explain to someone the other day how unpredictable coding is, but you're right, it's wildly unpredictable. Even for tiny projects they sometimes go so far off the rails.

I remember working on this one major project - an online music store - and one of the execs hired his very lovely wife to project manage a team of developers. She would create these fantastic charts in Microsoft Project and every day try to somehow - somehow - get the developers to either (a) give her timescales for each component, or (b) explain why previous component X took twice as long as they had scheduled.

I've been writing code for 40 years and I still get caught out myself. I try to give realistic deadlines and then some fucking bug in the compiler or other such insanity will come along and make me look like a fool.

watt• 3 years ago

If the code you write is uncovering bugs in compiler, you literally are too smart for your own good. Fwiw, I have seen how compilers are made. Makes me amazed anything actually compiles and runs.

g4zj• 3 years ago

I've been writing code for 40 years and I still get caught out myself. I try to give realistic deadlines [...]

Call it immaturity, or perhaps just not being a "team player", but I've been coding for 15 years and stopped giving time estimates about 5 years ago.

I simply refuse, and instead, explain that I would just be making something up, and I'm not interested in taking responsibility for an arbitrary guess, since I know from experience how it can come back to haunt me.

I try to suggest saving such speculation for a time, hopefully in the near future, where the initial research has been completed and _some_ progress has been made on the foundation of the requested work, so that my speculation can have some basis in reality.

If all else fails, I give a wild estimate that I don't expect anyone to agree to, such as 2 years for work that might not even take 2 or 3 months.

joshspankit• 3 years ago

At some point I ran across this strategy for estimating when you’re the one doing it: Take whatever time you think it’ll take, then double it. Then add 20%*

Things still go off the rails of course making an an imperfect solution, but it ends up being more right than any other method I’ve tried.

*If there are any unknown unknowns I might start multiplying by the number of big ones.

embwbam2• 3 years ago

In my experience, one can accurately predict development time by correctly working around the planning fallacy.

The planning fallacy is something like "Humans are hopelessly optimistic. Even if you think you're planning for the worst case, unexpected things will happen, and your estimate is more like the best case scenario. This is true even if you try to compensate for the planning fallacy by being more pessimistic".

So you can't fix it by padding more weeks, trying to think of all the things that might happen, etc.

The only way to compensate for this is detailed in Kahneman's "Thinking Fast and Slow" (Must-read for many reasons). That is to start with prior evidence. Rather than making your estimate, start with "How long did it ACTUALLY take the last time someone did something like this?" and only then modifying the estimate for circumstances.

There is a huge gap between these two methods:

1. "How long does it take an average team to create a SAS product? Maybe a year?" Then modify for "but our team is extra smart, so maybe I'll shorten that to 10 months"

2. "How long will this take? It's not a complicated product, and our team is smart: maybe 4 weeks for feature X, 6 for feature Y, 2 to deploy. I'll even throw in an extra 4 weeks of padding for unknowns, so 16 weeks".

commandlinefan7• 3 years ago

How long did it ACTUALLY take

Haha, I came to that same conclusion decades ago - I don't know if it would actually produce meaningful estimates or not, though, because every time I try to estimate that way, somebody more "important" than me comes along and throws out that estimate for one that fits the timeline he already had in mind.

AnthonyMouse2• 3 years ago

As far as I can tell there are two major problems with this.

The first is that you can predict that there will be unforeseen problems, but not how many of them nor how long each will take to resolve. The result is high variance. Then if the average such project takes 16 months, and you predict each such project will take 16 months, then some take 7 months and some take 17 months and some take 25 months and people get mad at you for underestimating some of them. But if you predict at the upper end of the range then people will get mad at you for giving 25 month estimates for projects that only take 16 months on average.

The second is that people know what the estimate is and work expands to fill all available time. When you know you have 16 months, you set a pace that will have the project finished in 16 months. Then in month 14 you encounter some unexpected problem that takes six months to resolve. If you'd found it in month 2 you'd have been fine, because you'd have cut some less critical features and spent the time on that, but now that's in the past and your project is five months late.

arzke• 3 years ago

That is to start with prior evidence. Rather than making your estimate, start with "How long did it ACTUALLY take the last time someone did something like this?" and only then modifying the estimate for circumstances.

This requires to have prior evidence within your team to start with as not all teams have the same velocity.

ianmcgowan• 3 years ago

This is the only thing that ever works for me, but a) you have to be doing repeated projects or high overlap, and b) it's hard to defend when a PM challenges. My response is usually "<shrug> this is how long this kind of thing has taken in the past". I do a lot of the same projects at different customers, so my gut feel is usually pretty close (and I have to give a ballpark quote, so there's some penalty to being over or under). But if it's something completely new, breaking it down in to detail tasks almost never results in a good estimate.

ebiester• 3 years ago

Let me add more nuance here. We are always balancing a set of constraints: total cost, speed of delivery, certainty of cost, and certainty of delivery for a given date are but four.

Agile, for example, suggests that decreasing the cost of change at the expense of certainty of cost will increase the total chances of delivery within cost and date. Outsources are (implicitly) arguing that the decrease of total costs are worth the other friction in the organization.

This all goes back to return on investment. In the most simplistic case, if I have ten people working on this task now, and this software will allow me to reduce that to six, then that's four people I don't have to pay for that task. If those four people are making $100,000 a year, and the software lasts 10 years before having to be replaced, that's 4 million dollars. The software plus maintenance needs to cost less than 4 million dollars to make sense. We all know the simplistic case.

But if I can trade $100,000 dollars in guaranteed cost for a reduction of $500,000 in risk in making that goal, I can trade cost for certainty and that's going to be a good trade. Now, as developers we know that these numbers are often fudged, but everyone realizes this is inexact and just the best we all can do. We need to know how to think about these in terms of arguing against bad investments and steer toward good investments by knowing how the tradeoffs works.

Silhouette3• 3 years ago

We are always balancing a set of constraints: total cost, speed of delivery, certainty of cost, and certainty of delivery for a given date are but four.

The real problem is that management think they can directly constrain those kinds of variables but in reality they are not inputs they're outputs. The real inputs are things you can actually control like what you're building and which of your team you assign to build which parts of it at which times. That might come in on time and under budget or it might not but shortening a timeline or decreasing a budget won't change how long it takes or what it costs.

Of course that sucks for managers who want to make informed decisions like whether to go ahead with a project that will require an uncertain amount of investment to complete but needs to generate more return than some threshold to be worth doing. Now the managers have to find a way to guess what the important outputs like cost and timescales will be as some sort of probability curve so they can make a rational decision based on the best but still incomplete information they have available. And even then a lot of managers insist on trying to be absolute about things like time and money when the actionable decisions they need to make are more often whether to start some project at all or which ongoing projects currently have the greatest need for any additional resources that are available.

commandlinefan7• 3 years ago

Agile, for example, suggests that decreasing the cost of change at the expense of certainty of cost will increase the total chances of delivery within cost and date

Yes, that's the promise. I've seen it tried as earnestly as possible over the past 20 or so years since the term became popular, but I've never seen it deliver on its promises either. I suspect that a lot of that is psychological - risk-averse management hears "on time and on budget" and consciously or unconsciously starts steering the development process back to a more predictable seeming waterfall style that negates the supposed benefits of "agile" development.

I assert, however, based on 30 years of observation, that no matter how carefully you follow an agile delivery framework, whatever that may be, actual delivery and cost will have no relationship whatsoever on your estimated cost and date.

But I also predict that you and I will continue to smile and nod when they insist that it must be finished by such and such date.

sanitycheck3• 3 years ago

You can make some good guesses about levels of unpredictability based on previous experience though.

For example:

If the job involves dealing with anyone from Samsung or LG, it will take 2-3 times as long as you think.

If the job involves adding in a commercial third-party component which has a website about it, and that website doesn't clearly explain what the benefit of the thing actually is, the job will take 10-20 times as long as it should.

If the job involves a new feature Amazon or Google have just launched on one of their devices, and you are being CC'd on email chains involving representatives of Amazon or Google who are claiming implementation is very easy, the job will take a minimum of 6 months and may never be completed because the feature will be dropped or replaced with V2 before V1 ever actually works.

commandlinefan7• 3 years ago

the job will take 10-20 times as long as it should.

But your CTO will think it will take 10-20 times less time than it should, because that's what the vendor promised.

cameldrv2• 3 years ago

Software does not have to be wildly unpredictable if you do good estimation work up front. Being good at this is very important if you work for clients that have a specific budget or are doing fixed bids for software. The main thing is that you have to take estimation very seriously. You have to put experienced people on the estimation task, and you have to spend a lot of time on it to get it right, but doing it right can pay huge dividends. A few strategies that have worked very well for me:

1. When doing estimates, estimate hierarchically and break down every task until the leaf nodes are 4 hours or less. If the leaf nodes are a week of effort, that means that you don't yet understand fully what the task entails. If you don't know what the task entails, it is very difficult to get an accurate estimate.

2. Maintain a database of various projects and features, and use them for comparison purposes. If you know how long something actually took in the past, you can much more accurately how long it will take to do something similar again.

3. Include specific time for various types of "overhead": meetings, code review, writing tests, debugging, making presentations, etc.

4. Derisk ahead of time things that are truly unpredictable. Doing something in a brand new programing language or framework, training an ML model on a new type of data, working on datasets of unknown quality, etc. For those you need to estimate out a project to determine feasibility and generate an estimate.

The main thing is that this is clearly not impossible. Years ago I got really interested in the Apollo program as an example of bringing in perhaps the most complex and risk-laden project ever attempted, on time. One of the things that clearly stands out about that project was that there was huge derisking happening first. At one level you had the Mercury and Gemini programs that were figuring out human spaceflight generally, and docking with Gemini, and also things like the first stage engines of the Saturn V were already in development in the mid fifties.

Palomides• 3 years ago

splitting a project into an open ended "exploratory phase" of undefined length followed by a "main phase" of defined length doesn't seem like it would make any change to the total time taken, just shifting around the unpredictability

commandlinefan7• 3 years ago

huge derisking happening first

How long did that huge derisking take? What was the budget for the time spent doing the derisking? I'm willing to bet the "huge derisking" ended up taking longer than the actual project (and was itself open ended and unpredictable).

tshaddox• 3 years ago

Software does not have to be wildly unpredictable if you do good estimation work up front.

You also need someone with 1) strong technical and/or product skills, 2) good communication skills and decisiveness, and 3) explicit executive authority. That's the only way to avoid scope creep and bikeshedding.

When you started working on a project without knowing what you're building, or end up delivering a very different project than what was originally planned, it's hard to even say "that project was late." No it wasn't: you just never built whatever you had promised to build by the original deadline.

vegetablepotpie2• 3 years ago

Everything you describe works well for estimating tasks that can be automated.

If you’re doing that kind of work, your employer will be happy to replace you with an AI system.

tomxor• 3 years ago

but what they delivered always had "bugs" (either actual incorrect behavior, or edge cases that nobody thought of until they actually used the software)

This is why they are not only inevitable but unending.

Most real world problems are so large and messy that they do not have closed form solutions. In this context "bug" more often means a deficiency, a reflections of changing or competing and incompatible requirements, than abstract mistakes.

In this context, I've learned to appreciate the choice becomes "which bugs", because "fixing" such bugs is not necessarily a net positive, especially when the fix is not a natural fit for the design or architecture of the overarching solution. The mindless, incessant fixing of such bugs on a case-by-case basis can result in accumulation of so much complexity and LoC it becomes self perpetuating in bug fixing alone. When you are conscious of this, and have enough respect and autonomy, it's possible to decide whether it's worth this intangible cost, or whether to put it into the pile of "wont fix", this kind of decision requires cross domain collaboration. But that pile is not worthless, you can periodically revisit it and see opportunities to a fundamentally different approach that may tick more boxes (or more preferable boxes) without being ad-hock.

Escapado• 3 years ago

And even though the points raised are mostly valid I think there is a lot of nuance to this. If you write control software for medical devices this is simply not true. Same goes if you handle incredibly sensitive data or critical infrastructure or assistance systems for air travel and you can probably come up with a a bunch more cases where bugs are not ok. And also I think this sentence

Where high quality is nice to have, but is not the be-all-and-end-all.

while being mostly true in all other cases I don't think it does it justice what mediocrity actually means for everyone involved. I have seen very varying levels of quality requirements/enforcement, testing, delivery speed and eventual bugs at different costs at different companies. And I in general found that higher quality software and less bugs were associated with much happier and more productive developers. And I would make the argument that less happy developers => more turnover => more second order costs. So while it might not make or break the company it certainly has a negative impact on profits downstream. And for customers too. Vendor lock in is a great thing for a company to profit from but if you really make a shitty product it opens up a lot of venues for competitors to eventually canibalize your market or from users to jump ship the next time they can. Take MS Teams for example. Lots of lock in but trust me the second better competitors are on the table I vigorously fight to switch. It's a slow burn but a burn nonetheless.

jonnycomputer3• 3 years ago

The classic case study.

https://en.wikipedia.org/wiki/Therac-25

drewcoo2• 3 years ago

The examples you give have regulations for software. Even if companies want to "move fast and break things," with medical devices there are required tests and the testing and results are audited by the feds (here in the US), so you're forced to move slowly and not be broken (in ways that are regulated).

That does not mean "high quality." You can still have an architecture that's tightly-coupled spaghetti code that resists all human efforts at debugging. It just means "we threw enough paperwork at the auditors that they're not asking for more."

jbreckmckye2• 3 years ago

I have, for the record, seen companies lose market share, commercial opportunities, and once have to go through financial restructuring due to missing product goals.

Technical debt / quality / boneheadedness was at least a factor in each case.

I've also worked in financial services where a bad enough bug can and will lose you your (jurisdiction-appropriate) license to to do business.

hyperhello• 3 years ago

The core of this argument is really about the emotional health of the workers.

An emotionally healthy person tends to want to maintain their environment. It doesn't do anything objective to have nice furniture and pretty paintings, but you like it in your house. You want to feel at peace, so you adjust little things...the kitchen would be nicer with the plates in a pile over here instead of just pulling them out of the dishwasher.

At work, you're the householder of a million technical details, and to feel good about it you want to maintain your environment! If that window has an ugly redraw you want to recode it, if the ribbon has big ugly icons you want to collapse it into a strip menu, if the email client freezes whenever WiFi gets spotty you get annoyed because the code is still from 1995 under the surface. The software you're making and using, in service to your own craft, has an effect on you. If it makes you feel ugly, you want to fix it. This blog post -- which probably wasn't written on company time -- is here because someone needed to reorganize their thinking to feel emotional health. I think we should insist on doing a better job than just sacrificing our work to feed the money machine, even if our self-described jobs are just to turn funding into paychecks.

gaze• 3 years ago

Precisely. I get not going overboard and being a perfectionist. One has to ship. But also, you should have SOME pride in your work. You should want to not make total shit.

switchbak• 3 years ago

This does seem like a bit of a fishing article, but I'll bite.

I've personally been involved in 2-3 software projects where the immediate ramifications related to software quality and delivery led to project failure.

The idea that bad (or no) architecture and terrible code won't have business impacts is just wrong. Perhaps he was lucky, perhaps we have collective survivorship bias, but I strongly disagree on the main premise here.

In one case, some bad decisions related to API design and customer extensibility led to a product that was simply impossible to change or evolve. That company did some custom work for a few clients on a shared framework, but after ~5-8 customers, the product was basically unchangeable. Some of the customers left, but we still couldn't change it, and the result was pretty predictable.

That you can limp along with terrible practices doesn't mean that better practices wouldn't improve the situation.

ye-olde-sysrq• 3 years ago

I think there's a few things at play when people talk about "software quality".

Some people hear it and think it means highly-structured abstraction towers vs low-structure imperative-heavy code. Some think about horrible spaghetti with 16 levels of indentation and 1-letter variable names (moving on to 2-letter once you exhaust the first 26) and if-statements whose predicates span twelve lines. Some people think about feature completeness and bugginess (or, rather, lack thereof).

I think they're dangerous over different time horizons and you can't just prioritize over all time horizons.

Spaghetti is dangerous in the short-term - it makes small changes difficult and unsafe. You'll end up tied by your own shoestrings and not even finish the project. Luckily if this is the only problem, it's also the easiest to clean up. (But it's not usually the only problem).

Less-structured code is good for when you don't understand your requirements well. (Which you typically never will a-priori, even if you think you do). It's easier to add more layers of abstraction than it is to change or remove them. If you prematurely build an abstraction tower, you will suddenly find yourself having to plumb information up and down the entire tower as the abstractions prove to leak or not be completely perpendicular to the kinds of features you need to add. But if you wait too long, you have too much unstructured code to refactor and you're now "chasing" people adding new features to the system while you try to refactor the system underneath them. So there's just some engineering gut-sense that's important to know when to stop and codify the patterns you've seen emerge into a useful, tight abstraction.

And then of course there's features and bugginess. It's possible for the shittiest code to have very few bugs and please users. The worst thing about the worst code I've ever seen in my career is that it worked - so it was hard to want to mess with it when it was so easy to ignore! And it doesn't help that what is intractable code to some people is only a little complicated to others. Especially if you use a big language and are combining a lot of esoteric/complex features, or have a complicated logical expression and you are a logician and others are not.

I've seen projects become unchangeable because the code was spaghetti. I've seen them become unchangeable because there was too much abstraction. I've seen them become unchangeable because there was too little abstraction. I've seen awful code make big impact and I've seen excellent code flop because it wasn't competitive with "worse" but more complete offerings.

(All of this reinforcing, of course, that it's impossible to estimate how long software will take..... all of it relies far too heavily on the intuition of your mid+senior engineering talent).

akasakahakada• 3 years ago

Bug is ok, if you afford to crash every Mars rovers onto the surface, or fucked up every ignition moment and blow up every space shuttles, etc.

If you don't care about anything that you should be care about, of course nothing will be an issue.

littlestymaar• 3 years ago

I got really surprised when I read the “original blog post” because it's less a blog post than is it some bland advertisement to “some online coaching that will make you a 10x engineer”. So when I see the OP qualifying as ”short and well-written, well worth the read“, I suspect all this is just a PR stun to attract attention to Cupać's paid online training program… (and this seems to be working pretty well since it's now on the top of the front page)

srott• 3 years ago

Working rapidly without bugs is preferable to working slowly with bugs.

dasil003• 3 years ago

The author's insight is fair on a personal level, but the thesis is not very useful as a generalization. The reality is that bugginess and velocity are all relative, and every situation is different. It's good to understand the leadership viewpoint and be realistic about the business that employs you so you can decide if it fits your values and look for a change if not. If you care a lot about the quality of what you produce, there are plenty of companies that need and value that. On the other hand, many people just want a steady job where expectations are clear and ambiguity isn't too high; mediocrity is one way to achieve this, and it can work fine (for a while at least) with sufficient moats.

spacebanana7• 3 years ago

Most enterprises that fail do so because they run out of money. In software business where the dominant cost is engineering salaries [1], poor tech practices with expensive fixes are a major factor in running out of money.

More abstractly, high productivity engineers help you to exhaust the idea space through delivering products into real world before you run out of money. It's better to fail because you discovered your supposed gold mine had no gold rather because you ran of mining resources half way through.

[1] Notably this doesn't apply to capital intensive business with high marketing or infrastructure costs

WaitWaitWha• 3 years ago

No, both this and the original blog entry are myopic.

There have been companies that went out of business, not only the software publisher but companies that relied on such software.

If the assertion is that "known unknown" bugs exist in code, and the publishers are aware that there might be bugs but in their QA/QC could not identify them, I can accept it. But, if the assertion is that there are "known known" bugs exists, I find it bordering unethical. The only way I can accept this if the publishers informs the buyer, and let them know what is the impact, the plan of action to remediate or mitigate the bug, and in what time frame.

"Known known bugs" (the publisher knew about the bug for extended period of time, and never informed the users) have destroyed multitude of businesses.

wokwokwok3• 3 years ago

So, here’s a simple game:

- take a blank piece of paper (this is “the software”)

- pick two random points on the paper roughly 3 inches apart (“a requirement”)

- draw a line between the two points; the line cannot cross any other line (“the implementation”).

Repeat the exercise multiple times.

You’ll quickly learn that unless you have a “system”, drawing a three inch line goes from taking about 2 seconds, to taking 10, 15 or 30 seconds, despite “the requirement complexity” being exactly the same every time.

Now try playing with people taking turns. :)

I like playing this game with people who think that writing bad code is fine, or that they can work by themselves and not worry about what other people do as long as “their own” code is good.

You can still solve pretty much any problem with enough time and effort; but if you don’t have a “system” for good organisation, eventually, you’ll be struggling to solve basic problems because of layered unmanageable requirements, no matter how smart or capable you are.

…it’s not about shipping bugs, it’s about fundamentally reducing the speed of delivery over time, incrementally, in an unbounded fashion that eventually makes systems impossible to make changes to without compromising on requirements.

(Obviously the game is contrived, but it works very well in explaining why past requirements some times have to go away to business people in my experience).

newaccount74• 3 years ago

In practice, the implementations will end up crossing a few times anyway (bugs) and noone cares much because software bugs are a part of life.

Speed only goes to zero if you insist on bug free software.

idbehold2• 3 years ago

What "system" would you use for random "requirements" (dots on a piece of paper)? The only reasonable thing to do is to push back on "requirements" that don't fit into the existing requirements/software/implementation.

wokwokwok3• 3 years ago

The thing about this game is that there is no requirement that is technically impossible.

That's provable; with no closed loops, any pair of points on the paper can be joined by a long enough line. That's why it's a good analogy for code.

...but if you, eg. a system for this; Use straight lines only and space the lines equidistant from other lines.

I mean, it's a game. Literally play the game a few times (grab a friend and do it together) and you'll very quickly figure out things that do work (leaving space) and dont work (draw crazy squiggly lines and lines that are too close together).

This is just a contrived example; if you want to make it more realistic allow 'you can erase a line and redraw it as long as the requirement is maintained', and then you can structure lines into groups, etc.

II2II• 3 years ago

Software doesn’t make or break an organization.

As someone who works for an organization that is on the receiving end of bad software, I can assure you this is not the case. There is lost opportunity, due to decreased productivity and outright lost clients. There is a souring of the work environment, an environment where people are constantly putting out fires and there is constant churn in staff because of that.

Perhaps the organization I work for is unique in that it is public sector and cannot go out of business (which appears to be the author's definition of broken). Perhaps the author has never dealt with software where edge cases involving lost and phantom data is a daily norm. Clients, real people with real lives, are literally lost by the system.

Things are so bad that losing several years of business records (or, more likely, paying to have them maintained by a retired system for some legally defined period) is considered as a viable option.

So yes, buggy software does break organizations.

wellpast2• 3 years ago

This is called software philistinism.

There are many blog posts like this one that deny (often deride) the value of software craftmanship, including the skillset of planning.

They almost always have this same conclusiveness to them as well ("she was right"), with no tangible backing other than personal anecdata ("everyone sucked who I worked with, so everyone must suck").

If you zoom out, the reality is that people in the software industry are spoiled AF. A majority of practitioners get paid a ridiculous amount and get away with murder.

It is true that current market dynamics give people in this industry pretty much zero incentive to get good at the craft. So people don't and then to rhetorically justify their philistinism, they write blog posts like this.

It is also true that many businesses get away with writing absolutely shitty software and stay in business because of these dynamics. But that doesn't make it "ok" and doesn't say anything about the enormous amount lost to such apathy.

int_19h3• 3 years ago

I don't think the blog post derides the value. Rather, it points out that, while fixing bugs is valuable to the dev (professional self-respect shipping quality products) and the user (obvious), the way software market works in practice is that it's not particularly valuable to the company - at least, not valuable enough to justify the expense. Speaking as someone with 20 years of software engineering experience, I think this is an accurate assessment of things. But at the same time you're absolutely correct that it doesn't make it okay, and it is actually detrimental to both users and developers for the aforementioned reasons.

But, well, this is capitalism for you - whatever is the most profitable thing to do is what gets done regardless of how not okay it really is. I think the question we should be asking is: if users hate using crappy buggy software, and devs hate writing it, but it still gets written, aren't our basic economic incentives clearly out of tune?

wellpast2• 3 years ago

Well said — and yes.

It’s definitely market dynamics, but I think something could change those market dynamics.

Maybe a craftsmen’s guild that unionizes skilled software practitioners.

JohnFen3• 3 years ago

Bugs are never OK. Slow delivery, though? That depends on what you mean by "slow delivery". I happen to think that the trend of rapid release has been pretty awful for software product and service quality, so if "slow delivery" is the absence of that, I think it's desirable.

Bugs are inevitable, but that doesn't make them OK.

But that's not the point the article is making. They're talking about whether or not bugs and slow delivery will break a company. Obviously it won't, because both have been a part of what the industry releases since the start.

_benj• 3 years ago

Well great, I hate it!

Yesterday somebody shared this with me and it made me reflect about what things could we learn from other engineering disciplines. It might be just me but I have this gnawing impression that a lot of the pains we have in software are self imposed.

https://web.cs.wpi.edu/~gogo/humor/hum_toast.html

senko• 3 years ago

"Software quality" is such ambigously defined term that all of these can simultaneously be true (for different products):

* software quality is (more) important

* software quality is not (very) important

* bugs are not important (aka "i don't test, but when I do, it's in production")

* the less bugs you have the better, but they're not gonna kill you

* bugs are safety-critical

* slow delivery is fine

* slow delivery is not fine

In the last 20 or so years working as software developer professionaly, in various teams and quite different projects, I've noticed that:

* people (decision makers, software buyers/clients) always say they want higher quality

* same people very rarely want to back that up with their budget

* people almost always prefer more features to more quality (even when explicitly told that more feature is not a good thing for eg. MVP stage of a startup)

* most people except dedicated QA engineers can only think of a happy path (if everything works the ideal way); this includes not only other engineers (who try to think up failure scenarios but often miss important ones), but also designers (when have you ever seen a design mockup for "DNS lookup failure connecting to the server" case?), and definitely product owners or clients (in context of a dev agency)

* I have never worked on a nontrivial software project that didn't change requirements/scope

Myself, as with many other software developers who chose this field out of passion and not stratospheric wages, was for a long time appalled by this apparent lax approach to quality? Don't you want to have the best possible product?

But working with a lot of diverse clients while doing dev agency work, and running several startups myself, has taught me that software quality is not, and (it pains me to write this) should not be, the top priority. Product-market fit, treating your customers well, having a sustainable monetization strategy, marketing and sales ... if you don't execute well on those, nobody will notice the (lack of) quality[0].

In most organizations (except the ones that are swimming in cash) it's a tough act to balance all of the above. While I wish for all the projects I work on to be the best they can be, a programming work of art if you will, I can certainly empathize for people who must prioritize otherwise.

[0] unless the quality is safety critical, in case I need to point this out explicitly.

AnthonyMouse2• 3 years ago

I disagree with this conclusion because it's taking as implicit the assumption that the organization is the thing to be victorious, which is a misunderstanding of organizational politics.

There is no organization. There is only people. Some of those people, like your boss or your investors, may want you to burn yourself out or do unethical things to your customers so they can make more money. It's your prerogative to tell them to fuck off, and go work somewhere else if you don't get your way. Some people don't have that capacity. Software developers generally do.

Perverse organizational incentives will pressure you to be bad. Refuse.

clintmcmahon• 3 years ago

Maybe this is my years of experience talking, but curious if anyone else sees a connection between a slow delivery (aka slower development time) and less bugs? The idea is that developers have more time to think about the solution, write better code and overall not feel so rushed just to get something out the door. In my experience, the more pressure put on developers to get the thing done, the more bugs are introduced.

Silhouette3• 3 years ago

curious if anyone else sees a connection between a slow delivery (aka slower development time) and less bugs?

I certainly see a connection (several actually) between having more bugs and following development processes that emphasise metrics like time between starting work on a job and running the code in production. The advocates of Accelerate and the annual Google DORA reports seem to have taken over the web development asylum lately but the underlying arguments and data are still dubious at best.

macinjosh• 3 years ago

The meme with the men at the gallows, nooses around their necks and one looking at the other saying "First time?" comes to mind. That is pretty much software development in a nutshell.

thanatropism• 3 years ago

Wise men from a prior generation said: worse is better.

esafak2• 3 years ago

https://en.wikipedia.org/wiki/Worse_is_better

phendrenad2• 3 years ago

I hate to be that guy, but this is an artifact of ZIRP. Watch what happens when interest rates go up and people start demanding more value for their SaaS subscription. It'll be a return to software development in the 1980s when most of our current development processes were developed (which most companies talk about but don't follow). Most companies aren't gonna make it out of their technical debt holes...

tikhonj• 3 years ago

Software doesn’t make or break an organization.

This is true... but it's also true of ≈everything else in the organization. An organization that could be broken by any single thing would not survive for long! This includes the things people tediously insist to be "more important" than software quality. It certainly includes the usual suspects that push people to cut software corners: companies don't break by missing OKRs or slipping schedules or not "focusing" or any other management pablum any more than they break due to software quality.

But, of course, all of those can contribute to a company failing—including bugs, slow delivery and poor quality code. The dynamics and the extent to which different factors matter is inherently context-specific.

Companies can fail despite doing the right thing and succeed despite doing the wrong thing. This doesn't make the wrong thing effective or reasonable, it just means we're operating in the domain of complex, stochastic systems, so our simple mental models of cause and effect become misleading.

The fact that low-quality software doesn't directly cause a company to fail does not imply that investing in software quality is not right. I am firmly convinced that maintaining high quality software will have better outcomes in expectation than cutting corners pretty much universally, if only because it simply doesn't cost much—it's a matter of setting the right culture more than anything else.

There are interesting observations to be made here. Variations on "software quality doesn't matter because it won't destroy the company" are not it.

nickelbob3• 3 years ago

This is true... but it's also true of ≈everything else in the organization.

Except for sales?

aidenn0• 3 years ago

It's hard to turn "bug free" into a selling point. Nobody has "bug free" as a checkbox on their decision matrix.

CKMo• 3 years ago

Bugs in cybersecurity products are not okay at all. We call those zero-days.

Responding to “Are bugs and slow delivery ok?”

Related Stories