Of Debt and Decisions

http://www.monkeysnatchbanana.com/2015/04/19/of-debt-and-decisions/

by spooneybarger • 11 years ago

53 17 11 years ago

I love this story. It offers an important reminder about how our industry really works.

Whenever a practice becomes established as a "best practice", such as automated testing, then the next question that arises is "Why isn't everyone doing this to the max?" And folks that don't follow best practice generally feel guilty and keep quiet. But those who follow fashions in tech tend to underestimate the opportunity costs of doing so.

This is the moral of the above story, and it is a good one:

"Hopefully you are horrified by all things we didn't do that are now considered best practices. No reproducible builds, lack of automated testing, creating and then repeatedly sidestepping technical debt. For a long time, that is how I told these stories: as horror stories that came complete with hand gesticulations to emphasize the "can you believe I had to do that?" nature of it all. Here's the thing though: we were making money. We were growing and supporting ourselves."

I spent this winter working for a startup that had the opposite problem of most startups. The CTO was a serious Pythonista, and he wanted to ensure that all the code was written was of high quality. He wanted to avoid the build up of technical debt. There were 9 programmers on the team, using Python on the backend and AngularJS on the frontend. And he felt they could invent a new way forward, in terms of combining Python and Angular. And so the team was reinventing the whole Python stack: they had their own serialization libraries, their own database libraries, their own web server, their own authentication system. Adding each new feature lead to very long meetings in which people were encouraged to be thoughtful -- everyone was suppose to think about the long term consequences of whatever design was chosen. In some ways, it was a careful style, but it had a cost: the 9 developers spent more than 50% of their time on infrastructure questions that had already been solved elsewhere. In terms of meeting the real needs of this small startup, the startup might has well have had a team of 4 developers -- the money paid to 5 of the developers was, in a sense, an act of charity being offered to the Python community.

We've all ready endless stories about startups that cut corners, ran up technical debt, and eventually succeeded. We read much less often about startups that doing everything perfectly and then become successful. And that suggests that, for startups, cutting corners is the more adaptive path.

fsk• 11 years ago

I've seen that a couple of times, startups that insisted on rolling their own code for everything instead of using standard tools.

I worked one place that wrote their own build tool (instead of using Make or Visual Studio). They wrote their own templating engine, instead of using PHP or ASP. They had their own version of MFC and String library, with everything implemented slightly weird, so I had look at the source whenever I needed to know how it actually worked. It was nuts. They spent a year writing their own tools for everything, which was a year they didn't spend working on the product.

kyllo• 11 years ago

I know of a non-software industry company that for a brief time seriously considered implementing its own proprietary relational database engine. Thank god that project got killed.

sukilot• 11 years ago

Your CTO was a different case. He wanted to reinvent the wheel, which is different from avoiding technical debt.

aioprisan• 11 years ago

This "Yeah we had a lot of technical debt but we kept finding inventive ways to sidestep having to pay it. The debt metaphor that we’ve adopted is misleading. You don’t always have pay off the debt you created; you can work around it and avoid ever having to pay it off. Technical debt only hurts if you have to pay it back."

afarrell• 11 years ago

http://www.higherorderlogic.com/2010/07/bad-code-isnt-techni...

zby• 11 years ago

For me the moral from that story is that in a mature market there is a lot of accreted wisdom and best practices - but in the early stages you don't have that and you need to trudge along with whatever you have. And judging those early projects through the lenses of the learned wisdom is silly.

Illniyar• 11 years ago

"Technical debt only hurts if you have to pay it back"

To continue the "debt" metaphor, when your product dies, it debt dies with it (or if you switch technologies, you might get a 'debt relief').

So for the thousands of startups and products that disappear or get pivoted before they can mature - it might be worth it to bet on not paying technical debt.

gwern• 11 years ago

Perhaps you could call that 'technical bankruptcy', as in, "we're declaring technical bankruptcy and writing an entirely new version".

decwakeboarder• 11 years ago

"technical write off" is probably a better term.

bcg1• 11 years ago

Might be more like a "technical jubilee"

benjamincburns• 11 years ago

or if you switch technologies, you might get a 'debt relief'

There's no free lunch here. When you do this you either pay off some of the debt by removing some value from the product, or you spend ages just reaching "value pairity" at which point you've likely accrued a whole host of shiny new technical debt.

Per the top commenter (jewel), I have to agree that this article misses the most important chunk of the metaphor - interest. Any time you have to work around limits imposed by architecture/technology/coding decisions, you're making an interest payment. Worse yet, side-stepping like the author describes is often like paying off one credit card with another - the premium compounds which dials up the interest payments. All of these sorts of things distract from delivering value and serve to further increase scheduling risk.

Point being, the author is correct that the case of having to make a large pay-off in order to move forward is quite rare. However, that doesn't mean that you should disregard good project planning and discipline. As you point out, there are cases when you should take technical debt deliberately as part of some tactical decision such as when you are just getting an MVP together, or when you need to deliver immediately in order to make payroll. However every project accrues technical debt which was not deliberate as well.

Regardless of the cause, you'll find that just like real loans, payments against the principle amortize best when they're made early. That said, it doesn't make sense to pay off a loan when doing so might cause you to miss out on some bigger opportunity, like a huge sale, or perhaps simply not starving.

ExpiredLink• 11 years ago

tl;dr?

bobbles• 11 years ago

Technical debt is only worth caring about if you have to 'pay' for it one day

jewel• 11 years ago

This article addresses one problem with technical debt, the idea that one day the debt will "come due" and you'll have to fix some latent problem in a hurry and very painfully. While that can happen, a lot of the time you can carry the debt forward, as the article illustrates well.

The bigger problem with technical debt is that it limits or slows your ability to do things. With real debt you have to make monthly payments and that reduces the total amount of cash you have to spend, which reduces your flexibility. With technical debt you "make a payment" whenever the speed you can make changes is reduced due to bad technical decisions of the past.

For example, if you don't have tests then you can still change the code, but you have to spend a great deal longer doing manual testing. When the inevitable bug slips though, you have to spend the time to fix it. Since it's often not catastrophic, non-technical management will be willing to pay a known ongoing cost instead of an large, unknown one-time cost to remove the problem altogether.

At my last job we were so mired in technical debt that progress had slowed to a near standstill. It was a 20-year-old codebase running on an IBM AS/400, mostly written in RPG (which is sort of like assembly language but for databases.) Field and table names were limited to 11 characters. When they migrated the product to the web, they kept the RPG code but added in a bunch of perl. When DB2 couldn't keep up, they started adding new data to MySQL.

In all, there were 500 thousand lines of code in the system. Adding a field to the most important table was a 20 hour process, including a late night or two to issue some commands while the system was doing its nightly maintenance. There were five programmers while I was there, and the majority of our time was spend keeping the system going. There was never time allocated for cleaning up the existing system, so instead we'd just try and add features without making any schema changes (or adding fields in MySQL instead, as I mentioned before).

ajtaylor• 11 years ago

You beat me to it. While you can "avoid" the debt by working around it, you're still paying by having to implement the workarounds rather than being able to do it in a more straightforward manner. All you're doing is swapping interest costs with opportunity costs. The piper has to be paid, one way or the other.

Of Debt and Decisions

Related Stories