The Birth of Legacy Software – How Change Aversion Feeds on Itself

https://software.rajivprab.com/2019/11/25/the-birth-of-legacy-software-how-change-aversion-feeds-on-itself/

by whack • 7 years ago

113 66 7 years ago

software.rajivprab.com

vearwhershuh• 7 years ago

I've seen plenty of projects blown up with massive, fearless refactors to do thing "the right way".

Careful with that axe, Eugene.

krnlpnc• 7 years ago

For real. Software development can be like building and maintaining a house.

Except none of the contractors agree on what materials to use. So one section is steel, another is wood, and another yet is brick. Meanwhile there is a 3rd party outside attempting to load the whole place onto a truck and ship it somewhere else.

tsimionescu2• 7 years ago

I think that the problem is most often something along the lines of: we have a house, it took 5 years to build and the family is very happy with it. But now, they also want a chip fab, and there are many common parts useful to both, so we don't think it's worth it to start from scratch - we need to modify the home to also house their chip fab, but make sure it's also still a good house to live in.

benjiweber• 7 years ago

Is a massive refactoring an oxymoron?

'a series of small behavior-preserving transformations, each of which "too small to be worth doing"' https://martinfowler.com/books/refactoring.html

arkh• 7 years ago

The solution from the book mentioned in the article (one of the must-read books for devs IMO) is to have tests for your functionalities. Those kind of tests will help a lot when refactoring.

cjfd• 7 years ago

One thing to watch out for, I think, is people who want to do this massive refactor (an oxymoron as another comment notes) and also have a work history consisting entirely of short employments. These people never get to experience the actual result of their refactoring.

hinkley4• 7 years ago

Anyone who tells you they’re doing a massive refactor is full of shit. The point at which you should stop trusting them is the moment that phrase escapes their lips or is used to describe them.

And I say that as someone who has committed a few, in both senses of the word.

The point is to be deterministic, and it can only be deterministic if it is small.

seren2• 7 years ago

In my open office next door, people have been refactoring a cpp98 monolith into more interdependent components to be able to have a better test suite, better CI integration, better deployment story. That sounds about right.

Well, the issue is that they have done it a bit sneakily, they removed all the legacy code they haven't understood. So the code is much more elegant, it has been moved to cpp11 or 14, it ticks every good practice. There is only one slight issue : it does not work. It somewhat work, but is not reliable and fails regularly in unexpected ways. And they've started 5 years ago, and haven't been delivering any business value since then.

At the beginning, it was okay because they had some leeway but now they are blocking the release of new products, and our market share is in free fall.

Heads have started to roll.

To be fair, a few years down the line, their team will likely be more productive and efficient, but I am still not sure that the cost of the rewrite was justified. Still the article is very on point on the risk of not paying your technical debt.

rusticpenn5• 7 years ago

The legacy code should have been running in parrallel with the new code , atleast until it reaches feature parity ( or reduced features - with other features removed as they have not been used)

Bombthecat• 7 years ago

I learned from Microsoft that rewrites are hard! (they tried it with word) and I think somewhere is the story how it went.. They rolled back later on and kept modernizing piece by piece now.

That, I think is the right way.

adrianN2• 7 years ago

Deleting legacy code you don't understand is the best way to remove fixes for bugs you didn't know you had.

goto114• 7 years ago

Deleting all code you don't understand is also stretching the term "refactoring" a bit beyond its usual meaning :-)

matwood2• 7 years ago

they removed all the legacy code they haven't understood.

This is exactly why rewrites or huge refactors typically fail. The new programmer sees code and doesn't understand why the code is there and thinks the last programmer was an idiot and deletes said code. Unknown to the new programmer is that code handles some weird edge case.

A rewrite should spend 80% of the time understanding the old code and 20% writing the new. But, that's no fun for most programmers who just want to code in the latest shiny so the new code ends up broken.

pjmlp• 7 years ago

Several years of wasted development effort have I seen.

hinkley4• 7 years ago

It doesn’t have to be that way. I’ve worked on plenty of code where the code at the end is barely the code we started with. Ship of Theseus style.

Bootwizard• 7 years ago

I quit my last job because my team was in this room spiral. And management's solution to the problem? Work harder.

As far as I know, everyone else still works there. I bet some of them sleep at their desks trying to keep adding more knots to that awful legacy system.

And it didn't get that way WHILE I was there. That project had been around since 1998. They were so afraid of change that they were still using the C++98 compiler. So I didn't even leave that job with relevant (read modern) C++ knowledge.

Funny thing: The server architect at my current job was one of the founding members on that project. He had the same kind of view towards it. I feel like that project has been in a doom spiral from the beginning. So 21 years at this point.

It blows my mind that there are these legacy code sweat shops out there barely holding software together like this. What a miserable existence.

romwell2• 7 years ago

And it didn't get that way WHILE I was there. That project had been around since 1998. They were so afraid of change that they were still using the C++98 compiler. So I didn't even leave that job with relevant (read modern) C++ knowledge.

This happened to me too!

I understand the reasons for that: it was highly specialized (scientific) software used in computer chip manufacturing. Most of the people on the team had a PhD in Engineering or Science, but few had extensive Computer Science training before joining the company.

The hardware industry is moving slowly, because you don't just upgrade a fab. Some of the clients were on RH5, so that was our build target.

Part of the codebase was in Fortran (and for good reasons too; this language is common in HPC/scientific scene).

All this resulted in a codebase that was part brilliant, part byzantine, poorly documented, and not even compiling with a modern C++ compiler (hence reluctance to start using C++11 or newer).

Thankfully, the automated test suite was holding it all together; but that was about it. As people left, they took systems knowledge with them that was written down nowhere, and nobody wanted to do a deep dive and document the still known parts.

Predictably, the priority has shifted to "quality" - that is, fixing bugs instead of innovating on the core functionality.

That was not the only reasons why I left for better opportunities, but it was a big factor. It wasn't the legacy system that was scary - no more scary than a hairy prototype that gets the job done, really - it's that nobody was going to put in the effort and take the risk to start moving from a decades-old "prototype" stage.

I believe that things finally started moving when the people there realized that there is no way but forward; I hope they are using modern C++ now. But that train was set in motion after I left.

rocqua• 7 years ago

This sounds a lot like the stories I have heard coming from ASML. Especially the part of people with a PhD in engineering writing code.

BlueTemplar5• 7 years ago

Isn't "modern C++" pretty much an oxymoron, and pretty much follows the above definition of "legacy software" ?

anovikov• 7 years ago

On the good side of things, their business must be doing really great because they have money to support that monstrosity, and have motivation to, meaning: it gives them a lot of cash.

mwilcox• 7 years ago

I think the best thing to do is use the existing system as a really, extremely detailed interactive requirements doc for a complete rewrite with modern practices and good documentation, then run both in production until you can switch the old one off

try_again2• 7 years ago

Complete rewrites seldom tend to work out well. Using a current version's behaviour as a "requirements doc" rather than the code or an actual up-to-date document probably misses out on a good deal of functionality that is rarely used but critical in some conditions. The thing about legacy software is that the software has encoded in it years or even decades of changes to functional and technical requirements, workarounds for edge cases that weren't foreseen, bug fixes, optimizations. Each of those can appear like a mess to the observer but it is foolish to think a new system will not run into similar issues. The idea of starting over to "do it right" is a fallacy because perfection does not exist. This is why the software industry is increasingly focused on processes that reap the benefits of quick turnaround and making change easier to deal with.

Believe me when I say the day you switch the old system off will never come in the vast majority of scenarios like this. The result is either wasted development time or, probably worse, you now have two systems to maintain and keep running.

rusticpenn5• 7 years ago

That is basically an example of sunk cost fallacy. Yes, those bug fixes are important, but its easier to find them and fix them again in a tool with proper architechture. The idea is to run the original and the redesigned code parrallely and find the differences in behaviour and fix them.

We recently moved from a heap of matlab code which was started with a student project in 2001 to a huge tool used in the industry today to a new implementation in C++ and python. It has been a huge sucess with our customers.

collyw2• 7 years ago

So go from a well tested production system to a completely untested (in the real world) system?

adrianN2• 7 years ago

I believe this type of rewrite is a lot less risky that what you're suggesting: https://martinfowler.com/bliki/StranglerFigApplication.html

goto114• 7 years ago

But why not just write good documentation for the existing system? Then you have the best of both worlds: Good documentation and a system which actually works.

fafk• 7 years ago

A complete rewrite is rarely a good idea. Joel Spolsky summed it up in on of his great blog posts: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

titzer2• 7 years ago

Don't do a rewrite until you have an effective grasp on what the requirements on both the old and the new system are/were. A running system is better than nothing, but it is no substitute for requirements engineering!

dgellow2• 7 years ago

Because no one in the team has a great understanding of the system

Isn't that the main issue? I understand to not want to do invasive changes when you're not familiar enough with a system, but the solution seems to be:

1. get familiar with the system (doing small non-intrusive changes is a good way to start to become familiar with it)

2. do more invasive refactoring

For sure don't start directly with the invasive changes, but at some point you need to get a good understanding of the system you are working on, or are responsible to maintain.

ekvilibrist• 7 years ago

The problem with "having a great understanding of a system" is that it's hard to pin down, to define what that actually means. You can always use that as an excuse, if you are gatekeeping a poorly constructed system and resisting change.

If you require "full understanding of our system" for me to add some new functionality in a module, then chances are pretty good that your system has a bunch of problematic dependencies, no?

dgellow2• 7 years ago

The article seems to be about "invasive and risky changes" (that's from the article). In this case I expect some level of gate keeping by people familiar with the system. If people doing the gate keeping aren't familiar with whatever they keep, then that's the first thing to work on.

If you require "full understanding of our system"

I wrote "familiar enough", and "good understanding", not "full understanding". "enough" and "good" will of course depends on the context.

Considering this:

then chances are pretty good that your system has a bunch of problematic dependencies, no?

In practice you need to understand some level of the context in which your module exist, hopefully not all of it, though of course it would be better to be able to just focus on the module itself. By "system" I don't necessarily mean the whole, complete infrastructure, it can be the module.

My point was that if people are blocking changes because of lack of understanding of something, the solution would be to actually get some level of understanding.

Edit: also, I assume good faith from gate keepers.

scarejunba2• 7 years ago

One of my biggest failures was while managing a team like this. In this organization I was a fairly well-liked leader and had both successful and failed projects and I was brought in to stabilize a core team that was having rampant attrition, had a two-year project that had practically stalled, and was feeling somewhat in a siege relationship with the rest of the company.

Interestingly, the whole situation was almost exactly what he describes here and a full year later I had failed to correct it (and probably made it worse). Here's what I did so that others may learn from:

* Since the team had stalled on that project for two years, I pretty much sidelined it and attempted to get us in a habit of getting wins so that we'd feel comfortable with changes. Result: We just accelerated the muddying of the codebase. The team wasn't constantly upset anymore but the size of the improvements you could make was decreasing - a sign that you're in the doom spiral.

* Set aside time for us to perform the refactors, but allowed the team to identify the primary pain points and focus on them. Result: The muddied code-base was like an underwater cave, the silt was in established places. The attempted refactors were just more muddying. With two years of living with an on-call rotation, the team's primary focus was in trying to remove the things that caused them direct pain, not the underlying causes of them, because they felt that the time we had was insufficient to do anything meaningful.

* I championed automated testing, replicable builds, and atomic changes but failed to really make the argument since the net takeaway was "doing it that way would be great but it would be too slow". It didn't really take, but perhaps I should have had an enforced approach to that rather than an approach to make an argument that convinces. We got somewhere with the testing but it was a big effort. That cost me a little bit on the culture since it had to come top-down from me.

* I tried 'leading from the trenches' so to speak, but that was unsustainable. Things worked while I was there, working as an engineer along with everyone else, but then I had to sacrifice other crucial things that were necessary to retain the state.

In the end, things got somewhat better there in terms of attrition and big system failures, but the doom spiral was still there. So it was like adding x years to failure rather than putting us on the path to success. I'd like to think that now, with some distance, I have a much better picture of what to do.

To be honest, I really don't think that adding a slew of manual testers is really the solution. The iteration speed will plummet because now there is one more hinge to the arm controlling output.

The annoying bit about the whole thing? None of that is useful to me now. When you're trying to build a startup you wish you had problems like this because it means you are already successful. Ugh. Maybe some day it will be of use.

jrochkind1• 7 years ago

I'd like to think that now, with some distance, I have a much better picture of what to do.

OK, what, please share!! You told us what didn't work, in retrospect with distance, what would you have tried differently?

scarejunba2• 7 years ago

I think I would have trouble expressing the nuance in a HN comment given how much time I'm willing to spend on it.

collyw2• 7 years ago

I have noticed that database changes on a live system are the hardest / most risky to implement. Also they are most likely to get to the heart of the problem. I learned what the article is talking about through experience - avoiding the "harder" changes and piling fix on top of other fixes until things become a mess.

titzer2• 7 years ago

As always, there is more than one valid perspective on this problem.

Given the very first example from TFA, the problem is solved (i.e. the new requirement is satisfied) by only adding a few lines of code. In fact, a massive refactoring was not required. I would suggest that this is actually A Good Thing(tm) and may even be indicative a Good Design(tm). If every new requirement requires the system to be overhauled, you're definitely in a worse situation.

The lack of tests/testing is a wholly separate issue.

If the new requirement comes with new tests, and even better, tests for the old behavior and the new behavior, all the better: you can refactor the system at a later time to make it cleaner and simpler and still meet the requirements, which is even better, since you have a fuller picture of what the actual requirements are.

Refactoring in the face of every new requirement smacks of poor initial design.

zbentley• 7 years ago

Refactoring in the face of every new requirement smacks of poor initial design.

This is true.

The idea that a system is well-designed if it allows changes/unexpected new requirements via small, mostly-additive, easily testable changes is also true.

But it is also true that systems remain well-designed until they aren't any more, because too many things have been changed/added. And identifying when that watershed has occurred (or, for extra seniority points, identifying in advance when it is likely to occur) is critical to good engineering over the long term. The point at which "you can refactor the system at a later time" becomes "that used to be the case, but now we need to actually pay down the debt, the tradeoffs have gotten too bad" is the most valuable to identify.

hinkley4• 7 years ago

“Make the change easy, then make the easy change” is one of the tenets of refactoring. You do, in fact, refactor the code with every requirement.

I really, really hated YAGNI about ten years ago but have come around a bit. Our users are empirically insane. You cannot guess what an insane person will do next, and trying will only make you insane, too.

I’ve gotten a lot of good mileage and a lot less stress by following the Rule of Three (architecting on the third example). I’ve learned to spot bullshitters versus sincere YAGNI folks (but many more if the former). The critical factor is identifying which decisions are reversible, not investing much energy in them, do everything you can to delay irreversible ones, and failing that make people pay attention to what they are choosing.

The Birth of Legacy Software – How Change Aversion Feeds on Itself

Related Stories