The joys of maintenance programming (2011)

https://typicalprogrammer.com/the-joys-of-maintenance-programming

159 71 3 years ago

x1f604• 3 years ago

Do people understand code by line-by-line reverse-engineering what the code is doing, or do they understand it by relating it to what they've written before?

If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.

For system design too - if you've designed lots of systems yourself, then when you see a new system, you'll be able to relate it to the systems that you've designed before.

So maybe building greenfield projects also makes you better at maintaining existing projects?

It'd be great if someone could point me to some existing literature on this topic. I've looked around and can't find any.

fatnoah• 3 years ago

I haven't had to do it a lot, but in previous cases where I came into a codebase blind and had to fix bugs or figure out what's going on, I've always started by mapping the flow first and then digging into the details of what's going on. When looking at the flow, I go breadth-first through everything and then dig in from there.

I maintain two major maps, which are what is the code logically trying to do, but also what exactly it's doing. (i.e. Sanitize inputs, check error conditions, do work, return result as a logical model and then "ensure foo is < 0 and > -10" etc. as the specific checks).

darkhorse222• 3 years ago

If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.

Let's hope what you write the first time is a good practice.

tivert2• 3 years ago

So maybe building greenfield projects also makes you better at maintaining existing projects?

I think it does, because it builds a higher-level sense of how something "could" or "should" be and familiarity with thinking at the system level.

I've had a lot of problems with people who (seemingly) only have experience maintaining projects. They seem to have a tendency to focus narrowly on "fixing" a bug close to its source, and often lack understanding of a lot of fundamental technologies (because they only attend to things that have broken), and get stuck by a kind streetlight fallacy where they fix things in more familiar areas. The end result is ugly kludges and whack-a-mole bugs where a superficial "fix" in one area causes a new bug to pop up in another.

ye-olde-sysrq• 3 years ago

> So maybe building greenfield projects also makes you better at maintaining existing projects?

Only to an extent, i think.

I've had a lot of problems with people who (seemingly) only have experience maintaining projects.

And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.

I think "i've never bootstrapped a project before" is easier to cure than "I don't have a good sense of what's expensive vs what's cheap tech debt."

Some tech debt is basically 0% interest. Was this small hack bad? Yeah. Will it need to be fixed, eventually? Yes, for sure. But does it compound every day, or does it just sit there being a little yucky? Very easy to determine in hindsight. Very hard to predict as you write the hack. The end result being the simultaneity in my example. People will over-engineer things that won't end up being problems and under-engineer things that will turn out to require compounding hacks and it would've been cheaper to just get it right the first time.

rvba• 3 years ago

There are lots of greenfield-only programmers who build something (by glueing stuff together), then they proceed to ship their barely working software to get that promotion / bonus and they run away to the next project

They never have to maintain anything, some poor soul has to fix the broken software.

They also dont even know what mistakes they made, since they arent there to solve it.

hinkley7• 3 years ago

I think there’s something to be said for taking a job on a late stage project early in your career, participating in RCA analysis of all of the problems, and then picturing yourself in the room when those decisions are made and asking if you would have made the same decisions or better ones. Then work next on a greenfield project to test your theories out and adjust/grow.

I never felt like I had 1 year of experience 5 times because I moved between several kinds of projects at different lifecycle phases and with different constraints, and drew lots of parallels between them. At the end of five years I had project histories in my brain that spanned more than ten years. And got jobs that should have gone to someone with 8 years’ experience. I do not think this was a coincidence.

hiAndrewQuinn2• 3 years ago

I love maintenance programming. I love legacy programming too. I have a story where the two worlds collide.

At my last job I had to work with a PowerShell codebase that `cloc`ed in at over 100,000 lines, and was absolutely ancient to boot, having been added to in bits and pieces by sysadmin after sysadmin over the course of 12 years total, in 3 different human variables-and-comments languages, all of whom were Windows power users with very clear ideas of what needed to be done, but also all of whom were very clearly learning to program for the first time, at least in the wonky PoSH paradigm. AKA: Classic big ball of mud situation. Every new customer had to have their details run through this codebase, however, and I saw the potential to eventually boil this down to a simple 6- or 7-parameter cmdlet (PoSH for "shell script"). I got my undersized chisel and toy hammer and got to work.

In some ways starting work on the proj felt more like beginning to dig into an ancient COBOL database in a System/360 mainframe than anything resembling the state of the art in 2023, or even 2011. There was a single VM that everything had to run on, and the script couldn't even be called remotely via SSH because the only way in was via a web browser based remoting session. The connection was spotty and would often drop halfway through coding in there in my dinky little Notepad++ session, requiring minutes of reauthentication just to get back to work. Unit testing (which PowerShell had a surprisingly good third party library for, called Pester) was out of the question. And: There was no version control! No wonder it got to 100k lines - most of the code was being entirely unused! But what?

The first few Saturdays I spent didn't go very far. I was mostly trying to figure out where the logic even began and ended in this mess of spaghetti. But eventually I started to piece together an overall structure for the thing. I made a pact to myself not to delete any of the many, many files that had accrued over the years until I knew every file that the code would touch first. I started doing some lightweight encapsulation with the -WhatIf flag, and with firebreaks (sticking `return` after each tiny function) so I could work on one linear piece at a time, and gradually tiptoe my way forward to the finish line. Eventually I got there, and managed to strip the whole thing down to about 1500 lines.

The joy of this kind of programming is that, even if you don't start with regression tests or modern practices, you do know from having ran through the original process a million times exactly what the end product is supposed to look like. Since no one is paying attention to your refactoring work in the first place, you're allowed to take your sweet time. To me it was obvious this mess of code was going to become a blocker for the business at the pace we were going: We were a SaaS, and yet we had to manually provision customers behind the scenes like this, in a process which took days to weeks? And we didn't even have a proper CLI we could call for this? Jackpot!

I left the company about 2 months after wrapping this project up and demoing it to my coworkers in search of better pay, but I'm really glad I got the chance to take such a fearsome codebase and tidy it up like that. There is just something satisfying about that kind of work, it's like gardening. I also got really, really good with Notepad++'s find and replace across all files functionality. There is in fact a skill to it! (Mostly that you should always, always find and replace the longest strings first!) Although I think these days I'd probably reach for a similar function in vanilla vim first, or even in bare `sed` if I was feeling gutsy.

(I'd end with "If you, dear reader, are in a similar PowerShell quagmire you need someone to pull you out of, I'm available for consulting at $X00/weekend", but alas. My weekends for the next few months are already sworn to another equally wonky project for my current employer. At least I'm getting overtime this time. ;) )

windock• 3 years ago

I felt so much joy just reading this! Yeah, it is truly great to just simplify.

augustk• 3 years ago

Mostly that you should always, always find and replace the longest strings first!

In Notepad++ you can mark "Match whole word only" or use a regular expression with word boundaries, e.g.

  \bsome string\b

hiAndrewQuinn2• 3 years ago

True and helpful, but perhaps not as universalizable as past me would have wished. I think IntelliJ or JetBrains or one of those big honking IDEs has support for proper intelligent refactoring of variable names like this.

jebarker2• 3 years ago

I'm currently on a sort of sabbatical from greenfield research projects. I'm doing maintenance programming on a large widely used open source project. It's been really refreshing. I didn't realize how much I was feeling the pressures of 15+ years of greenfield research. By contrast the maintenance programming is clearly defined, easier to put down at the end of the day and unambiguously useful. I'm also rapidly improving in my debugging, code understanding, profiling and optimizing skills which will make me a better programmer when I do return to writing new code.

aidos• 3 years ago

I’ve described this to other programmers before. Greenfield projects are fatiguing because you still have options—too many options. Every layer is still open for you to rethink and rework. Let the hand wringing commence!

As the months and years wear on though, things change. Each bug has an obvious solution, and you’ll take the path of least disruption. Implementing new features is a matter of figuring out which trade offs you’re most ok with. Maybe you need to refactor something first, but that’s an obvious path itself.

You’re painted in a corner. A beautiful corner where the only safe thing to do is to tread carefully. But you don’t mind, because you know where you are and what you need to do to get to where you want to go.

jebarker2• 3 years ago

Nicely put. I'd also add that for me being a researcher became my identity and so my self-worth became tied to the perception and adoption of the research. With an established project (especially one I didn't write) there's already a stable acceptance of it's value and so I'm free to just maintain and improve it.

dang2• 3 years ago

Discussed at the time (of the article):

The joys of maintenance programming - https://news.ycombinator.com/item?id=2267261 - Feb 2011 (29 comments)

pramodbiligiri• 3 years ago

The dupe detection on HN during submission doesn’t find much older submissions?

dang2• 3 years ago

For much older submissions it doesn't try, because reposts on HN are fine after a year or so (https://news.ycombinator.com/newsfaq.html).

There are a bunch of reasons for that, but two interesting ones are (1) historical material is particularly welcome here, and (2) we want cohorts of new users to get an 'education' in the 'classics'. It's too easy for grizzled veterans to assume that everyone has already seen the perennials, but even they once read The Story of Mel, or whatever, for the first time!

JohnBooty8• 3 years ago

Maintenance programming for pay can be rewarding iffffffffffffffff management has a deep understanding of and appreciation for the task. There can be a lot of joy and satisfaction in adding tests, fixing edge cases, fulfilling user needs, "quality of life" improvements, optimizing. Honing.

My experience is that this is almost never the case.

Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.

Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.

But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.

jckahn• 3 years ago

Agreed. Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money? It seems much more sensible to leave the working code alone and focus on new revenue-generating opportunities.

bbbobbb• 3 years ago

That's not what the post is about. They describe the process as mostly fixing issues with existing software, so the motivation is that something is broken or not working as required anymore.

Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.

riku_iki• 3 years ago

Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money?

CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.

dagw• 3 years ago

It is not at all obvious that trying to develop new revenue-generating opportunities will actually bring in more revenue long term compared to optimising and improving existing revenue generating sources. And it's not like the old code stops working once you start reworking it. Give your engineers N months to rework the old code, and if it's not better you don't have to deploy it.

Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.

juliangamble• 3 years ago

Security and risk. You can run several different code/pom/package.json scanner tools that will articulate a risk that will send your IT security department into a panic attack, which you can sell as time you need to spend fixing things.

whateveracct• 3 years ago

CEOs tend to be short-sighted when it comes to engineering management, that's true.

fsckboy• 3 years ago

focus on new revenue-generating opportunities

in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.

hinkley7• 3 years ago

Why would you turn off manufacturing equipment to do routine maintenance? It’s making you money when it’s turned on, and not when it’s turned off, after all.

throw239487• 3 years ago

Does anyone have any tips on this beyond the additional "sales" type stuff where you have to put in a ton of extra effort to measure and sell your work?

I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.

I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.

JohnBooty8• 3 years ago

I think one of the keys is consensus-building. If one engineer tells management that it would be a great idea to do XYZ, management doesn't necessarily know whether to believe them.

But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.

hooverd• 3 years ago

Nobody cares about "eliminated a whole class of tickets" aside from the support staff, and really who cares about the support staff at most companies. What would the phone monkeys know about our software. /s

obviouslynotme2• 3 years ago

It's worse that that. Most companies give career incentives to new projects. Google is probably the most well known offender, but they aren't the only one.

Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.

chii3• 3 years ago

the most valuable asset of a programmer is his mental model of your system.

Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.

Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!

hinkley7• 3 years ago

Google has turned it into a sport though. That deserves special mention.

ern• 3 years ago

I was almost bounced by HR during a screening interview a couple of years ago when they found out I had been doing maintenance work for the previous six months. I had to explain that it was a tiny part of my overall work experience. The hiring hypocrisy is strong.

Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.

JohnBooty8• 3 years ago

That's absolutely mental. I know maintenance work is uncool and kind of a career... well, not a stain, but it's not glamorous.

May I ask what sort of company this was? Some "unicorn" startup? Stable oldcorp?

swader999• 3 years ago

The joy of Greenfield is short lived. At some point, it becomes too costly to fully refactor and you are stuck with what you chose and edging closer and closer to legacy dev.

Some projects though just keep on trucking along and doing the next part with the newest shiny thing. You end up with five ways to access data, three ways to send emails and fifteen different methods of report generation. Frankenstein architecture! This architecture also emerges when you have leaders that don't make decisions and peeps debating all day on slack and then doing their own thing anyway because they didn't really want to offend anyone and everyone's ideas are great.

stringtoint• 3 years ago

Nice post. In the past I've had colleagues (also senior people) who disliked touching "legacy" parts (e.g. Java based microservices compared to newer Express.js or Kotlin based microservices which don't provide half the value that the legacy parts do). I also think that maintenance programming can be rewarding. However, I do believe that it does not make sense to throw entry-level engineers into the maintenance mess alone: they still need some guidance in my experience. Then you'll be able to reap the benefits such as getting better in debugging and understanding more about the business problems that the software is solving.

vsareto• 3 years ago

Skilled programmers recognize what makes good code good, and what makes bad code bad

You're still somewhat biased, so you can get put on projects where the style is different from yours. If you happen to think that style is bad, it won't be fun to work on.

You shouldn't want to change the style. Chances are you'll do it incompletely (job change) or the time spent won't be worth it. If someone else comes along and doesn't like your style, we're just going to repeat this process.

There's been way too many different approaches that work for developers to be unified on what styles are best.

Rexxar• 3 years ago

Style is not what make code good or bad.

lodovic• 3 years ago

I'm still hoping someone will create the ultimate linter that can format source code in your desired style on checkout, and formats it back to the repository's style after a push

Jabrov• 3 years ago

Let's say I wanted to do this. How do I actually get started?

Let's say I make a website that showcases my qualifications and lists out the services I offer.

How do I actually get a client?

tunaoftheland• 3 years ago

The author of the article also has some thoughts on those topics: https://typicalprogrammer.com/how-to-start-freelancing-and-g...

I’ve only skimmed it and have started reading but the parts about highlighting/phrasing of one’s marketable skills seems good.

jakobson14• 3 years ago

After the NX bit, and cookies, and shadow stacks, and now tagged pointers (again) I'd rather stop pushing the guardrails for shitty code down into the hardware. It costs performance and the bandaging never ends.

SKILNER• 3 years ago

I gave a presentation many years ago to a software process group in Silicon Valley entitled, "Unsolved Problems of Software Maintenance". The problems are all still there.

Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.

I heard this first from Joel Spolsky, "it's easier to write a program than read it."

Which led me to write an article on the subject, "How To Read A Program," about the cognitive aspects of trying to learn unfamiliar code. https://www.itjungle.com/2009/01/28/fhg012809-story01/

The article uses AS400 applications as an example, but is general enough that it applies to applications in most languages.

This is a field begging for help from AI.

hinkley7• 3 years ago

Someone needs to start doing comp-lit style studies of computer code instead of literature.

Most of the problems of API design could be remedied by people stepping through their own code and looking askance at the whole process.

summerlight• 3 years ago

One more benefit is to learn how to tame complexity. The most complex piece of software doesn't come from nowhere; it is built over years, if not decades. People will never experience the same level of complexity in the "greenfield" programming. And that experience changes the entire mindset, from how to write the codes to how to structure the entire engineering org.

Cthulhu_2• 3 years ago

I wish I spent a year doing this at my previous job instead of doing what I was hired to do, rebuild the project. The rebuild would've taken another 2.5 years, even if I had help.

But the existing project could've used some improvements; there was an inefficient SQL result to XML string to JSON thing happening with every request, so doubling response times could probably have been done within a week of work or so, and quadrupling it with a few weeks of work (there was thousands of LOC of SQL to XML code involved, but in most cases it could've been SQL to JSON directly).

gregjor• 3 years ago

Author of the article here. Thanks @pramodbiligiri for posting. Surprising and humbling to see this old article on the HN front page. I haven’t read it in years.

When I wrote that, in the wake of the 2008/9 meltdown, I intended to give programmers some career options to think about. More interesting and stable work out there than FAANGs and startups.

The joys of maintenance programming (2011)

Related Stories