Red Flags Signaling That a Rebuild Will Fail

http://www.pkc.io/blog/five-red-flags-signaling-your-rebuild-will-fail/

by pkcsecurity • 8 years ago

289 137 8 years ago

lgleason• 8 years ago

I recently left a project that demonstrated most of these traits. Usually these things are the top of the ice-burg.

teddyh• 8 years ago

Know your burgs and bergs. A “burg” (or burgh) is a fortification, or more usually refers to a city built around (or inside) that fortification. A “berg” is a mountain, or a large hill. Therefore, an iceberg is an “ice mountain”, and a “burgermeister” is a “city master”; i.e. a mayor.

de_watcher• 8 years ago

You forgot to mention that you should use "tip" instead of "top" in this idiom.

Here is a video with more detail: https://www.youtube.com/watch?v=dQw4w9WgXcQ

alkonaut• 8 years ago

The truth I think is more often that the legacy system is too old and brittle to improve, and customers are demanding ever more complicated features from it.

So you rebuild as a new system as a gamble, because even though it shows all the traits described, the new system is at least one that anyone is willing to develop, and one where features can be added, and to which people can be recruited.

We know big rebuilds have small chances of sucess. But that doesn’t mean you shouldn’t do big rewrites. You are in a bad place if you even consider. Maybe the big rewrite means the company has an 80% risk of going under. Still could be that safe bet.

Chyzwar2• 8 years ago

The rewrite is usually when it is too late for the project. Need for re-write mean that project maintenance was ignored and technical debt reached critical levels.

I would start by firing people that led to this situation.

Buttons840• 8 years ago

They're already gone, almost certainly.

lovich• 8 years ago

I've found that they are usually still there but as they are the CEO/CTO it's difficult to get them fired

a_imho• 8 years ago

People write legacy systems from day 0, especially in resume driven development.

clintonb• 8 years ago

If you fire those people, you remove your source of expertise on the old system. Yes, they did a poor job of maintaining the old system, but their knowledge may be valuable to understanding the old system and creating requirements for the new system to reach parity.

JumpCrisscross• 8 years ago

I would start by firing people that led to this situation

Thereby fomenting Red Flag #4, not "working with people who were experts in the old system.”

Chyzwar2• 8 years ago

I am not saying to fire everyone. I am just saying that someone needs to be responsible. If you keep the same people in power they will repeat the same mistake. You need to keep domain knowledge but clueless management is just a burden.

maxxxxx5• 8 years ago

" I would start by firing people that led to this situation."

You are one of those blessed people who can architect a system and the architecture holds up for decades. From my experience most systems will end up in a big mess over time if features get added. There is almost no way around it.

flukus2• 8 years ago

You are one of those blessed people who can architect a system and the architecture holds up for decades.

This is exactly why maintenance is needed. Proper maintenance that includes things like updating the architecture and gradually migrating the whole system to that architecture, rebuilding small unwieldy components, updating and migrating database schemas as the product evolves, removing unused features.

If a product is just getting bugs patched and nothing else then it isn't really being maintained, it's being deprecated. Unfortunately as an industry we still think that there are distinct build and maintenance phases and that the latter can be done with less resources.

bokonon12• 8 years ago

Yup. So much of the time the system starts out at one thing and morphs to another. That can easily lead to core problems with your architecture

borplk2• 8 years ago

Often those people are high up enough that they are not going anywhere.

For example an executive/management team that over-commits the organisation and creates a culture of rewarding technical debt and punishing maintainers.

Rather than fixing these issues they will continually search for a super hero employee who is going to come in on a white horse on monday and fix it all up in two weeks.

maxxxxx5• 8 years ago

"Red Flag #4: You aren’t working with people who were experts in the old system.”

I think this is most important. A lot of people want to rewrite because they don't understand the current system and don't want to bother learning. Before you rewrite you really should understand the current state deeply.

_asummers• 8 years ago

Even having the entirety of the original dev team there, time takes its toll on recollection of reasoning behind some of the strange decisions made in something that would warrant a rewrite. Much preferable to not having them, of course.

DiabloD3• 8 years ago

Something I do is if the code looks weird or is rather small for how much work went into it, I leave a comment that says why this was done... just so I can remind myself in 6 months when I go "who the fuck wrote this garbage... oh, me."

munk-a2• 8 years ago

#4 is sort of terribly worded, the summary line is something that is important and pretty independent, make sure you're working with expert users of the system... then the explanation brings in a Senior Dev as a good resource to tap. This is the wrong direction, you really want to consult with the system experts to see their rationale for requesting what might seem like odd functionality in the first place.

#4 also mixes a good deal with #5 in that any changes you make (even purely good ones in your view) will require retraining of users and cause a kerfuffle when rolled out to your user base, people _hate_ change.

pbreit2• 8 years ago

This strikes me as dangerous. Didn't the experts build the first system? Don't you want to deliver a fresher system? Won't the experts be attached to the old way of doing things?

grantism• 8 years ago

No necessarily. It depends on what has lead to the need for a rebuild. Sometimes there weren't previously the resources to "do things properly", Sometimes a feature might only added for a specific client, etc.

You need that previous knowledge to know the "why" of things & if that why is still valid.

IMHO it's more dangerous if you're working with experts who don't want to improve the system.

maxxxxx5• 8 years ago

Keep in mind that you will be one of these people in a few years for whatever you are doing now. The previous people most likely weren't dummies but had to deal with the technology and constraints at the time they built the system in the same way you are doing it now.

zbentley4• 8 years ago

Didn't the experts build the first system? Don't you want to deliver a fresher system? Won't the experts be attached to the old way of doing things?

With all respect, that means you should not be in a position to rewrite legacy code, or to commit others to such a rewrite.

If all the experts you have worked with have been, in your eyes, overly attached to the old way of doing things, you have one of two issues:

- You have not had enough experience in the field, and have not worked with experts that actually have perspective about when/how to rewrite, abandon, or rework their code.

- You have dogmatically condemned people who think that the latest-and-greatest tech may not be a good solution to the problems at hand to the "old fogey" bin.

Either issue means you're not ready to make decisions at this level. Learn more. Research more. Watch more. Listen more.

Weirdly, gaining this perspective has less to do (in my experience) with years on the job, and more with diversity of team/business environments worked in.

pjc50• 8 years ago

Ah, "the public have had enough of experts", the attitude responsible for most of our present political disasters.

mcguire• 8 years ago

On the other hand, experts will frequently demand that the new system do the same thing as the old system, in the same way.

You can't blindly listen to the experts.

ebikelaw• 8 years ago

Yep, seen that. I worked on a system where the company did not really want a reimplementation but they destaffed a project in one site and reconstituted it with all new people at another site. The new people decided to rewrite from scratch. A year and a half later I start getting questions by email from the new people, questions indicating that not only do they not understand the implementation of the legacy system, they also do not clearly understand the business requirements that resulted in that implementation. Meanwhile, the maintenance of the old system had been neglected to such an extent it had fallen behind critical company-wide mandates. This was more of a lesson about why you shouldn’t destaff a project over some petty geographical squabbles, but also quite clearly about why you should always incrementally reimplement software rather that rewriting it.

majormajor• 8 years ago

The way I've phrased something similar before is "don't do a full rewrite if you couldn't write up a plan for refactoring in place to fix the problems with the old system."

If you can build that plan, and make the case that it will be easier to do the full rewrite, go for it. But if you couldn't put together the fix-in-place plan, you might not understand everything the old system does well enough to actually estimate the size of a rewrite...

(This isn't solely for full-parity rewrites: if you're dropping features, what does that look like dropping from the old system?)

gwbas1c• 8 years ago

I was involved in a rewrite where it would have been much easier to refactor the old system.

A year into the process one of the c-level leaders pulled me into a room and asked why I couldn't fix the legacy code, and I basically told him that he should have pushed back on it. I couldn't fix the legacy code because that would be months of refactoring that should have been done instead of the rewrite.

Context: the legacy code had some design flaws that required major refactoring, but the legacy code "worked" except for very large deployments. The only problem was that the legacy system wasn't modular, so it didn't have unit tests and wasn't cross platform. All of those problems are easier to tackle via refactoring instead of a full rewrite.

mehrdadn3• 8 years ago

The way I've phrased something similar before is "don't do a full rewrite if you couldn't write up a plan for refactoring in place to fix the problems with the old system."

Hmm... there have been a number of times when I've banged my head against the wall trying to figure out how to make my own code do something, until I finally bit the bullet and decided to rewrite the entire chunk from scratch and suddenly it took a fraction of the time I had spent trying to fix it to get it written and working. Not sure how to reconcile this with the advice you gave.

antsar• 8 years ago

Also known as Chesterton's Fence.

https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...

arendtio• 8 years ago

I can't state it better. If you don't understand the decisions made during the development of the old system you are unlikely to come up with something much better.

pspeter32• 8 years ago

I think people also deeply underestimate the time it will take. We've undergone an incremental rewrite for ~4 years at Asana.

jupake• 8 years ago

Used your software once before. Loved it! You guys should do a blog post about your rewrite experience. Would love to know what your tech stack was and what your new one looks like.

toshaga• 8 years ago

This could be relevant: https://blog.asana.com/2017/08/performance-asana-app-rewrite...

solox3• 8 years ago

With this good article I think I have a good question.

The reference to Martin Fowler’s strangler pattern (https://www.martinfowler.com/bliki/StranglerApplication.html) was mentioned in the article to grow the new system in the same codebase until the old system is strangled. In my case (Ionic 1 to 2) however, both the entire framework and the language are different. How should the strangler pattern work in this case?

ronpeled• 8 years ago

There's no super easy way here. One way to get this done is find independent areas of the app that can be replaced without coupling. Then start building up as you go with the new system. At some point you'll be about 70% through of which you can decide if you want to make the jump and focus your efforts to completely uproot the old one.

Sorry for the abstract reference here, but it applies to almost any replatforming out there. In most cases it is a very expensive operation for a business and needs some major reasons in order to justify such a move.

omegaworks• 8 years ago

Are there any examples out there for how to do this with React in an existing AngularJS codebase?

Forge36• 8 years ago

The company I'm working at is doing this currently. The new product is on the web and the old one is a full client windows program. The biggest hurtle will be to find the balance between largest/smallest pieces which can be transitioned as seamlessly as possible.

curyous• 8 years ago

I'm surprised at what gets called a pattern these days. Fowler didn't describe it as a pattern, but just because Mr. Guru said it, it is now a pattern?

pbreit2• 8 years ago

The ways it's referred to as a "StranglerApplication" in this post [1] does suggest more than "just saying it".

1: https://www.martinfowler.com/bliki/EventInterception.html

twunde• 8 years ago

For webapps you would use a reverse proxy such as nginx or haproxy and replace your application page by page. Then configure the reverse proxy to send all requests to /home to go to the new stack and all other requests go to the old stack. Then flip the switch for every page you finish converting. For backend work, it's similar. You can have an api built in a new stack and it can just have a different endpoint or use a reverse proxy. Backend workers can pick up work from a different queue or you can switch the old job worker off and turn on the new one, and then monitor that everything is working as planned. The really important thing about the strangler pattern is that you need some easy way to turn on bits of functionality while turning off the corresponding old parts. It can be feature flags, it can be routing middleware. You can rip out the guts of the angular routing mechanism and use that to flip the switch.

wink2• 8 years ago

Seconded. Took part in a moderately big rewrite with this strategy and it worked pretty well.

Identify key components and subsystems and rewrite them one by one. From the outside you seem to be switching over one REST endpoint after the other, but of course internally it's a bit more difficult, but applications often enough have enough parts that are not SO intertwined that you can do stuff like this. It's a bit related to how you break up a monolith. Find bigger, less coupled parts and shave them off and just touch the glue code.

nerdponx2• 8 years ago

Another red flag not mentioned here: the old system doesn't have an end-to-end suite of functional test cases you can rely on.

Ensorceled2• 8 years ago

Red Flag #6: Key stake holders keep moving the goal posts.

If your goal moves from feature comparable but on a modern platform, to new features, to a complete reinventing of the product all without actually shipping ... you might be in trouble.

I had a rebuild go 6 months over. In the heated executive meeting at t+3 months I was called to defend my team and pointed out that the VP Product had just delivered “final” specs literally the day before. How could we be on track with development if PM is 3 months past “end of development” with design specifications. The fact that the specs were changing weekly because “we’re agile” is a whole other issue.

whatshisface• 8 years ago

People sometimes complain about how developers like to "write the operating system and then a language" when it comes to handling every foreseeable permutation of what the program might every be desired to do, but we're all so used to unstable requirements that sometimes the metaphorical programming language research is the only thing that will be general enough to find a use next week.

oneplane• 8 years ago

I almost had a few similar situations, but after pointing out that being agile doesn't just means changing requirements but also changing time paths or simply different deliveries after each change it got a whole lot clear what agile (and scrum) is good for, and what it's not good for (i.e. agile process but expecting waterfall results doesn't work).

Cthulhu_3• 8 years ago

The fact that the specs were changing weekly because “we’re agile” is a whole other issue.

The article touches on that too; simplified it's stating that if you're not live within 6 months, you're doing waterfall.

Ensorceled2• 8 years ago

That’s not waterfall. Waterfall you don’t start dev before specs are final.

Waterfall isn’t just a synonym for “the wrong way to do it” :-)

nostrademons• 8 years ago

#5 has a converse - oftentimes, the only way to get a rebuild to succeed is to drop features, and it's a major red flag if management insists on 100% feature parity.

The way to distinguish this from the #5 situation in the article is to ask if you're dropping features because they're hard or because nobody uses them. The former is a red flag; the latter is a green flag. Before you embark on a rebuild, you should have solid data (ideally backed up by logs) about which features your users are using, which ones they care about, which ones are "nice to haves", which ones were very necessary to get to the stage you're at now but have lost their importance in the current business environment, and which ones were outright mistakes. And you should be able to identify at least half a dozen features in the last 3 categories that you can commit to cutting. Otherwise it's likely that the rewrite will contain all the complexity of the original system, but without the institutional knowledge built up on how to manage that complexity.

maxxxxx5• 8 years ago

I have seen that before. You kill yourself refactoring a feature only to find out it's never or barely used. Deleted code and features are the best.

hinkley2• 8 years ago

You hamstring the product to make a feature work one way then find out that what they really wanted would have been easier to implement but they never asked because they thought that would be harder.

philliphaydon2• 8 years ago

I once worked on a feature that apparently lots of clients were asking for. It took 4 weeks to implement. Went to production. Never heard anything of it. 2 years later if we could modify the feature to work for another usecase. I looked at the database. The feature had never... ever... been used.... rows returned = 0

throwaway2048• 8 years ago

One thing to be careful not to fall afoul of when you choose to remove features is assuming there is some kind of meaningful average user.

A good example is MS Office, there are an huge amount of features that only 5% of users might ever use, but the majority of users are likely to use quite a few of these niches individually, and if you remove all the low use features, you piss off basicly everyone.

I think the mistaken idea of an average user is why a lot of metrics driven software seems to get more and more useless with every update.

(I cant see the present/away status of contacts in the newest skype, really guys? )

lordofmoria• 8 years ago

Before you embark on a rebuild, you should have solid data (ideally backed up by logs) about which features your users are using, which ones they care about, which ones are "nice to haves", which ones were very necessary to get to the stage you're at now but have lost their importance in the current business environment, and which ones were outright mistakes.

This is so important. I've been on many a project where, 3 months in, we wish we had historical tracking data on user activity to back up our instincts to cut a particular feature that seems worthless. The worst part? Even if you add it immediately, you'll have to wait 2-4 weeks to get a sufficient amount of data.

manicdee• 8 years ago

Also important to realise that a feature that is rarely used (view history, remove user) might be more important than one used more often (dashboard widget that nobody pays attention to)

kornish3• 8 years ago

The worst part? Even if you add it immediately, you'll have to wait 2-4 weeks to get a sufficient amount of data.

I think this was the problem a product like Heap [1] was designed to solve: just track all user actions, forever, and then assign pipelines after the fact based on what you want to check up on.

Don't work at Heap or anything, just love the team and product.

[1] https://heapanalytics.com/

munk-a2• 8 years ago

I think it's important to separate feature improvements from a technical rewrite, ideally in the rewrite you mostly just make things work the way they did, sometimes you might fold a feature improvement into it but if you come out of the rewrite with a more stable product that has about the same usage stories you should consider it a success.

Sometimes you will want to fold features into a rewrite (remove prompting the user to confirm X twice) sometimes this will ease development and be worth it but other times it'll pay off to just retain the old functionality but add it to a list to be user tested later.

Once the tech is solidly over then take a swing at updating the poor UI, do it agiley so you can back out of changes that the user base rejects since (at least within my more modest usage studies) not everything people depend on comes up or gets reported. I'd much rather rollback a design feature branch then have users get change fatigue when you're forced to rollback your new shiny rebuild and the whole project ends up being shelved.

mehh• 8 years ago

If only my current project had done this we would of saved millions!

brazzledazzle• 8 years ago

There have been a couple times where I’ve tried to use a feature that should have been awesome but was terrible then it got pulled in a newer version of the product. It was incredibly frustrating to wait for a fix that never came. Data on what’s used is good but you need to get feedback about what sucks to go along with it.

perlgeek• 8 years ago

And you should be able to identify at least half a dozen features in the last 3 categories that you can commit to cutting.

Ideally, you disable them in the old software, and observe how many people complain.

Too often, product management commits to cutting a feature, and then caves in when paying customers complain. It's best to know in advance which category a feature really falls in.

wink2• 8 years ago

Ideally, disabling features in the old software is not so complicated that a rewrite suddenly sounds even more enticing. /s

eecc• 8 years ago

100% feature parity sounds like the advice in #4, involve Marge, without actually having a Marge to call. That’s supernatural development;)

micheljansen• 8 years ago

A bigger red flag, in my experience, is an unwillingness to even consider dropping any features. Often combined with a desire to add new features during the rebuild. Always goes wrong.

Cthulhu_3• 8 years ago

Feature parity is the reason why some of the projects I've worked on caused #2 - can't get customers to switch if there's no parity yet. The MVP for some of those projects took a year to get to. Mind you it'd probably have been 6 months if they didn't opt to go for a microservices architecture.

tomelders4• 8 years ago

I’ve carved a career out of rebuilds. I’m working on a rebuild right now. There’s a ton of companies out there who’ve done very well with their home grown antiquated systems from the late 90’s and early 00’s that are now facing stiff competition from young upstarts who had feature parity from day one and are knocking out new features at break neck pace because they’re leveraging the latest and greatest in tools, technology, and thinking.

I’ve always been a big believer in rebuilding your product from the ground up. I think it’s something you should always have going on in the background. Just a couple of devs whose job it is to try and rebuild your thing from scratch. Maybe you’ll never use the new version. But I think it’s a great way to better understand your product and make sure there’s no dark corners that no one dare touch because they don’t understand what it does, how it does it, or why it does it the way it does.

And I’ve always believed that if you don’t want to rebuild your app from scratch, then don’t worry, a competitor will do it for you.

So I agree with every point raised in this article. And I think it does a great job of articulating the issues that often go unspoken. But I’d like to add one more. And for me, this is the biggest issue for any company wanting to rebuild it’s product.

If your sales team has more clout than your designers and developers, then you’re fucked. And in the enterprise software world, this is the norm. An uncheked sales team that get’s whatever it wants has already killed your product and made it impossible to rebuild. Their demands are ad-hoc, nonsensical, and always urgent. So urgent that proper testing and documentation are not valid reasons to prevent a release. Their demands are driven by their sales targets, and the promises they make to clients are born out of ignorance of what what your product does, and how it does it.

This is not true of all companies. Many companies find a reasonable balance between the insatiable demands of a sales force and the weary cautiousness of their engineers. But if your company submits to every wish and whim of your sales team, and you attempt to rebuild your product, then you’re screwed.

omeid23• 8 years ago

I’ve always been a big believer in rebuilding your product from the ground up. I think it’s something you should always have going on in the background. Just a couple of devs whose job it is to try and rebuild your thing from scratch.

Hahaha. Just a couple of devs?

tomelders4• 8 years ago

Their job isn’t to build the whole thing. Their job is to research and explore how new ideas and tools might be useful to your business.

It’s just R&D. It’s not an exotic idea.

flukus2• 8 years ago

I’ve carved a career out of rebuilds

What's your learning process? If you don't do maintenance how do you know your rebuilds aren't creating the same problems that lead to the systems needing replacement?

I've got a very well founded distrust of people that only work on green field projects, they're generally responsible for the system's that need rebuilding.

omeid23• 8 years ago

I have also come to believe that people who jump to rebuilds also tend to have very shallow technical skills and are not keen or capable of studying and analyzing a system at depth.

tomelders4• 8 years ago

Well there's little else you can do when your app is a Java applet and your runtime has just vanished from the web. These things still exist, and people like me are rebuilding them.

I don't appreciate the snark in your comment.

konschubert• 8 years ago

Just a couple of devs whose job it is to try and rebuild your thing from scratch.

That may be good value for big established corporates, but for startups and smaller companies I don't think it is.

Djvacto• 8 years ago

Well (on a relative scale), won't most startups or smaller companies be more in the phase of "writing" as opposed to "re-writing"? I think the advice above would in theory apply to companies big enough to have legacy codebases.

hvidgaard2• 8 years ago

I’ve always been a big believer in rebuilding your product from the ground up. I think it’s something you should always have going on in the background. Just a couple of devs whose job it is to try and rebuild your thing from scratch.

Their time is much better spend working on improving the "legacy" codebase. Simple refactoring and splitting the codebase in a modular fashion, mean you can work on limited parts of the system in isolation. This makes incremental improvements and switch to new tech much easier, and certainly less risky than a rewrite.

spronkey• 8 years ago

Depending on how heavily coupled the legacy codebase is, "Simple refactoring" really may not cut it.

I mean, you can write a bunch of pinning tests, then try to prise out various bits and pieces, sure.

But what if all the stuff you're trying to prise out can now be accomplished with a few open source libraries that didn't exist way back, with a very simple rewrite of your business logic on the top?

That's a situation I've encountered quite a few times - a lot of legacy code that's largely boilerplate, with business logic drizzled over the lot, oozing into the little cracks.

esdkl22• 8 years ago

If your sales team has more clout than your designers and developers, then you’re fucked. And in the enterprise software world, this is the norm. An uncheked sales team that get’s whatever it wants has already killed your product and made it impossible to rebuild. Their demands are ad-hoc, nonsensical, and always urgent. So urgent that proper testing and documentation are not valid reasons to prevent a release. Their demands are driven by their sales targets, and the promises they make to clients are born out of ignorance of what what your product does, and how it does it.

Well said. This is easily my #1 biggest pain point as a developer.

darkerside• 8 years ago

It's very hard to get a man to understand something when his salary depends on his not understanding it. By the same principle, as someone who has built a career out of rebuilds, we shouldn't be surprised that you'll recommend this solution for a majority of hypothetical problems. I don't think you are intentionally misleading people, and I'm sure that you want the best for your clients and that you believe that's what you're providing. It's just that, for anyone else reading this thread, please realize that you're getting one side of the story.

Incremental rebuilds are not sexy. Adding unit tests to legacy code (thereby making it not legacy code according to Michael Feathers) is not sexy. Sticking with the tried and true technology is not sexy. But they are typically the most successful approaches for those not compensated for changing things for change's sake.

wellpast2• 8 years ago

Red Flag #1 should be that you’re doing a rebuild.

CydeWeys• 8 years ago

I'm potentially looking at a situation like this right now at work. We're on a NoSQL DB and it's just not working too well for us anymore, so we would like to transition to something that provides more relational semantics (PostGres, Spanner, something like that). Migrating the backend between one kind of DB and another is non-trivial, especially because the whole ORM needs to be ripped out as well. It's not a full rebuild of the application but it's definitely substantial in effort level.

Sometimes a rebuild is just necessary, because you are on a tech stack that is no longer working for you, for whatever reason. How would you solve that kind of problem?

wellpast2• 8 years ago

As incrementally as possible. Eg, does your entire data model need to move at once? (Probably not.)

grey-area• 8 years ago

I'd definitely vote for PostgreSQL, it can handle large loads effortlessly, it's reliable, and yet they keep adding great features.

It could also function pretty much like a nosql db initially, to ease your transition, then you could migrate gradually to using it as a relational db. You need strong checks on data integrity before you start - you could consider double writing (to old orm using nosql + new orm using psql), and comparing data stored to be sure you don't miss anything at first, before you switch?

hvidgaard2• 8 years ago

The first thing you do is refactor with the existing DB, so you have a clear DataStore component. Then you make your shiny Relational DB implementation of that DataStore. Now you run both side by side and for everything you do in the old DB you do the same in the new DB, and you compare the results. At some point you can turn off the old DB with confidence and sleep well knowing that the new DB behaves the way you expect.

realusername• 8 years ago

Sometimes you just have to. In one previous company, the "system" we were trying to trash was an unmaintainable VBA CRM homegrown mess which was creating lots of internal issues in the company due to the nature of spreadsheets. It took almost a year to replace but it was 100% worth it.

borplk2• 8 years ago

This is so frequently true that people are tempted to make it a strong NEVER. But that is also a mistake.

There are some legitimate cases where you really should be rebuilding.

You may not have seen such a case since they are rare, but they do exist.

A good rule of thumb is to try your absolute best to avoid a rebuild. If at the end of your hard work you still feel defeated and forced to go with the rebuild option, you probably should rebuild.

lyqwyd• 8 years ago

This article really captures the risks of a rebuild. I’ve been through a number of them, all but 1 abject failures. The one success was driven by the executive understanding that the company would fail without a rebounds, and it was still 6 months late, resulted in one of the cofounders being fired, an extremely painful rollout, and the company still failed, due to other problems.

My firm belief is that when you need a rebuild, you are already well into a fail state as a company. Not to stay there can be no recovery, but it is an indication of some deep problems for the company, beyond anything the engineering department alone can resolve... and if the rebuild is not coming from the executive leadership, it is an even bigger issue as it will more likely lead to bigger problems than it will solve.

kazishariar• 8 years ago

¯\_(ツ)_/¯:'Dual commits' to the rescue! -pun intended

gaius3• 8 years ago

Missing the biggest red flag of all, engineers wanting to just play with new toys and pad their CVs. Ask the engineers why they want to rebuild and listen carefully to the answer and if it’s vague handwaving and buzzwords (microservices! Containers! New JS framework!) and no hard numbers to justify it, just say no.

For example “we spend X/year on AWS but if we spend Y to rewrite in C++ we need fewer VMs and can cut that to Z/year” is simple calculations. If your engineers can’t even do that, their motives are suspect.

shoo• 8 years ago

http://boringtechnology.club

adrianN2• 8 years ago

The problem is that Y and Z are just numbers you make up. Reliably estimating them is impossible without at least building a prototype.

gaius3• 8 years ago

Sure, but prototypes cost orders of magnitude less than Y. And your engineers can scratch their new toy itch at zero risk.

ebiester• 8 years ago

On the other hand, “we cannot hire anyone to work in COBOL/Perl 5.8/Tcl/other outdated language” is a very real problem. It turns out that 2018, developers are judged for working too long in old technologies even when we know as in industry that a developer can learn a new language.

gaius3• 8 years ago

I wonder if that’s really true. I bet loads of people would be delighted for the chance to go on using their old favourites.

rwmj• 8 years ago

Is "rebuild" new jargon for "rewrite", or does it mean something different? I thought the article was going to be about builds failing.

ConceptJunkie2• 8 years ago

Yeah, I did too until I started reading the article.

Using the normal sense of "rebuild" didn't make sense.

ellimilial• 8 years ago

This is gold.

I've become a member of a team the company scrambled to deal with a `legacy` python/SQL - based ingestion/storage system in an effort to 'harden' it. Despite my best efforts, we are going for a full rewrite into java/spring/avro/mongo/es. We have internal users talking SQL and utilising the system at the moment, a fair amount of data is relational.

I have run out of ideas how to convince the team and stakeholders, will have a one-shot chance to talk to VP. Any ideas how to voice the concerns about the full re-design (perhaps I'm just being difficult)?

sonnyblarney• 8 years ago

1. Given the risk, cost and limited upside, the onus is on the refactor team to prove that it needs to be done. Where is the ROI, factor in the risk. Where is this in the stack of things to do? Are there better ROI things?

2. Consider 'what the point' is in the first place, because the entire world could be run on python/SQL and it would be 'hard'. I don't think anyone would consider 'Mongo' to be 'hard' usually people use it because it's fast and easy, not hard. Consider maybe only replacing one part at a time, i.e. Java-SQL.

3. Consider a simple clean up or refactor. No need to learn no languages and tools when maybe you just need a house clean.

4. People seem to be going back to SQL because of it's inherent standardization - so many reporting and analysis systems use SQL as an interface, to the point where even NoSQLs are starting to use SQL.

pedalpete• 8 years ago

I'm a big supporter of "replacing one part at a time", and wish I had done that on a rebuild I'm just completing.

In fact, I thought I was. We split our app into 3 parts, rebuilt part 1, then part 2, but part 1 couldn't be released to customers until part 2 was done, and we kept our legacy system supporting the majority of our users until we are done with part 3, which is nearing completion now.

I thought that was "replacing one piece at a time", but it isn't most users aren't touching it until part 3 is done, and at that point, they are experiencing a new system from scratch.

mratzloff• 8 years ago

Without knowing the performance requirements and where the current system is failing, it's hard to know if the technology stack will work for your needs—with one exception.

If users speak SQL, they will reject Mongo. The users of the system are the ones who will determine project success or failure.

Think about the data analysts, product owners, etc. who use the system. Interview them. Find out exactly how they use the system currently. Do they query in an ad hoc way? Do they rapidly iterate on their queries? Watch them interact with the system. If it's any way other than through dashboards that an engineer updates on request, you are in for rough seas.

Users must always determine the contours of a new system. There are big data solutions that speak SQL. Some are cloud-based, some are not. Some are faster than others. The team should be able to show you why they rejected those as solutions.

bkovacev• 8 years ago

This is yet another article where there's a clear managerial-only approach. Sorry, but I dont dig this.

As a developer you're constantly fighting managers who want to rush things to get them out and who will eventually blame you for a bug/non-defined behavior once you hit a certain milestone.

To me it seems the author of the article doesn't understand the tech debt. If you've ever worked in a startup you'd know that the requirements are ever-changing, thus that if a certain payment system is put in place, it might evolve to the point where you really need to refactor it and in order to enable the refactor you have to refactor the whole business flow as well. If there's more than 2-3 features affected by a new feature, a big refactor is definitely needed.

Only one solution offered, which I dont think is adequate because why would I leave something in that was only meant to provide value for short term and then build on top of it till I kill the old system?

LolNoGenerics• 8 years ago

His argument is against rewriting a whole codebase. Refactoring is surely an alternative.

jpeeler• 8 years ago

Firefox seems to be doing pretty well with their incremental rewrite into rust. I do wonder how long it will take to complete the transition versus doing a complete rewrite instead.

Red Flags Signaling That a Rebuild Will Fail

Related Stories