I’m always a bit baffled how someone can see a system with business logic they don’t understand and think “rearchitecting it will make it clear”.
To rearchitect it, you have to understand it. The difficulty is that rearchitecting something sounds sophisticated, while understanding the function of an existing system sounds like grunt work.
Once you do understand it, you’ll hopefully start to have ideas for rearchitecting things, but most of the rewriting crowd never reaches that point.
On the other hand, the most efficient way to understand the business logic might not be to study the code base. It's legacy after all, lots of noise along the signal.
Now if there's no documentation and the first devs are gone...
It can make sense to start elsewhere, but in this kind of system, the running code is as important, if not more important than any documentation or intent. It’s what serves transactions, changes must be evaluated based on how they’ll affect it.
The documentation is probably not in line with the code, and the assumption should be that they’re both wrong.
Agreed. I suggest adding probes to validate logic. Dumping to some wide event store like honeycomb. Then you can cave off individual pieces, run refactored code in parallel (but as a no-op) to validate the refactor is understood. Making sure to add unit tests and integration tests.
Even if it's "only" a refactor the resulting tests and confidence allow for future rearchitecting.
That’s why a piecemeal replacement approach is almost always better: it makes more business sense because you’re not off into the void doing a rewrite, and you can replace bits on which you’ve already got a good grasp.
You announce the old thing is irrecoverably drowning in tech debt. You spin up a team to rewrite it from scratch. Work begins.
Meanwhile the old thing remains a moving target: it's running the core business, so changes are still necessary. The developers working on it know that it's going to be made obsolete by the new thing soon, so they don't have any incentive to go beyond the smallest effort possible to add the new features. Technical debt continues to mount.
Meanwhile, the team working on the new thing are ambitious and probably a little naive. They start out at a great pace - it's greenfield after all - but as time progresses it becomes apparent that nobody fully understands the behavior and scope of the thing they are replacing. If it was well documented and tested it wouldn't need to be replaced, after all...
After months (or even years) without delivering value, the pressure is on to "ship it", so the new system is launched to handle a subset of what the old system handled - or often for some new feature that was too hard to build with the now mostly unmaintained old system.
... so now you have TWO systems in production - the janky old system that nobody wants to touch, and a new system which handles just a few production features and is 80% inactive code that is meant to replace the old system, eventually.
If you're really lucky the company won't have lost patience with the new system and will allow that work to continue. The longer this all takes, and the longer the old system stays in production and stubbornly continues to work, the higher the risk that "priorities have changed" and the new system total replacement work is abandoned, leaving you with two systems where you used to have one.
If I run into a situation like this in the future, my strong recommendation is going to be shoring up the old system with as much automated testing as possible and then seeing if targeted refactors can get it to the desired shape. My hunch is that in many cases that will have a much higher chance of success than the siren call of a greenfield replacement.
Everything that you’ve said was true, but only because you did the “ burn it all to the ground” step second. If you destroy the old system before the rewrite, then people can’t say that your system is wrong for doing it differently.
Obviously, this is not a serious suggestion, but I’ve seen too many green field replacements scuttled because they didn’t replicate behaviour that was wrong the first time. For example, heard a talk by a fellow who dealt with a bug where the sales tax charged to customers was higher than the actual sales tax and the company pocketed the difference. The company lawyers and accountants both confirmed that the new code was correct and the old code was committing tax fraud, but his manager insisted on replicating the old calculations because of Chesterton’s fence. Since no one in the company would explain why they had been engaged in a felony for decades, the coder didn’t know why the system acted the way it did and should not change a system that he didn’t understand.
Hah, I just realized the linked article covers the same ground:
The closest option that a mega-corp such as Amazon has is what I call a side-channel, where they split off a team that builds a new, completely disconnected system with only the minimal set of features needed for some new use case. Moving forward they then have the option to direct more new use-cases at this simplified, separate system. Importantly, the old system must remain and be maintained (it's not an 'end'), because all the old use-cases still exist, and new organizational-level pain is felt whenever deciding which to use in the future
I feel like this is a common belief, that rewrites hardly ever work, but I think it's just because we're biased towards remembering the ones that fail really horribly. I think people rewrite systems successfully all the time. I've certainly done it plenty of times. But I think it's heavily dependent on domain, the size of the system, the reasons for the rewrite, etc. I just don't like discouraging people from cleaning up messes :)
You announce the old thing is irrecoverably drowning in tech debt. You spin up a team to rewrite it from scratch. Work begins.
I've actually done this successfully once - legacy system built on a now unsupported platform and language, technical debt declared to be insurmountable, new system from scratch. I was very worried about it and recommended exploring the strangling fig pattern. We did some of this - some subsystems were used by both systems, but mostly it was a newly built. This was a sizeable system with complex user interface requirements and some tricky business rules, including a configurable workflow system.
We had a few things going for us:
the main dev of the original system was on the new team and willing to listen to others and be open minded about it.
we had stakeholders with a deep understanding of the requirements in the room (including the main dev I just mentioned)
we had someone who really helped us gel into a properly functioning, actually collaborating team rather than individuals throwing PRs over the wall.
if I may toot my own horn, I'm experienced, helped mentor and offered technical leadership. Strong focus on keeping the new system as well defined and declarative as possible.
It wasn't easy, but we had good working conditions. It took a long time of steady incremental progress with business support. And I'm pretty sure that even years after this, the older implementation is still running too, but I could be wrong.
I'd read lots of stuff about avoiding total rewrites before I started, which is why I was so wary. But it can be done. We were lucky too: I don't think it's easy to replicate the things I listed that made it work.
Anecdotally I've seen cases of this unfolding multiple times, leading to a layer cake of partial replacement systems, reflecting different decades of the latest hot technology, dating back to a primordial system builtin COBOL. They were underway a third time replacing everything with microservices in k8s. I didn't see it through to the end but I have no doubt the powers will be will bail half way through this time as well as the billable hours keep mounting.
When things get that bad, it's generally easier/less stressful to find a new job than push such an idea through whatever organization made the mess.
There's a pithy law of software whose name I forget that goes (roughly) "software architecture mimics the organization that created it." The kind of organization that can agree on how to burn things down and start anew without killing the product is precisely the kind of organization that won't need to.
I've strolled through a few "haunted graveyards". Once you've seen a few you can tell which systems have been worked by many hands without a definite owner.
Fundamentally, domain experience and continuity result in simpler, tighter systems. People want to contribute, usually by adding features. Each feature adds constraints and invariants, which dilutes a cohesive vision of the system. Accretion is easy, cutting things out is much harder to justify, so washing a system through multiple people reduces conceptual integrity.
A "fix everything now" mentality doesn't work. Climbing into a local maximum by simple changes does work: rename variables, comment discoveries in the code itself, document and automate workflows so the knowledge survives, and write tests to discover behavior and lock it in when you make changes. Once you start understanding things and have improved automated testing, you can make larger, safer changes.
The two biggest references to read about this are "Working Effectively with Legacy Code" by Feathers, and the much less known Object-Oriented Reengineering Patterns.
The problem is as soon as you start putting out the fires, people hear about progress and want to jump into this "safer" codebase and start adding features again.
"Where does it end? How does it end?"
...development velocity approaches zero because nothing can be shipped without a breakage.
The systems usually don't "collapse". The cost of code changes grows, and the speed and correctness of changes erode until people avoid it. Behavior gets put outside the system, and the problem slowly infects the code that interfaces with it, and it compounds as turnover affects those teams.
hyperpape | 22 hours ago
I’m always a bit baffled how someone can see a system with business logic they don’t understand and think “rearchitecting it will make it clear”.
To rearchitect it, you have to understand it. The difficulty is that rearchitecting something sounds sophisticated, while understanding the function of an existing system sounds like grunt work.
Once you do understand it, you’ll hopefully start to have ideas for rearchitecting things, but most of the rewriting crowd never reaches that point.
Loup-Vaillant | 18 hours ago
On the other hand, the most efficient way to understand the business logic might not be to study the code base. It's legacy after all, lots of noise along the signal.
Now if there's no documentation and the first devs are gone...
hyperpape | 16 hours ago
It can make sense to start elsewhere, but in this kind of system, the running code is as important, if not more important than any documentation or intent. It’s what serves transactions, changes must be evaluated based on how they’ll affect it.
The documentation is probably not in line with the code, and the assumption should be that they’re both wrong.
schneems | 11 hours ago
Agreed. I suggest adding probes to validate logic. Dumping to some wide event store like honeycomb. Then you can cave off individual pieces, run refactored code in parallel (but as a no-op) to validate the refactor is understood. Making sure to add unit tests and integration tests.
Even if it's "only" a refactor the resulting tests and confidence allow for future rearchitecting.
But it's easier said than done.
sjamaan | 15 hours ago
That’s why a piecemeal replacement approach is almost always better: it makes more business sense because you’re not off into the void doing a rewrite, and you can replace bits on which you’ve already got a good grasp.
gunduzc | a day ago
Perhaps I'm just a naive youngster, but I feel like sometimes it's necessary to burn it all to the ground and start from scratch.
(In business contexts, the starting from would scratch have to happen before ditching the old codebase, I'm assuming.)
simonw | a day ago
In my experience it's so rare for that to work.
You announce the old thing is irrecoverably drowning in tech debt. You spin up a team to rewrite it from scratch. Work begins.
Meanwhile the old thing remains a moving target: it's running the core business, so changes are still necessary. The developers working on it know that it's going to be made obsolete by the new thing soon, so they don't have any incentive to go beyond the smallest effort possible to add the new features. Technical debt continues to mount.
Meanwhile, the team working on the new thing are ambitious and probably a little naive. They start out at a great pace - it's greenfield after all - but as time progresses it becomes apparent that nobody fully understands the behavior and scope of the thing they are replacing. If it was well documented and tested it wouldn't need to be replaced, after all...
After months (or even years) without delivering value, the pressure is on to "ship it", so the new system is launched to handle a subset of what the old system handled - or often for some new feature that was too hard to build with the now mostly unmaintained old system.
... so now you have TWO systems in production - the janky old system that nobody wants to touch, and a new system which handles just a few production features and is 80% inactive code that is meant to replace the old system, eventually.
If you're really lucky the company won't have lost patience with the new system and will allow that work to continue. The longer this all takes, and the longer the old system stays in production and stubbornly continues to work, the higher the risk that "priorities have changed" and the new system total replacement work is abandoned, leaving you with two systems where you used to have one.
The best article I've read about completing this process responsibly is Migrations: the sole scalable fix to tech debt by Will Larson.
If I run into a situation like this in the future, my strong recommendation is going to be shoring up the old system with as much automated testing as possible and then seeing if targeted refactors can get it to the desired shape. My hunch is that in many cases that will have a much higher chance of success than the siren call of a greenfield replacement.
rprospero | 20 hours ago
Everything that you’ve said was true, but only because you did the “ burn it all to the ground” step second. If you destroy the old system before the rewrite, then people can’t say that your system is wrong for doing it differently.
Obviously, this is not a serious suggestion, but I’ve seen too many green field replacements scuttled because they didn’t replicate behaviour that was wrong the first time. For example, heard a talk by a fellow who dealt with a bug where the sales tax charged to customers was higher than the actual sales tax and the company pocketed the difference. The company lawyers and accountants both confirmed that the new code was correct and the old code was committing tax fraud, but his manager insisted on replicating the old calculations because of Chesterton’s fence. Since no one in the company would explain why they had been engaged in a felony for decades, the coder didn’t know why the system acted the way it did and should not change a system that he didn’t understand.
simonw | a day ago
Hah, I just realized the linked article covers the same ground:
greysonp | 16 hours ago
I feel like this is a common belief, that rewrites hardly ever work, but I think it's just because we're biased towards remembering the ones that fail really horribly. I think people rewrite systems successfully all the time. I've certainly done it plenty of times. But I think it's heavily dependent on domain, the size of the system, the reasons for the rewrite, etc. I just don't like discouraging people from cleaning up messes :)
faassen | 18 hours ago
I've actually done this successfully once - legacy system built on a now unsupported platform and language, technical debt declared to be insurmountable, new system from scratch. I was very worried about it and recommended exploring the strangling fig pattern. We did some of this - some subsystems were used by both systems, but mostly it was a newly built. This was a sizeable system with complex user interface requirements and some tricky business rules, including a configurable workflow system.
We had a few things going for us:
the main dev of the original system was on the new team and willing to listen to others and be open minded about it.
we had stakeholders with a deep understanding of the requirements in the room (including the main dev I just mentioned)
we had someone who really helped us gel into a properly functioning, actually collaborating team rather than individuals throwing PRs over the wall.
if I may toot my own horn, I'm experienced, helped mentor and offered technical leadership. Strong focus on keeping the new system as well defined and declarative as possible.
It wasn't easy, but we had good working conditions. It took a long time of steady incremental progress with business support. And I'm pretty sure that even years after this, the older implementation is still running too, but I could be wrong.
I'd read lots of stuff about avoiding total rewrites before I started, which is why I was so wary. But it can be done. We were lucky too: I don't think it's easy to replicate the things I listed that made it work.
marginalia | 21 hours ago
Anecdotally I've seen cases of this unfolding multiple times, leading to a layer cake of partial replacement systems, reflecting different decades of the latest hot technology, dating back to a primordial system builtin COBOL. They were underway a third time replacing everything with microservices in k8s. I didn't see it through to the end but I have no doubt the powers will be will bail half way through this time as well as the billable hours keep mounting.
jarofgreen | 14 hours ago
You might find Kill It With Fire: Managing Aging Computer Systems (And Future Proof Modern Ones) by Marriane Bellotti interesting
gunduzc | 13 hours ago
Thank you for the recommendation, I love the title.
landon | 7 hours ago
When things get that bad, it's generally easier/less stressful to find a new job than push such an idea through whatever organization made the mess.
There's a pithy law of software whose name I forget that goes (roughly) "software architecture mimics the organization that created it." The kind of organization that can agree on how to burn things down and start anew without killing the product is precisely the kind of organization that won't need to.
pyj | 20 hours ago
I've strolled through a few "haunted graveyards". Once you've seen a few you can tell which systems have been worked by many hands without a definite owner.
Fundamentally, domain experience and continuity result in simpler, tighter systems. People want to contribute, usually by adding features. Each feature adds constraints and invariants, which dilutes a cohesive vision of the system. Accretion is easy, cutting things out is much harder to justify, so washing a system through multiple people reduces conceptual integrity.
A "fix everything now" mentality doesn't work. Climbing into a local maximum by simple changes does work: rename variables, comment discoveries in the code itself, document and automate workflows so the knowledge survives, and write tests to discover behavior and lock it in when you make changes. Once you start understanding things and have improved automated testing, you can make larger, safer changes.
The two biggest references to read about this are "Working Effectively with Legacy Code" by Feathers, and the much less known Object-Oriented Reengineering Patterns.
The problem is as soon as you start putting out the fires, people hear about progress and want to jump into this "safer" codebase and start adding features again.
The systems usually don't "collapse". The cost of code changes grows, and the speed and correctness of changes erode until people avoid it. Behavior gets put outside the system, and the problem slowly infects the code that interfaces with it, and it compounds as turnover affects those teams.
baetylboy | 20 hours ago
What needs to happen at the org/management level in order to properly encourage a real fix for this, or make sure it never happens in the first place?