I think it's worth remembering that no matter how complex systems are, at the heart of it, we can understand them. And it's fundamentally our job to do that, the rest is just transmuting understanding into something useful.
I think this is only true for systems with proper design and abstractions. Imagine a system where you add a log line somewhere in an innocuous place and the entire system's behavior changes (something I have seen and experienced). Yes, it can be understood, but you attach a debugger and the system behaves in yet another different way. Now imagine every single thing in the system works like this. Given 10 years, I could understand that system. But given a deadline and a time to market, I sigh, remove the log line and ship it and hope that the next generation product does it better.
Long story short: Everything can be understood, but not within a reasonable time frame.
I've had systems like that. It's usually paid off to understand them. It's made it easier to modify them later, more reliable when I was on call for them, and when they invariably broke, it was much less stressful. I had eliminated a huge number of the questions that needed to be asked to get them fixed quickly before everything was on fire, so I could find the right place to fix them with more confidence.
Taking the time to understand the weird behaviors of systems is the most important part of being able to get things running smoothly and iterating quickly.
And the more you do it, the easier it gets. The best I've seen is someone going from a filesystem crash to reading the disassembly for a mutex to fixing a compiler bug in 15 minutes flat. They had a lot of practice with reading code.
Beyond that, in a long-lived system, the person who takes the time to understand how the code is, then more time to understand how it should be, then even more time to make it actually be that way, can be an unsung (or even actively disparaged) hero for the future friction they eliminate. (Commonly known as "refactoring".)
My comment explained that there are systems that take so long to understand and fix that it's not worth it - nobody will pay this kind of money for a skilled engineer to sink man-years into an aging system. Your response "It's usually paid off to understand them." is essentially just a dismissal of that point. The reality is that even if you were technically correct, the market will not pay for it. I love to understand and fix everything, but dogma does not suspend reality.
That's horrifying. Does the behaviour changing by adding logging happen due to code exhibiting undefined behaviour, or really bad race conditions, or were there other causes for it too?
This really speaks to me, especially the Lambda example which mirrors my experience. But my experience also is that your Lambda function will fail in a way that you can't begin to understand without opening those same 30 documentation tabs. So sure, prioritize getting something running, but don't imagine you're off the hook at that point.
This is one reason open source is such a powerful principle: it makes it easy to dive into the details when you really need to. A while back there was a gap in the PostgreSQL documentation that someone at work ran into (which is a rare occurence…thank you, PG maintaners!). They were flabbergasted when I opened the PG repo, found the code that looked at the option, and told them definitively what it does.
But in a culture where just expecting someone to read the documentation may already be a stretch, expecting someone to read the code may just be a dead artifact of my long career that started before we had a Google, Stack Overflow, and LLMs to supply answers.
How does tetris work? Well, how many layers deep do you want to go? It's possible to go all the way to "*handwave* it's physics" if you're determined, but it's going to take you on a journey thru parsers, virtual machines, stack languages, registers, arithmetic logic units, flip flops, and boolean gates. (For some values of "tetris" anyway.) For every layer, you can peel it back to see what's beneath, or in the ordering of the book, build it from what you built in the last chapter.
[OP] orib | a month ago
I think it's worth remembering that no matter how complex systems are, at the heart of it, we can understand them. And it's fundamentally our job to do that, the rest is just transmuting understanding into something useful.
BenjaminRi | a month ago
I think this is only true for systems with proper design and abstractions. Imagine a system where you add a log line somewhere in an innocuous place and the entire system's behavior changes (something I have seen and experienced). Yes, it can be understood, but you attach a debugger and the system behaves in yet another different way. Now imagine every single thing in the system works like this. Given 10 years, I could understand that system. But given a deadline and a time to market, I sigh, remove the log line and ship it and hope that the next generation product does it better.
Long story short: Everything can be understood, but not within a reasonable time frame.
[OP] orib | a month ago
I've had systems like that. It's usually paid off to understand them. It's made it easier to modify them later, more reliable when I was on call for them, and when they invariably broke, it was much less stressful. I had eliminated a huge number of the questions that needed to be asked to get them fixed quickly before everything was on fire, so I could find the right place to fix them with more confidence.
Taking the time to understand the weird behaviors of systems is the most important part of being able to get things running smoothly and iterating quickly.
And the more you do it, the easier it gets. The best I've seen is someone going from a filesystem crash to reading the disassembly for a mutex to fixing a compiler bug in 15 minutes flat. They had a lot of practice with reading code.
wrs | a month ago
Beyond that, in a long-lived system, the person who takes the time to understand how the code is, then more time to understand how it should be, then even more time to make it actually be that way, can be an unsung (or even actively disparaged) hero for the future friction they eliminate. (Commonly known as "refactoring".)
BenjaminRi | a month ago
You did not have a system like that if you understood it eventually.
[OP] orib | a month ago
Is that a tautology?
BenjaminRi | a month ago
My comment explained that there are systems that take so long to understand and fix that it's not worth it - nobody will pay this kind of money for a skilled engineer to sink man-years into an aging system. Your response "It's usually paid off to understand them." is essentially just a dismissal of that point. The reality is that even if you were technically correct, the market will not pay for it. I love to understand and fix everything, but dogma does not suspend reality.
0x2ba22e11 | a month ago
That's horrifying. Does the behaviour changing by adding logging happen due to code exhibiting undefined behaviour, or really bad race conditions, or were there other causes for it too?
scally | a month ago
I would even go so far as to say the job is to understand and enhance understanding, not to write code.
Anyone and anything can generate code. There is little value to code without understanding.
wrs | a month ago
This really speaks to me, especially the Lambda example which mirrors my experience. But my experience also is that your Lambda function will fail in a way that you can't begin to understand without opening those same 30 documentation tabs. So sure, prioritize getting something running, but don't imagine you're off the hook at that point.
This is one reason open source is such a powerful principle: it makes it easy to dive into the details when you really need to. A while back there was a gap in the PostgreSQL documentation that someone at work ran into (which is a rare occurence…thank you, PG maintaners!). They were flabbergasted when I opened the PG repo, found the code that looked at the option, and told them definitively what it does.
But in a culture where just expecting someone to read the documentation may already be a stretch, expecting someone to read the code may just be a dead artifact of my long career that started before we had a Google, Stack Overflow, and LLMs to supply answers.
technomancy | a month ago
My favorite proof-by-example of this is nand2tetris: https://www.nand2tetris.org/
How does tetris work? Well, how many layers deep do you want to go? It's possible to go all the way to "*handwave* it's physics" if you're determined, but it's going to take you on a journey thru parsers, virtual machines, stack languages, registers, arithmetic logic units, flip flops, and boolean gates. (For some values of "tetris" anyway.) For every layer, you can peel it back to see what's beneath, or in the ordering of the book, build it from what you built in the last chapter.
carlana | a month ago