I think it's worth remembering that no matter how complex systems are, at the heart of it, we can understand them. And it's fundamentally our job to do that, the rest is just transmuting understanding into something useful.
I think this is only true for systems with proper design and abstractions. Imagine a system where you add a log line somewhere in an innocuous place and the entire system's behavior changes (something I have seen and experienced). Yes, it can be understood, but you attach a debugger and the system behaves in yet another different way. Now imagine every single thing in the system works like this. Given 10 years, I could understand that system. But given a deadline and a time to market, I sigh, remove the log line and ship it and hope that the next generation product does it better.
Long story short: Everything can be understood, but not within a reasonable time frame.
I've had systems like that. It's usually paid off to understand them. It's made it easier to modify them later, more reliable when I was on call for them, and when they invariably broke, it was much less stressful. I had eliminated a huge number of the questions that needed to be asked to get them fixed quickly before everything was on fire, so I could find the right place to fix them with more confidence.
Taking the time to understand the weird behaviors of systems is the most important part of being able to get things running smoothly and iterating quickly.
And the more you do it, the easier it gets. The best I've seen is someone going from a filesystem crash to reading the disassembly for a mutex to fixing a compiler bug in 15 minutes flat. They had a lot of practice with reading code.
Beyond that, in a long-lived system, the person who takes the time to understand how the code is, then more time to understand how it should be, then even more time to make it actually be that way, can be an unsung (or even actively disparaged) hero for the future friction they eliminate. (Commonly known as "refactoring".)
That's horrifying. Does the behaviour changing by adding logging happen due to code exhibiting undefined behaviour, or really bad race conditions, or were there other causes for it too?
This really speaks to me, especially the Lambda example which mirrors my experience. But let me point out that my experience also shows your Lambda function will fail in a way that you can't begin to understand without opening those same 30 documentation tabs. So sure, prioritize getting something running, but don't imagine you're off the hook at that point.
This is one reason open source is such a powerful principle: it makes it easy to dive into the details when you really need to. A while back there was a gap in the PostgreSQL documentation that someone at work ran into (which is a rare occurence…thank you, PG maintaners!). They were flabbergasted when I opened the PG repo, found the code that looked at the option, and told them definitively what it does.
But in a culture where just expecting someone to read the documentation may already be a stretch, expecting someone to read the code may just be an dead artifact of my long and much-storied career.
[OP] orib | 9 hours ago
I think it's worth remembering that no matter how complex systems are, at the heart of it, we can understand them. And it's fundamentally our job to do that, the rest is just transmuting understanding into something useful.
BenjaminRi | 7 hours ago
I think this is only true for systems with proper design and abstractions. Imagine a system where you add a log line somewhere in an innocuous place and the entire system's behavior changes (something I have seen and experienced). Yes, it can be understood, but you attach a debugger and the system behaves in yet another different way. Now imagine every single thing in the system works like this. Given 10 years, I could understand that system. But given a deadline and a time to market, I sigh, remove the log line and ship it and hope that the next generation product does it better.
Long story short: Everything can be understood, but not within a reasonable time frame.
[OP] orib | 6 hours ago
I've had systems like that. It's usually paid off to understand them. It's made it easier to modify them later, more reliable when I was on call for them, and when they invariably broke, it was much less stressful. I had eliminated a huge number of the questions that needed to be asked to get them fixed quickly before everything was on fire, so I could find the right place to fix them with more confidence.
Taking the time to understand the weird behaviors of systems is the most important part of being able to get things running smoothly and iterating quickly.
And the more you do it, the easier it gets. The best I've seen is someone going from a filesystem crash to reading the disassembly for a mutex to fixing a compiler bug in 15 minutes flat. They had a lot of practice with reading code.
wrs | an hour ago
Beyond that, in a long-lived system, the person who takes the time to understand how the code is, then more time to understand how it should be, then even more time to make it actually be that way, can be an unsung (or even actively disparaged) hero for the future friction they eliminate. (Commonly known as "refactoring".)
BenjaminRi | 30 minutes ago
You did not have a system like that if you understood it eventually.
0x2ba22e11 | 4 hours ago
That's horrifying. Does the behaviour changing by adding logging happen due to code exhibiting undefined behaviour, or really bad race conditions, or were there other causes for it too?
scally | 2 hours ago
I would even go so far as to say the job is to understand and enhance understanding, not to write code.
Anyone and anything can generate code. There is little value to code without understanding.
wrs | an hour ago
This really speaks to me, especially the Lambda example which mirrors my experience. But let me point out that my experience also shows your Lambda function will fail in a way that you can't begin to understand without opening those same 30 documentation tabs. So sure, prioritize getting something running, but don't imagine you're off the hook at that point.
This is one reason open source is such a powerful principle: it makes it easy to dive into the details when you really need to. A while back there was a gap in the PostgreSQL documentation that someone at work ran into (which is a rare occurence…thank you, PG maintaners!). They were flabbergasted when I opened the PG repo, found the code that looked at the option, and told them definitively what it does.
But in a culture where just expecting someone to read the documentation may already be a stretch, expecting someone to read the code may just be an dead artifact of my long and much-storied career.
carlana | 32 minutes ago