Worth noting that speed doesn't have to be your only goal. You can do things like "oh there's a bugfix here, let's spin this off into a preparatory commit that can be reviewed in isolation". Or "I don't like this type structure because it permits some invalid states to be representable; let's go fix that". Or "okay, I think the testing story doesn't quite give me enough confidence, let's spend some time prototyping more advanced techniques like PBT/fuzzing/formal methods". These are all things I'd have clubbed into the same commit or left as a TODO tech debt comment in the past. None of them are related to speed, except in the most general sense of correct software saving you time down the road. The marginal cost of doing things right is now (stunningly) lower.
LLMs are very open-ended as tools, and the value you get out of them mirrors the values you bring to them.
(edit: there are also risks to building too many half-finished prototypes that never see their way to production. So it's not all good. But I think overall this is a huge boon for rigor-focused engineering.)
I often find myself emphatically agreeing when I read ai takes like this. However another voice in my head mentions that this isn’t what’s happening in most companies. While it absolutely can increase rigor, more often then not I see it being used to race to the lowest quality MVP possible. Perhaps that reflects more on the company culture rather than LLMs but it’s still disappointing to see. I hope the industry will get its head on straight and produce higher quality software.
Using coding agents to build higher quality software - as opposed to cutting corners and churning out faster slop - is definitely a decision companies need to make.
I can spin a clanker with a single prompt in my historical projects, and it will start finding real bugs without even trying hard. I know because it keeps happening.
Guess what? Humans are sloppy too. I am sloppy. Mistakes happen. LLMs just bring it upfront because they work so fast and are quite a bit dumber in many respects. Mistakes compound, so if you just naively let clanker make code changes, you're going to have a bad time real fast.
But it's lazy to think that just because naive use of the tool is flaky, the tool is not useful.
My current go to slop-flow is actually getting 5 specialized reviews, including architecture, maintainability, reliability/security, etc. automatically after every change being made, and I have a whole design documentation system to keep things organized and agent's decision making improves significantly Is it perfect? No. Is it better than naive approach? Sure it is. Can it be done better? I hope. And that's also the fun part. A new weird tool in my toolbox to play with.
I am also skeptical about whether it makes the whole process - development, review and finally deploy faster; I find that sometimes I might generate something faster with a prompt, but then it takes more time to validate and understand to the extent that I find myself wondering - wouldn't it have been faster to just write by hand in the first place? And arguing that it depends on the case... judging also takes time and energy, that I would rather spend on something else.
But, as long as the person submitting the PR takes ownership I am agnostic about their LLM use - it simply does not matter as far as quality, correctness, consistency and all other metrics we find important are concerned. If it is better/faster for you to implement with the LLM, do it, if not, do it on your own - the ownership stays the same, it is on the human author.
As far as I personally am concerned, I love AI for learning and extending & deepening my understanding - but I do write most of the code manually, since I still find it more productive - all things considered, not only the typing part.
I think pretty much anyone who actually is optimistic about LLM’s and coding would agree that the last six months and the last three months specifically have significantly improved the utility of these. Do you actually use the latest for-pay cloud models? Do you have controls in place that prevent entire categories of failure, such as the principle I developed out of having worked in the auditor space, MFIC? https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8c8d7e Do you have tooling that helps keep already-developed code and project structure in context such as https://github.com/pmarreck/dirtree and https://github.com/pmarreck/codescan ? (Note that these are also useful to forgetful human developers or those who are or have become unfamiliar with a codebase!)
Anyway, either you make it work and enjoy the benefits, or you rebel and continue to write bespoke hand-coded things (while ALSO frequently writing bugs and security holes, btw) while others (perhaps writing occasionally flakier code, but far faster) pass you by.
Lastly, speaking as someone who spent 2 man-years working on a million-LOC Ruby on Rails codebase for Desk.com that is now retired, corpo code is ephemeral, which lines up nicely with LLM-produced code.
In your case, working on Erlang internals, it makes admittedly less sense to trust it at all, but I don’t see the argument against letting it write at least things function-by-function that you then review- you’d obviously know what’s subpar, but I bet you’d be pleasantly surprised SOME of the time.
You can work with the knitting needle or the loom. Perhaps “both” is ideal, depending on the context.
My problem with LLM Coding Assistants is that I cannot see, in the face of the scientific evidence, how they can help someone write code better or faster.
It's hard to take pieces like this seriously when your own personal experience - and that of so many other credible, experienced developers who you spend time with - has shown how they help someone write code better and faster, time and time again.
"Show me the peer reviewed study". I don't need to. I've seen enough already.
"But the METR study says developers over-estimate the impact of AI tools on their own productivity". I'm happy for you that could find one study that reaffirms your priors.
That is the standard of proof we have for pretty much all other programming practices. We didn't adopt git or rust because there was a peer reviewed study proving that it would increase productivity.
Plus, if I could find a peer reviewed study claiming that using git or rust was bad for my productivity then I would ignore it, because it would be wrong. Most studies of programming practice are pretty terrible anyway and obviously won't generalize from 10 undergrads with an hour of experience in a tool to a large company of experienced professionals with years of training.
Feel free to take that into account when evaluating my work from now going forward.
I do not think peer-reviewed academic papers are the gold standard in helping us understand how software engineering works. The incentives in that system produce weird, hard to follow results - at a glacial pace.
There's a big difference between academic papers that help understand physics or biology - where you really do need rigorous research experimental processes and benefit from peer-review - and fields like software engineering where you can do the work yourself.
How many practicing software engineers do you know who read papers about software engineering more than once or twice a year, if that?
How many practicing software engineers do you know who read papers about software engineering more than once or twice a year, if that?
That argument would have more weight if software was well engineered in general; do you considered it is? What was considered a supercomputer when I was young is barely enough to render most websites, even when actual content is effectively just text. I think we can do better.
I think software today compared to software ten or twenty years ago is enormously better, and lets me get way more done.
Do you think software engineers don't read papers because they are lazy and bad at their jobs, or do you think it's because they've tried spending time reading papers and found them not to provide good enough ROI on that time?
I exist in the glacial space you describe (though, blessfully, not in human aspects research). The interpretation that I and many colleagues hold is that software has, in general, become more fault tolerant (in the sense of avoiding catastrophic failure), but simultaneously more defective. The incentive structure in software development is currently far more favourable to develop quickly and fix later strategies, largely due to a lack of accountability mechanisms for the companies selling software or software services.
My hunch is that it only applies to a very narrow aspect of programming (which is usually not mentioned). As an example, I never found a good definition of "unit test" that can apply across a wide range of languages and when I ask for a definition, I'm scolded for asking. As such, I've always found it hard to apply to the stuff I write in assembly or C. Unit testing seems to be much better for a language like Java or Javascript, which I don't use.
I have to avoid the term "unit test" in the same way that I avoid "TDD"!
Unit tests carry all sorts of baggage around only testing a single "unit" (whatever that means), which inevitably leads to a growing pile of mocks.
I prefer to avoid mocks as much as I can - I like tests that are as close to answering the question "does the feature work?" as possible.
So I'm stuck with "automated tests", because I don't want to get into the weeds concerning if something is an integration test or a unit test or a smoke test or whatever other terms people want to use.
Those approaches do have specific subgenres. Your idea is close to the classicist/Detroid style testing. As opposed to the mockist/London style. It's a shame those labels are not commonly used. Same with Beck's version of "unit of functionality" vs corporate Java "public method is a unit".
Fair enough, academic publishing has a lot of issues, and to be honest I know next to nothing about the academic field of software engineering per se (I'm an amateur!). However I don't think personal experience is worth more than (imperfect) attempts at collecting more data than mere testimonies and other forms of subjective experience. In the field I know vaguely more about (human health), practitioners (clinicians) don't have to read papers to benefit from whatever researchers are finding out, it trickles down to them in various ways. It isn't perfect, but it's basically all we have besides superstition. IMHO we would have better healthcare overall if personal experience was drastically more critically scrutinized by doctors and patients (us!). Which is why that "I've seen enough already" really made me choke a little. Maybe drawing a parallel between medicine and software engineering is ridiculous though. :)
A good researcher would recognize existing studies seem to be leaving a lot out and critique them accordingly. Blindly accepting studies is not what a researcher does.
You mean, the same that doctors use to explain why they don't wash hands?
Or the one they used to defend bloodletting?
Science literally was built on the bodies of the people we killed by trusting "own personal experience".
Note that I demand at the end to be proven wrong. Show us, in away that is robust and methodologically sound, where it differs from what we have found with robust empirical research.
I am ready to change my mind. But, I am sorry, "own personal experience" will not cut it. Nor a dozen anecdotes that explain nothing but just tell us we are wrong.
It is how we killed thousands of people in the past. Science and engineering allowed us to make the life of billions of people better. Sorry that I will keep defend them, even if it hurts your own personal experience.
This is not about the METR stuff. It is about a pattern over centuries of this happening. We developed science and engineering to counter that and it works. Sorry, I will not throw it away because you think you are better. I am too humble for that, and I need to sleep at night.
What was the last peer-reviewed paper you read about the craft of building software that positively influenced you and helped you become a better software developer?
That is a good question. Hmm... Probably research on TDD, I will bring it up once I am back from travel. Mostly research on TDD at this point shows no real impact, hence why I do not push it hard.
That's a fun example because it goes directly against my own personal experience, too!
I'm a firm believer in automated testing because I've seen directly how much more productive it makes me - because I can ship a new feature for a project I haven't touched in years without having to manually check that I haven't broken any of the existing, documented behavior.
I'm quite confident I could find holes in any paper that attempted to prove otherwise.
(Test-first development, on the other hand, has failed to convince me. I've tried writing the tests first and found it to slow me down without improving the quality of my code... until the last six months where I've found forcing the coding agents to write their tests first appears to produce less hallucinations and code that's better factored and more likely to pass review - anecdotally though.)
That's why qualified I qualified "test first" development in my parenthetical at the end.
I know from experience that "TDD" is an ambiguous term - some people use it to describe red/green test-first development, but it's often used as a label for any kind of automated testing.
So in my reply I deliberately used the less ambiguous terms "automated testing" and "test-first" - precisely because I do understand the difference.
[in my experience] it's often used as a label for any kind of automated testing
I'm a fan of your work and believe you make this claim honestly but... this seems absurd. Who is using "TDD" to describe all automated testing in general? That seems like an abuse and misunderstanding of terminology rather than a legitimate difference in opinion.
In my recent round of TDD clarifications, one surprising experience is that folks out there don’t agree on the definition of TDD. I made it as clear as possible in my book. I thought it was clear. Nope. My bad.
What common term or acronym would you use to describe automated tests that might or might not be written test-first?
This is a major source of frustration for me. I dislike that the acronym TDD, in its classic definition, embeds test-first which I think distracts from the value of automated testing generally.
In my own writing I avoid saying "TDD", so I'm stuck with "automated testing" (the word "testing" by itself overlaps with manual testing) instead.
So any time an online discussion mentions TDD I feel compelled to clarify if they mean test-first. Which is what I did here.
If I could wave a magic wand I would redefine TDD to mean development that incorporates automated testing, and enforce TFD for the less useful test-first development pattern.
Thanks for clarifying. However, I'm not sure that your quotation supports the claim that there are folks out there using "TDD" to refer to automated testing in general. In my view, TDD refers to a development workflow, which describes the points in the development lifecycle that involve writing/running automated tests. Different people can and do disagree about what the specifics of this workflow are, as your Kent Beck quotation illustrates, but invariably it refers to a kind of development workflow. On the other hand, automated testing is closer to a component of a project, which implements automated validation of the other components.
To be clear, I do appreciate that you're being explicit about what TDD means--I just remain uncertain that anyone defines it as encompassing all automated testing.
What common term or acronym would you use to describe automated tests that might or might not be written test-first?
I don't have one. I just say "automated testing", or even just "testing"--qualifying "manual testing" explicitly when needed. To me, automated testing is something that most serious projects practice and view as useful (with some notable exceptions, of course--I recall the SumatraPDF developer doesn't see much utility in them), so I rarely need to argue for its inclusion. TDD, on the other hand, is far more controversial, being a particular development workflow.
If I could wave a magic wand I would redefine TDD to mean development that incorporates automated testing, and enforce TFD for the less useful test-first development pattern.
Again, I can always appreciate the goal to be explicit, but I'm not so sure that TDD is a poor choice of term. Certainly TFD is a clearer one, but I wouldn't be in favor of repurposing TDD to describe the use of automated testing. I write a lot of automated tests, but I would not characterize my process as test-driven; they are supplementary.
I'm not @Diana, and it's not a (or several) peer-reviewed paper(s) but in my case, it was two books. The first is Writing Solid Code (and it's not lost on me this was published by Microsoft). It was C-centric, but there is solid advice there about API design (like design-by-contract).
The second was Thinking Forth. Get past the Forthisms, and it's about thinking about the problem to be solved. Today the advice to think might be dismissed as taking too long ("TIME TO MARKET!!!!!!") but I've found it works well for me.
And here's a bonus third book, The Mythical Man-Month. It's less about coding per se and more about software project management and why there is no Silver Bullet that will make programing trivial. Even with LLMs, you have the communication overhead over design to contend with.
Oddly enough I was revisiting the Mythical Man Month just last night. It's impressive how much of that advice from 1975 is directly applicable to building software with coding agents today. The section on "conceptual integrity" in particular.
Amusingly there's only one idea from it that I think coding agents subvert: Brooks's law itself, which says that adding more developers to a project makes it later.
The reason that law holds so firm is the communication overhead that each new developer adds. I suspect that coding agents dramatically reduce this effect, because an agent is such an effective tool for answering questions about how the system works already.
I've seen anecdotal evidence (yes, it's always anecdotal these days) that adding a solid engineer with deep coding agent experience to a project really can help that project deliver value faster.
There are some things that can be vague and worth evaluating. Then there's "I got the result I needed in less time than I'd take to type it at full speed". Then there's a whole class of problems where the review is just unnecessary, like "correctly transform known inputs to known outputs" (parsing, compression, various encodings and transformations) - everything interesting about it can be evaluated externally without ever seeing the code.
So at some point the burden of proof really flips. "You claim I could get this result without LLMs faster than I can type it (if LLMs are not an improvement)... how?"
I am on the fence with LLMs strictly for code generation (using them a lot in general), but the arguments presented here are pretty bad: for me, it does not matter who the author of a PR is - the same quality standards apply; human or machine.
What matters more is the ownership - if someone prepares mostly LLM-generated PR, they must take the ownership and responsibility for it; they should do their best to, as previously, slice PRs into a context & size for other humans to be as digestible as possible.
Nothing changed there - code is still the ultimate specification of software that we, software developers, must own and understand.
"Show me the peer reviewed study". I don't need to. I've seen enough already.
The strawman here is asking for something that would convince them - because your own personal experience isn't enough to convince others, and shouldn't be, of course. And your response is that you're already convinced so you don't need any more evidence, which is a complete non-answer.
I hate to say this, because it sounds insulting and I don't mean that you yourself are in a cult - but this is actually how members of cults communicate. They take challenges to their beliefs personally and defend them but have no idea how to convince others.
Edit: never mind, I missed that the answer I looked for is already there. Section “The Limits Of Reviews” says “it seems that a maximum of 400 LOC/H is a good maximum speed acceptable for efficience”. That’s measured in LOC/hour, not LOC/review as I initially thought.
You summarize these conclusions from research on code review:
A review that last more than 1h is too long.
A review that has to be effective cannot be more than 400LOC at a time, in that time.
And when calculating how many LOC a developer can review per day, you translated those conclusions into this input:
For every 400 LOC written by a LLM Coding Assistants (at best, less for code that are hard to review), we need one professional senior developer to spend 1h of his time reviewing the code.
But that doesn’t necessarily follow from those conclusions. If a senior developer could review 400 LOC generated by LLMs in half an hour, and therefore review 800 LOC in one hour, that would be consistent with those conclusions. I don’t see evidence that 400 LOC must take at least 1 hour to review.
Was there such evidence in the code review research, and you simply forgot to mention it?
Still relevant after the correction above:
I’m also curious exactly which papers on code review led to those 1h and 400 LOC numbers. If you didn’t save the names of those papers and are only summarizing from memory, then you don’t have to spend lots of time searching for them again – I ask just in case you already have that information.
1h for 400 LOC smelled like BS to me so I calculated what mine would be for work.
In the last 6 months I've reviewed 173 PRs totaling 139k lines changed. For a normal 40 hr work week we can see the rate of review by assuming what % of my week I spent reviewing. If I spent 25% of my time reviewing, that's ~540 lines/hr, 15% -> 900 lines/hr, 5% -> 2700 lines/hr.
I'd guess my real throughput is 500-1k lines/hr, somewhat close to the (not) cited stat.
I don't think the initial claim in this post has aged well: AI agents did make a lot of mistakes a year ago, but it's not clear at all that this is still true. Like a lot of people, I saw an inflection point with "frontier" models around November of last year.
It's true that LLMs change the volume of code and thus put new pressures on our review capacities, though. But our responsibility as engineers is to push back and ensure that review loads match our actual capacities.
sunshowers | 4 hours ago
Worth noting that speed doesn't have to be your only goal. You can do things like "oh there's a bugfix here, let's spin this off into a preparatory commit that can be reviewed in isolation". Or "I don't like this type structure because it permits some invalid states to be representable; let's go fix that". Or "okay, I think the testing story doesn't quite give me enough confidence, let's spend some time prototyping more advanced techniques like PBT/fuzzing/formal methods". These are all things I'd have clubbed into the same commit or left as a TODO tech debt comment in the past. None of them are related to speed, except in the most general sense of correct software saving you time down the road. The marginal cost of doing things right is now (stunningly) lower.
LLMs are very open-ended as tools, and the value you get out of them mirrors the values you bring to them.
(edit: there are also risks to building too many half-finished prototypes that never see their way to production. So it's not all good. But I think overall this is a huge boon for rigor-focused engineering.)
vaguelytagged | 2 hours ago
I often find myself emphatically agreeing when I read ai takes like this. However another voice in my head mentions that this isn’t what’s happening in most companies. While it absolutely can increase rigor, more often then not I see it being used to race to the lowest quality MVP possible. Perhaps that reflects more on the company culture rather than LLMs but it’s still disappointing to see. I hope the industry will get its head on straight and produce higher quality software.
simonw | 2 hours ago
Using coding agents to build higher quality software - as opposed to cutting corners and churning out faster slop - is definitely a decision companies need to make.
vaguelytagged | an hour ago
Unfortunate to see how many companies are making the wrong decision imo.
sunshowers | an hour ago
Yes, this is very distressing to me. Feels like the inexorable logic of capitalism.
dpc_pw | 2 hours ago
I can spin a clanker with a single prompt in my historical projects, and it will start finding real bugs without even trying hard. I know because it keeps happening.
Guess what? Humans are sloppy too. I am sloppy. Mistakes happen. LLMs just bring it upfront because they work so fast and are quite a bit dumber in many respects. Mistakes compound, so if you just naively let clanker make code changes, you're going to have a bad time real fast.
But it's lazy to think that just because naive use of the tool is flaky, the tool is not useful.
My current go to slop-flow is actually getting 5 specialized reviews, including architecture, maintainability, reliability/security, etc. automatically after every change being made, and I have a whole design documentation system to keep things organized and agent's decision making improves significantly Is it perfect? No. Is it better than naive approach? Sure it is. Can it be done better? I hope. And that's also the fun part. A new weird tool in my toolbox to play with.
BinaryIgor | 4 hours ago
I am also skeptical about whether it makes the whole process - development, review and finally deploy faster; I find that sometimes I might generate something faster with a prompt, but then it takes more time to validate and understand to the extent that I find myself wondering - wouldn't it have been faster to just write by hand in the first place? And arguing that it depends on the case... judging also takes time and energy, that I would rather spend on something else.
But, as long as the person submitting the PR takes ownership I am agnostic about their LLM use - it simply does not matter as far as quality, correctness, consistency and all other metrics we find important are concerned. If it is better/faster for you to implement with the LLM, do it, if not, do it on your own - the ownership stays the same, it is on the human author.
As far as I personally am concerned, I love AI for learning and extending & deepening my understanding - but I do write most of the code manually, since I still find it more productive - all things considered, not only the typing part.
pmarreck | 4 hours ago
Well, that’s clearly a problem, then.
I think pretty much anyone who actually is optimistic about LLM’s and coding would agree that the last six months and the last three months specifically have significantly improved the utility of these. Do you actually use the latest for-pay cloud models? Do you have controls in place that prevent entire categories of failure, such as the principle I developed out of having worked in the auditor space, MFIC? https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8c8d7e Do you have tooling that helps keep already-developed code and project structure in context such as https://github.com/pmarreck/dirtree and https://github.com/pmarreck/codescan ? (Note that these are also useful to forgetful human developers or those who are or have become unfamiliar with a codebase!)
Anyway, either you make it work and enjoy the benefits, or you rebel and continue to write bespoke hand-coded things (while ALSO frequently writing bugs and security holes, btw) while others (perhaps writing occasionally flakier code, but far faster) pass you by.
Lastly, speaking as someone who spent 2 man-years working on a million-LOC Ruby on Rails codebase for Desk.com that is now retired, corpo code is ephemeral, which lines up nicely with LLM-produced code.
In your case, working on Erlang internals, it makes admittedly less sense to trust it at all, but I don’t see the argument against letting it write at least things function-by-function that you then review- you’d obviously know what’s subpar, but I bet you’d be pleasantly surprised SOME of the time.
You can work with the knitting needle or the loom. Perhaps “both” is ideal, depending on the context.
Upvoted because I know you care about code. 😉
[OP] Diana | 3 hours ago
seems my reply never made it.
i don't work on internals. I work on corpo code too. I am the one that clean up behind LLM users.
note that all you just said doesn't invalidate a single line I wrote. Quite the contrary.
simonw | 4 hours ago
It's hard to take pieces like this seriously when your own personal experience - and that of so many other credible, experienced developers who you spend time with - has shown how they help someone write code better and faster, time and time again.
"Show me the peer reviewed study". I don't need to. I've seen enough already.
"But the METR study says developers over-estimate the impact of AI tools on their own productivity". I'm happy for you that could find one study that reaffirms your priors.
hjvt | 3 hours ago
That's a bafflingly low standard of proof for someone who calls themselves a researcher.
jamii | 3 hours ago
That is the standard of proof we have for pretty much all other programming practices. We didn't adopt git or rust because there was a peer reviewed study proving that it would increase productivity.
Plus, if I could find a peer reviewed study claiming that using git or rust was bad for my productivity then I would ignore it, because it would be wrong. Most studies of programming practice are pretty terrible anyway and obviously won't generalize from 10 undergrads with an hour of experience in a tool to a large company of experienced professionals with years of training.
While we're at it, here is my not-peer-reviewed study of the peer reviewed literature - https://www.scattered-thoughts.net/writing/qnd-review-of-ppig-1989-2015/
simonw | 3 hours ago
Feel free to take that into account when evaluating my work from now going forward.
I do not think peer-reviewed academic papers are the gold standard in helping us understand how software engineering works. The incentives in that system produce weird, hard to follow results - at a glacial pace.
There's a big difference between academic papers that help understand physics or biology - where you really do need rigorous research experimental processes and benefit from peer-review - and fields like software engineering where you can do the work yourself.
How many practicing software engineers do you know who read papers about software engineering more than once or twice a year, if that?
nicoco | 2 hours ago
That argument would have more weight if software was well engineered in general; do you considered it is? What was considered a supercomputer when I was young is barely enough to render most websites, even when actual content is effectively just text. I think we can do better.
simonw | 2 hours ago
I think software today compared to software ten or twenty years ago is enormously better, and lets me get way more done.
Do you think software engineers don't read papers because they are lazy and bad at their jobs, or do you think it's because they've tried spending time reading papers and found them not to provide good enough ROI on that time?
My hunch is the latter.
addison | an hour ago
I exist in the glacial space you describe (though, blessfully, not in human aspects research). The interpretation that I and many colleagues hold is that software has, in general, become more fault tolerant (in the sense of avoiding catastrophic failure), but simultaneously more defective. The incentive structure in software development is currently far more favourable to develop quickly and fix later strategies, largely due to a lack of accountability mechanisms for the companies selling software or software services.
spc476 | an hour ago
My hunch is that it only applies to a very narrow aspect of programming (which is usually not mentioned). As an example, I never found a good definition of "unit test" that can apply across a wide range of languages and when I ask for a definition, I'm scolded for asking. As such, I've always found it hard to apply to the stuff I write in assembly or C. Unit testing seems to be much better for a language like Java or Javascript, which I don't use.
simonw | an hour ago
I have to avoid the term "unit test" in the same way that I avoid "TDD"!
Unit tests carry all sorts of baggage around only testing a single "unit" (whatever that means), which inevitably leads to a growing pile of mocks.
I prefer to avoid mocks as much as I can - I like tests that are as close to answering the question "does the feature work?" as possible.
So I'm stuck with "automated tests", because I don't want to get into the weeds concerning if something is an integration test or a unit test or a smoke test or whatever other terms people want to use.
viraptor | 36 minutes ago
Those approaches do have specific subgenres. Your idea is close to the classicist/Detroid style testing. As opposed to the mockist/London style. It's a shame those labels are not commonly used. Same with Beck's version of "unit of functionality" vs corporate Java "public method is a unit".
jyounker | 59 minutes ago
The definition from this 1969 paper is pretty good:
nicoco | 47 minutes ago
Fair enough, academic publishing has a lot of issues, and to be honest I know next to nothing about the academic field of software engineering per se (I'm an amateur!). However I don't think personal experience is worth more than (imperfect) attempts at collecting more data than mere testimonies and other forms of subjective experience. In the field I know vaguely more about (human health), practitioners (clinicians) don't have to read papers to benefit from whatever researchers are finding out, it trickles down to them in various ways. It isn't perfect, but it's basically all we have besides superstition. IMHO we would have better healthcare overall if personal experience was drastically more critically scrutinized by doctors and patients (us!). Which is why that "I've seen enough already" really made me choke a little. Maybe drawing a parallel between medicine and software engineering is ridiculous though. :)
jado | 3 hours ago
A good researcher would recognize existing studies seem to be leaving a lot out and critique them accordingly. Blindly accepting studies is not what a researcher does.
nicoco | 2 hours ago
"But homeopathy works on me and my dog, I don't care about your stinky data!"
[OP] Diana | 3 hours ago
You mean, the same that doctors use to explain why they don't wash hands?
Or the one they used to defend bloodletting?
Science literally was built on the bodies of the people we killed by trusting "own personal experience".
Note that I demand at the end to be proven wrong. Show us, in away that is robust and methodologically sound, where it differs from what we have found with robust empirical research.
I am ready to change my mind. But, I am sorry, "own personal experience" will not cut it. Nor a dozen anecdotes that explain nothing but just tell us we are wrong.
It is how we killed thousands of people in the past. Science and engineering allowed us to make the life of billions of people better. Sorry that I will keep defend them, even if it hurts your own personal experience.
This is not about the METR stuff. It is about a pattern over centuries of this happening. We developed science and engineering to counter that and it works. Sorry, I will not throw it away because you think you are better. I am too humble for that, and I need to sleep at night.
simonw | 3 hours ago
What was the last peer-reviewed paper you read about the craft of building software that positively influenced you and helped you become a better software developer?
[OP] Diana | 3 hours ago
That is a good question. Hmm... Probably research on TDD, I will bring it up once I am back from travel. Mostly research on TDD at this point shows no real impact, hence why I do not push it hard.
simonw | 3 hours ago
That's a fun example because it goes directly against my own personal experience, too!
I'm a firm believer in automated testing because I've seen directly how much more productive it makes me - because I can ship a new feature for a project I haven't touched in years without having to manually check that I haven't broken any of the existing, documented behavior.
I'm quite confident I could find holes in any paper that attempted to prove otherwise.
(Test-first development, on the other hand, has failed to convince me. I've tried writing the tests first and found it to slow me down without improving the quality of my code... until the last six months where I've found forcing the coding agents to write their tests first appears to produce less hallucinations and code that's better factored and more likely to pass review - anecdotally though.)
[OP] Diana | 3 hours ago
Note that I did not say "automated testing". I said "TDD". There is a huge difference and if you don't know it, that is, actually, quite scary.
simonw | 3 hours ago
That's why qualified I qualified "test first" development in my parenthetical at the end.
I know from experience that "TDD" is an ambiguous term - some people use it to describe red/green test-first development, but it's often used as a label for any kind of automated testing.
So in my reply I deliberately used the less ambiguous terms "automated testing" and "test-first" - precisely because I do understand the difference.
jo3_l | 2 hours ago
I'm a fan of your work and believe you make this claim honestly but... this seems absurd. Who is using "TDD" to describe all automated testing in general? That seems like an abuse and misunderstanding of terminology rather than a legitimate difference in opinion.
simonw | an hour ago
Kent Beck:
Sound like a classic example of semantic diffusion. I've fallen victim to that one myself.
What common term or acronym would you use to describe automated tests that might or might not be written test-first?
This is a major source of frustration for me. I dislike that the acronym TDD, in its classic definition, embeds test-first which I think distracts from the value of automated testing generally.
In my own writing I avoid saying "TDD", so I'm stuck with "automated testing" (the word "testing" by itself overlaps with manual testing) instead.
So any time an online discussion mentions TDD I feel compelled to clarify if they mean test-first. Which is what I did here.
If I could wave a magic wand I would redefine TDD to mean development that incorporates automated testing, and enforce TFD for the less useful test-first development pattern.
jo3_l | an hour ago
Thanks for clarifying. However, I'm not sure that your quotation supports the claim that there are folks out there using "TDD" to refer to automated testing in general. In my view, TDD refers to a development workflow, which describes the points in the development lifecycle that involve writing/running automated tests. Different people can and do disagree about what the specifics of this workflow are, as your Kent Beck quotation illustrates, but invariably it refers to a kind of development workflow. On the other hand, automated testing is closer to a component of a project, which implements automated validation of the other components.
To be clear, I do appreciate that you're being explicit about what TDD means--I just remain uncertain that anyone defines it as encompassing all automated testing.
I don't have one. I just say "automated testing", or even just "testing"--qualifying "manual testing" explicitly when needed. To me, automated testing is something that most serious projects practice and view as useful (with some notable exceptions, of course--I recall the SumatraPDF developer doesn't see much utility in them), so I rarely need to argue for its inclusion. TDD, on the other hand, is far more controversial, being a particular development workflow.
Again, I can always appreciate the goal to be explicit, but I'm not so sure that TDD is a poor choice of term. Certainly TFD is a clearer one, but I wouldn't be in favor of repurposing TDD to describe the use of automated testing. I write a lot of automated tests, but I would not characterize my process as test-driven; they are supplementary.
simonw | an hour ago
I wish I had a snappy acronym I could use in place of "automated testing".
spc476 | 3 hours ago
I'm not @Diana, and it's not a (or several) peer-reviewed paper(s) but in my case, it was two books. The first is Writing Solid Code (and it's not lost on me this was published by Microsoft). It was C-centric, but there is solid advice there about API design (like design-by-contract).
The second was Thinking Forth. Get past the Forthisms, and it's about thinking about the problem to be solved. Today the advice to think might be dismissed as taking too long ("TIME TO MARKET!!!!!!") but I've found it works well for me.
And here's a bonus third book, The Mythical Man-Month. It's less about coding per se and more about software project management and why there is no Silver Bullet that will make programing trivial. Even with LLMs, you have the communication overhead over design to contend with.
simonw | 3 hours ago
Oddly enough I was revisiting the Mythical Man Month just last night. It's impressive how much of that advice from 1975 is directly applicable to building software with coding agents today. The section on "conceptual integrity" in particular.
Amusingly there's only one idea from it that I think coding agents subvert: Brooks's law itself, which says that adding more developers to a project makes it later.
The reason that law holds so firm is the communication overhead that each new developer adds. I suspect that coding agents dramatically reduce this effect, because an agent is such an effective tool for answering questions about how the system works already.
I've seen anecdotal evidence (yes, it's always anecdotal these days) that adding a solid engineer with deep coding agent experience to a project really can help that project deliver value faster.
viraptor | an hour ago
There are some things that can be vague and worth evaluating. Then there's "I got the result I needed in less time than I'd take to type it at full speed". Then there's a whole class of problems where the review is just unnecessary, like "correctly transform known inputs to known outputs" (parsing, compression, various encodings and transformations) - everything interesting about it can be evaluated externally without ever seeing the code.
So at some point the burden of proof really flips. "You claim I could get this result without LLMs faster than I can type it (if LLMs are not an improvement)... how?"
BinaryIgor | 4 hours ago
I am on the fence with LLMs strictly for code generation (using them a lot in general), but the arguments presented here are pretty bad: for me, it does not matter who the author of a PR is - the same quality standards apply; human or machine.
What matters more is the ownership - if someone prepares mostly LLM-generated PR, they must take the ownership and responsibility for it; they should do their best to, as previously, slice PRs into a context & size for other humans to be as digestible as possible.
Nothing changed there - code is still the ultimate specification of software that we, software developers, must own and understand.
simonw | 3 hours ago
100% agree with that. A computer can never be held accountable. That responsibility is uniquely human.
sunshowers | an hour ago
I personally endeavor to apply higher standards to machine-generated code, not merely the same.
bendmorris | 22 minutes ago
The strawman here is asking for something that would convince them - because your own personal experience isn't enough to convince others, and shouldn't be, of course. And your response is that you're already convinced so you don't need any more evidence, which is a complete non-answer.
I hate to say this, because it sounds insulting and I don't mean that you yourself are in a cult - but this is actually how members of cults communicate. They take challenges to their beliefs personally and defend them but have no idea how to convince others.
roryokane | 4 hours ago
Edit: never mind, I missed that the answer I looked for is already there. Section “The Limits Of Reviews” says “it seems that a maximum of 400 LOC/H is a good maximum speed acceptable for efficience”. That’s measured in LOC/hour, not LOC/review as I initially thought.
You summarize these conclusions from research on code review:And when calculating how many LOC a developer can review per day, you translated those conclusions into this input:But that doesn’t necessarily follow from those conclusions. If a senior developer could review 400 LOC generated by LLMs in half an hour, and therefore review 800 LOC in one hour, that would be consistent with those conclusions. I don’t see evidence that 400 LOC must take at least 1 hour to review.Was there such evidence in the code review research, and you simply forgot to mention it?Still relevant after the correction above:
I’m also curious exactly which papers on code review led to those 1h and 400 LOC numbers. If you didn’t save the names of those papers and are only summarizing from memory, then you don’t have to spend lots of time searching for them again – I ask just in case you already have that information.
[OP] Diana | 3 hours ago
I have it somewhere, i can go check.
Note that most of them are more at the 100 to 200 loc/h limit, but they tend to be older, when programming languages were a bit more prone to bugs.
Ping me in a few days, rn i am travelling.
jado | 3 hours ago
1h for 400 LOC smelled like BS to me so I calculated what mine would be for work.
In the last 6 months I've reviewed 173 PRs totaling 139k lines changed. For a normal 40 hr work week we can see the rate of review by assuming what % of my week I spent reviewing. If I spent 25% of my time reviewing, that's ~540 lines/hr, 15% -> 900 lines/hr, 5% -> 2700 lines/hr.
I'd guess my real throughput is 500-1k lines/hr, somewhat close to the (not) cited stat.
yossarian | a minute ago
I don't think the initial claim in this post has aged well: AI agents did make a lot of mistakes a year ago, but it's not clear at all that this is still true. Like a lot of people, I saw an inflection point with "frontier" models around November of last year.
It's true that LLMs change the volume of code and thus put new pressures on our review capacities, though. But our responsibility as engineers is to push back and ensure that review loads match our actual capacities.