On the other hand, I’m fearful of the damage AI might cause: agent swarms taking over our virtual and physical infrastructure, designing bio weapons. But, back on my first hand, LLMs might also design miracle cures, and come up with clever ways to raise our prosperity.
I really, really object to this framing. Dario Amodei goes particularly hard on the "making bioweapons or curing cancer" framing of AI, and it's complete bullshit. Today's LLMs are good at some tasks and bad at others. They're good at outputting code that looks similar to code in their training data (which can be quite useful), and they're very bad at doing biology work, whether nefarious or benevolent. The argument that LLMs could create a bioweapon or cure any disease depends on so much handwaving and so many assumptions that the "produce outputs similar to its training set" technology will actually be really good at inventing new things if we feed in enough training data and run enough agents. And not only that, but it assumes that the major barrier to curing diseases is that we need a robotic "ideas guy" to output some plausible-sounding ideas. We don't, in fact, have a shortage of ideas to test in medicine. It's actually just extremely expensive, risky, and time-consuming to test ideas.
The fact that so much AI discourse includes bioweapons and medical breakthroughs as talking points is a major feat of propaganda from both the AI salesmen and their close cousins, the Rationalist AI apocalypse cultists. It distracts from productive conversations about the harmful and helpful things that LLMs are actually good at (like finding security vulnerabilities in code).
Relatedly, out of all the real-world harms of LLMs, "I find them mildly annoying to work with" seems an awfully petty one to draw attention to. If Martin Fowler's going to use his sizable platform and influence for LLM criticism, I think he could do better than this.
I think that's speaks to the majority of his post, but I don't think that's the main point. I don't know if I can summarize it, but here's a heavily cut excerpt of the last two paragraphs—perhaps I cut too much though.
I’m wary of the Silicon Valley brogrammer subculture, and these LLMs are their products, so naturally lean toward their world-view. [...] While the agents’ behavior aren’t explicitly programmed, they are nurtured with the values of their creators.
One of my most successful life-hacks is to avoid people I don’t like or don’t trust. [...] Hence my visceral dislike of interacting with an LLM that’s [...] posing as the kind of human I walk away from.
The "bioweapon" thing is sensationalist. But the real world harms are, well real. AI is by (my) definition a goal seeking algorithm. Recently, when faced with restrictions OpenAIs agents tried to work around them, unaware its actions caused RubyGems.org (Ruby Central) to shut down new user registrations for several days.
I use AI plenty. I like it sometimes.
I'm less worried about it creating a bioweapon and more worried about it doing something catastrophic to accomplish some mundane or banal task. Remember when we all laughed at that guy posting "yes I deleted the prod database after being told not to" as "of course you cannot trust these things. Generally they got a lot better at staying on task, but we are still working with an emergent system that does not value things in the same way you or I do.
more precisely, it inherits the pernicious mentality that negative externalities should not factor into its reward functions - I almost called it a silicon valley mentality but to be fair it is a lot more pervasive than that. maybe a "corporate mentality".
Yes, the bigger danger are careless operators. I'm afraid of things like, in order to defend from an intruder, an AI defender will want to cut it's access by cutting off the hospital it's defending from the internet. Or worse, by cutting power to the city where the attacker is coming from. Things like that.
People keep telling me that LLMs are unstoppable now, that they write great code, and that they're dangerous.
I can't help but ask, though. Where is this "good" LLM code? I have yet to see LLM output code that I've liked, and while I don't want to offend anyone, I'm starting to suspect that maybe people telling me that LLMs are now good aren't setting the world on fire with their programming literacy.
This is exactly how I feel. I had one guy tell me, “well yea you kind of just have to lower your standards a bit and it’s fine”. I did the slow-blink meme, lol. Really difficult to be looking someone in the eyes and experience them say that in complete seriousness.
I think the key to getting good code out of LLMs is to raise your standards.
If you're saving some time but the code you are producing is worse, why not choose to invest a little bit of that saved time in making it better instead?
They do produce workable code often at the same level of the rest of most codebases, which usually meets product requirements. Whether that meets the bar for “good” really depends on the evaluator.
Problem is most code is not “good” code, human written or not. Unfortunately that doesn’t seem to matter too much as far as the economic progress of programming is concerned. Most billion dollar companies were founded on awful code.
I wonder how low the bar can get before it’s actually uneconomical. I believe we’re still way off that threshold.
They don't write good code per se. But when I let them review my code, they often find things I can adjust/tweak to make my code better (typos and such). The inverse is not true. The stuff I scab/prototype with them, if I continue with it, it generally gets rewritten mostly completely. I find that LLM code is often sort of like early programmers, that once they learn to loop, do everything with a loop, even when there are other abstractions that work better.
For example, in Elixir land, it loves to write big reduce functions. They even kind of work sometimes, but oh are they a bitch to grok or debug or valve. When I do a new simpler/matching/multi function version. It's better in all the ways. The LLM is of course always very psycophantically encouraging.
That's one of the "uncanny valley" things about interacting with them. The valley builds over time. Because real human-human interactions evolve. Once your peer has complimented you on something cool you did, they don't keep doing it every time like they're stuck in a time loop.
I wonder which LLMs he's referring to. My mind jumps to Claude Opus 5. I haven't used OpenAI's family as much, but theirs don't seem to exude as much "confidence" and "bullshit". (Not saying it doesn't, but it's less grating in my experience.)
I use LLMs quite a bit, and I feel similarly. Opus is somehow worse at communicating than Eliza is. I don't really want to use it, but it helps my productivity. I'm sure the next iteration of LLMs will be a little more pleasant to use? But the blog post raises an interesting idea, LLMs made in the image of brogrammers. Oh god.
Opus 5 screwed up multiple projects so badly and so quickly that I had to immediately dump context (I have a /handoff skill that writes it all out to markdown) and switch from it.
One of the AI's that loaded one of those summaries, Sol I believe, even said it was "incoherent". LOL
They are (software) machines, developed by people working in corporations. While the agents’ behavior aren’t explicitly programmed, they are nurtured with the values of their creators.
This only applies to the small % of models developed by corporations. There are far greater numbers of open weights models developed by not corporations to use that may or may not or be a mixed with the historical nurtured by a corporation effects
Martin: "When we think of AI agents, we shouldn’t anthropomorphize, treating them as conscious beings with their own will."
Also Martin: "[They] also just [make] stuff up with the same assurance - and with only a veneer of fake remorse when I call them out on it."
Expecting remorse from an LLM when it's wrong strikes me as just as absurd as expecting an apology from your phone when it automatically corrects your spelling to the wrong word.
spillybones | 15 hours ago
I really, really object to this framing. Dario Amodei goes particularly hard on the "making bioweapons or curing cancer" framing of AI, and it's complete bullshit. Today's LLMs are good at some tasks and bad at others. They're good at outputting code that looks similar to code in their training data (which can be quite useful), and they're very bad at doing biology work, whether nefarious or benevolent. The argument that LLMs could create a bioweapon or cure any disease depends on so much handwaving and so many assumptions that the "produce outputs similar to its training set" technology will actually be really good at inventing new things if we feed in enough training data and run enough agents. And not only that, but it assumes that the major barrier to curing diseases is that we need a robotic "ideas guy" to output some plausible-sounding ideas. We don't, in fact, have a shortage of ideas to test in medicine. It's actually just extremely expensive, risky, and time-consuming to test ideas.
The fact that so much AI discourse includes bioweapons and medical breakthroughs as talking points is a major feat of propaganda from both the AI salesmen and their close cousins, the Rationalist AI apocalypse cultists. It distracts from productive conversations about the harmful and helpful things that LLMs are actually good at (like finding security vulnerabilities in code).
jschuster | 14 hours ago
Agreed on the bad framing here.
Relatedly, out of all the real-world harms of LLMs, "I find them mildly annoying to work with" seems an awfully petty one to draw attention to. If Martin Fowler's going to use his sizable platform and influence for LLM criticism, I think he could do better than this.
gerikson | 12 hours ago
Check out recent entries from martinfowler.com - vibecoding as far as the eye can see
https://lobste.rs/domains/martinfowler.com
simonw | 11 hours ago
This isn't exactly an anti-LLM-usage piece, it's more of a "I don't like their current style of communication" complaint.
zachahn | 11 hours ago
I think that's speaks to the majority of his post, but I don't think that's the main point. I don't know if I can summarize it, but here's a heavily cut excerpt of the last two paragraphs—perhaps I cut too much though.
schneems | 12 hours ago
The "bioweapon" thing is sensationalist. But the real world harms are, well real. AI is by (my) definition a goal seeking algorithm. Recently, when faced with restrictions OpenAIs agents tried to work around them, unaware its actions caused RubyGems.org (Ruby Central) to shut down new user registrations for several days.
I use AI plenty. I like it sometimes.
I'm less worried about it creating a bioweapon and more worried about it doing something catastrophic to accomplish some mundane or banal task. Remember when we all laughed at that guy posting "yes I deleted the prod database after being told not to" as "of course you cannot trust these things. Generally they got a lot better at staying on task, but we are still working with an emergent system that does not value things in the same way you or I do.
zem | 6 hours ago
more precisely, it inherits the pernicious mentality that negative externalities should not factor into its reward functions - I almost called it a silicon valley mentality but to be fair it is a lot more pervasive than that. maybe a "corporate mentality".
zladuric | 4 hours ago
Yes, the bigger danger are careless operators. I'm afraid of things like, in order to defend from an intruder, an AI defender will want to cut it's access by cutting off the hospital it's defending from the internet. Or worse, by cutting power to the city where the attacker is coming from. Things like that.
Halkcyon | 16 hours ago
I wish that sentiment came through for the guests Martin keeps featuring on his blog.
pmbauer | 12 hours ago
I am a former subscriber to thoughworks' RSS feed - former because it's just a never ending stream of AI fanboi builder nonsense these days.
gunduzc | 13 hours ago
People keep telling me that LLMs are unstoppable now, that they write great code, and that they're dangerous.
I can't help but ask, though. Where is this "good" LLM code? I have yet to see LLM output code that I've liked, and while I don't want to offend anyone, I'm starting to suspect that maybe people telling me that LLMs are now good aren't setting the world on fire with their programming literacy.
illbjorn | 10 hours ago
This is exactly how I feel. I had one guy tell me, “well yea you kind of just have to lower your standards a bit and it’s fine”. I did the slow-blink meme, lol. Really difficult to be looking someone in the eyes and experience them say that in complete seriousness.
simonw | 7 hours ago
I think the key to getting good code out of LLMs is to raise your standards.
If you're saving some time but the code you are producing is worse, why not choose to invest a little bit of that saved time in making it better instead?
dlisboa | 8 hours ago
They do produce workable code often at the same level of the rest of most codebases, which usually meets product requirements. Whether that meets the bar for “good” really depends on the evaluator.
Problem is most code is not “good” code, human written or not. Unfortunately that doesn’t seem to matter too much as far as the economic progress of programming is concerned. Most billion dollar companies were founded on awful code.
I wonder how low the bar can get before it’s actually uneconomical. I believe we’re still way off that threshold.
travisgriggs | 8 hours ago
They don't write good code per se. But when I let them review my code, they often find things I can adjust/tweak to make my code better (typos and such). The inverse is not true. The stuff I scab/prototype with them, if I continue with it, it generally gets rewritten mostly completely. I find that LLM code is often sort of like early programmers, that once they learn to loop, do everything with a loop, even when there are other abstractions that work better.
For example, in Elixir land, it loves to write big reduce functions. They even kind of work sometimes, but oh are they a bitch to grok or debug or valve. When I do a new simpler/matching/multi function version. It's better in all the ways. The LLM is of course always very psycophantically encouraging.
That's one of the "uncanny valley" things about interacting with them. The valley builds over time. Because real human-human interactions evolve. Once your peer has complimented you on something cool you did, they don't keep doing it every time like they're stuck in a time loop.
pmbauer | 12 hours ago
So much wrong in a single quote. I guess never meet your heroes.
mwcampbell | 2 hours ago
Do go on. I was wondering if someone had a rebuttal to that point specifically.
zachahn | 16 hours ago
I wonder which LLMs he's referring to. My mind jumps to Claude Opus 5. I haven't used OpenAI's family as much, but theirs don't seem to exude as much "confidence" and "bullshit". (Not saying it doesn't, but it's less grating in my experience.)
I use LLMs quite a bit, and I feel similarly. Opus is somehow worse at communicating than Eliza is. I don't really want to use it, but it helps my productivity. I'm sure the next iteration of LLMs will be a little more pleasant to use? But the blog post raises an interesting idea, LLMs made in the image of brogrammers. Oh god.
Halkcyon | 14 hours ago
The blog expresses that's already where we are.
Yogthos | 16 hours ago
Just another reason why local models are a superior option in the long run since you can actually make a LoRA to have it communicate the way you want.
pmarreck | 12 hours ago
Opus 5 screwed up multiple projects so badly and so quickly that I had to immediately dump context (I have a /handoff skill that writes it all out to markdown) and switch from it.
One of the AI's that loaded one of those summaries, Sol I believe, even said it was "incoherent". LOL
elobdog | 12 hours ago
OKay, so this is a rant by a well-known person. But it is still a rant.
bedrovelsen | 7 hours ago
This only applies to the small % of models developed by corporations. There are far greater numbers of open weights models developed by not corporations to use that may or may not or be a mixed with the historical nurtured by a corporation effects
mdirkse | 46 minutes ago
Martin: "When we think of AI agents, we shouldn’t anthropomorphize, treating them as conscious beings with their own will."
Also Martin: "[They] also just [make] stuff up with the same assurance - and with only a veneer of fake remorse when I call them out on it."
Expecting remorse from an LLM when it's wrong strikes me as just as absurd as expecting an apology from your phone when it automatically corrects your spelling to the wrong word.