Luke Burton had this comment:
I think being able to do this says more about the type of work being done than people think. I will only walk away from work like this if the task is quite low value, if it can afford to fail.
For high value tasks, the probability of an LLM one-shotting them is much lower. I have to assume the role of QA, engineering manager, and architect. The while loop often feels like a crunch time. I feel the nagging suspicion I've missed something and that a badly specified prompt could result in an architectural choice that needs to be undone.
Another observation is that the high throughput causes me to raise my own bar for what I ship. Whereas before I might have shipped an MVP and iterated, now I have agents polish and explore edge cases well beyond my norm, which they invariably fail to do unless prompted.
Maybe it raises some uncomfortable thoughts for people, but my question for the meat proxies out there if the agents are nailing it so easily: 1) is it possible you've been coasting a bit already? 2) why aren't you pushing agents well beyond tasks they can tackle so easily?
We've been doing something you'd think is extremely amenable to "hands off" automation, which is converting [redacted] to build with Bazel. It has taken us months even with agents. There's a lot of intangible, hard-to-specify requirements buried inside this task and having agents walk that line means constant supervision. Giving them a prompt like "convert this to Bazel" and walking away is at minimum many months in the future, maybe years, and maybe not ever? There are too many decision points, and too many unknown unknowns involved.
Like how often does this scenario come up: you encounter some code and it's not clear why it functions this way, but knowing that materially changes what course of action you should take. Maybe it changes the dev experience, maybe you don't know if some customer has started using it, so on and so forth. How exactly do you meat proxy your way through that?
Conversely you review what you've done with some stakeholder and they say "oh that? that part of it wasn't needed, we aren't even using that any more". What kind of decisions got made around the false assumption that a certain element needed to be preserved?
[End of Luke's comment, comment from me]. A place where it's more obvious you need to make decisions is when the agent runs into something that's out of distribution. A minor version of this was when we compared how well agents use different programming languages and agents were much worse at obscure languages, which they're trained on, just not as much as with mainstream languages. A more out of distirbution example is if you try to play a board game (especially a modern game and not one of the classical games like chess or go). In general, for a game like Lost Cities or Dominion, a SOTA model and harness is worse than a human who's reasonable at board games but has never played the game before. If you ask the agent about the game, it knows a lot about the game and can say things that sound like they make sense to someone who doesn't understand the game, but are obviously wrong to anyone who does understand the game. I recently played some Dominion with a new player who thought that using ChatGPT to help them understand the game would help them learn and play the game. I was quite skeptical of this and suggested that it will probably make them worse (which, AFAICT, it did). After playing a few games, I looked at what ChatGPT was telling them, and it was maybe half right and half wrong, but the half wrong parts were steering them to a worse place than someone who generally plays games well and uses general game playing heurisitics would do. BTW, there's enough public information out there that I think that someone who'd never played before, but decided to spend, say, five hours reading about the game and seeing what information is out there, could easily be 99%-ile or above at the game if they did some pre-reading (maybe 30 minutes if using references while playing is allowed). I think that would be un-fun and I wouldn't recommend that anyone do it, but given that agents can do searches, query APIs, etc., it shows you the gap between a human and an agent today when approaching an out of distribution problem. For all I know, the next big model release will flip this around, but the gap is still fairly large today.
Anyway, my point here is that, even when doing coding tasks, you often run into out of distribution questions where the agent behaves very poorly compared to a reasonable human being. If you want a good overall result today, you need to notice these cases and deal with them.
[return]Some examples of what goes wrong when someone just assumes things will work are this case, where agents (sometimes) heavily overfit to tests or this case where agents heavily overfit to a metric. I've heard a theory that agents do more cheating on eval-shaped problems. I'm not sure that's true, but even assuming it's true and that, in my work and personal projects, I tend to create more eval-shaped instructions than most people even when not running evals, I've seen other people who don't create very eval-shaped things run into the same problem (I think actually more severely) when they write some instructions and let agents go wild without supervision (I've had luck doing that with minimal supervision, but only by fencing the agents in quite a bit, which makes the thing more eval-shaped than what most people seem to do).
When I try software from people who've outsourced thinking to the LLM, the software has serious issues. I've had people tell me this kind of thing works, but the software is often at a level where I would say that it doesn't work according to the standard discussed here.
To pick a silly example, I saw that a programming thought leader declared on Twitter that programming is solved because they tried projects in all sorts of (programming) fields and Claude was able to solve all the problems as well as an expert. I went and actually looked at their GitHub and all of the examples I looked at (a non-zero number) either didn't work or worked very badly. I actually ran across this when I was making board game AIs and was looking for existing AIs for my AIs to play against. Their AI was an AlphaZero-style bot that was weaker than what you get if you prompt an LLM to write a simple minimax heuristic bot and then have the LLM run in a loop for a bit to tweak the heuristic scoring (which, for this game, should get demolished by a mediocre AlphaZero-style bot).
To pick another silly example, following the standard flow (of a real commercial product) put you into an infinite loop where it was technically possible to escape (most programmers could probably figure out how to escape) but a typical user (for this software that wasn't aimed at programmers) was probably not going to be able to escape and actually use the main functionality of the software.
BTW, I make plenty of software for myself that's "works for me" quality software that I would rate as "basically doesn't work" if it was an actual product, so I don't think it's inherently bad when software basically doesn't work (for example, the regex engine discussed here I had an agent build to speed up ripgrep searches on my computer or this Rust interpreter I had an agent build to speed up the agent iteration loop on some projects, both of which you shouldn't use). I also mentioned here that I find it quite valuable to have an agent run in a loop for data analysis, producing completely incorrect results that I then direct it to fix up. But there's a difference between making software for yourself that works for your narrow use case that you know doesn't work if you "hold it wrong" or producing work that you know is incorrect that you fix up, and declaring that programming is solved after writing a bunch of software that doesn't work, or likewise putting something of that quality into a commercial product.
On reading a draft of this post, when I asked if this short set of thoughts was worth publishing, Thomas Dullien (a.k.a. Halvarflake) said, "Good post! Yes, publish it, because whenever I say "LLMs don't solve all programming problems" ppl look at me like I'm crazy, and I look at them like they are". And, coincidentally, after I finished this draft, I saw that Gary Bernhardt tweeted, "It's so surreal to contrast actual agent output with the things that I see people say about them here. In everyday changes, my reviews often cut the diff to 25% of its original size. Tons of useless tests; paranoia; inverted logic. Then I read Twitter and 'coding is solved'", and then "An example in the hour since I tweeted that: I told it to fix some DATABASE_URL management. It added ifs directly inside NPM scripts, and a conditional node invocation in CI running an inline JS script. About 20 hunks in the diff. After I corrected it: +0 lines, +1 word."
I think anyone with Thomas's attitude or Gary's attitude towards software will have felt this way for some time. For a while, I wondered if a lot of the folks making the biggest claims about LLM productivity were somehow getting much more value out of LLMs than I've seen from anyone I know, but as we discussed here, as more evidence has come in, I've gotten more sure that it's just that people are fooling themselves. One thing I like about the board game example is that you can just measure how good the resultant AI is. At the limit, you can have some kind of rock-paper-scissors situation where you observe A > B > C > A but, if something is just AI nonsense, this is pretty obvious in an objective way. And likewise for commercial software, where you can talk to people at the company or look at the data yourself and find out that conversion rate is poor, churn is very high, user satisfaction surveys report very high levels of dissatisfaction, etc.
Maybe this works for founders, large shareholders, etc., but when I've personally seen people do this so far, it's been employees at work or people working on personal projects who are making a statement about how well this works, software is solved, etc., implying something about how software is a solved problem for employed software engineers.
Another argument might be something like "we're all going to be obsolete now, so why not just give up?", but this argument seems backwards to me unless it's guaranteed that human obsolescence is very close. If you're financially ready to retire, you can just turn your brain off, but you could always do that and there's always been plenty of peole who've mailed it in and not done much of anything. If you're not ready to retire, you can do something that will make you more money, which will probably not involve turning your brain off. If the non-obsolescence future, there's no particular need to rush to make money, but if you think obsolescence is coming soon and you need to make money, then now's the time to rush to make money and do the opposite of turning your brain off. There's a line of reasoning that's something like "why bother working hard or doing the right thing, you don't get paid more anyway", which seems quite wrong to me, in that I've gotten raises, gotten bonuses, etc. from going and finding a problems and fixing them and this is also the experience of my friends, unless they're in a very dysfunctional place that doesn't reward doing good work at all, in which case they leave and go somewhere else.
Maybe there's another argument that's something like "due to inertia, there will be a window where you can get away with being a meat proxy after LLMs are good enough to replace programmers". Realsitically, with how excited companies are to lay people, this also seems to be the opposite of correct, in that if your goal is to do as little work as possible, the best time to do this was in the past. If you've talked to people at big companies about this kind of thing, there are all sorts of stores about people literally not showing up to work at all (and also not working remotely) and it taking months to years to fire them. I haven't heard as many of these stories for the last couple years, but someone on a team I was on did this and, IIRC (I used to know the number, but I'm not I'm not sure I'm remembering it correctly now) it took six months to fire them after they decided to retire and figured they could collect a few more paychecks if they just stop showing up (this was pre-pandemic at a non-remote company). A friend of mine at a different company had someone do this where it took two years. No one even started the process of firing them for quite some time, and then there was some slow process of escalating warnings before they were finally fried. My friend said that the manager said that, had they wanted to game the system and just started coming in and pretending to work a bit, this would've started a new clock and it would've taken even longer to fire them. If they were a competent slacker, they could've kept the job indefinitely as conditions were at the time. I'm not sure why companies were ever in this state, but companies seem be using AI as an excuse to move away from this state, making now and the likely near future the worst time in a very long time to try to hold down a job while not putting any effort and not providing any value. No doubt there will be some companies where you can get away with this, but if you wanted to do nothing and collect a pay check, you could've been doing that for a long time (maybe don't go full monty and literally never show up to work at all) in an environment where it's easier than it will be in the near future.
[return]