Back in the days when natural language translation was based on deterministic dictionaries and rules it was expected that with a long enough set of rules the machine would just translate ideally.
It happens that no matter how much they tried, a true language to language mapping is too subtle, ever changing and culture rooted to be created by hand. So, they gave up and built LLMs.
So fundamentally, compiler is so predictable because we learn to speak its language. While LLMs translate from one to another which is inherently lossy and, as we've just recited, solved with tech invented after the probabilistic turn.
And this is good. One more different tool for different tasks, right?
It's not only nondeterminism, it's also opacity. If something weird happens when you compile your code, you can dig into the compiler and figure out why it's doing that, and maybe fix it. An LLM is a complete black box, even for the people who created them.
Fundamentally: programming languages have semantics; prompts don't.
While I agree with the basic thesis of this post and have said that same thing to friends, I think this line is wrong, and makes a claim that's much too strong
If I say "never use the getFoo() api", this statement has a meaning that can be studied (https://plato.stanford.edu/entries/logic-deontic/). The challenge is that even with a deterministic LLM, will I know:
Will this particular prompt be obeyed?
Will the machine maliciously comply (e.g. by performing a series of operations that are equivalent to the getFoo method, but which do not literally call it)
Will the machine realize that I spoke too broadly, and that it's in the rare situation where getFoo is in fact required?
This is not to deny that natural language lends itself to vagueness, and code is sometimes more suitable for the things that we want to do as developers (But not always! There is a reason that we communicate with other developers in natural language).
Yeah I could have definitely improved that statement. Perhaps "formal semantics" is better. But I didn't want to say that because there are plenty of languages without a formalized semantics where the expectations about what's expected are still clear.
Is setting the temperature to zero enough to make an LLM deterministic? There's a lot of parallel numericalcomputation that is likely scheduled differently in each execution. Is it possible that operations accumulate different rounding errors, and that those errors compound to produce different results?
Yeah I mention in the caveat section that the approaches I show don’t actually guarantee determinism. But I think that’s fine as creating a “deterministic LLM” is just a tool for argument.
Generating programs from specifications is called Program Synthesis and it goes all the way back to the days of Alonzo Church. Program Synthesis is usually framed in terms of logic, but Genetic Programming is essentially the same problem framed in terms of optimization. A Field Guide to Genetic Programming is a good introduction to that topic. One can think of coding agents like Claude Code as GP systems that use an LLM to do guided search, and whose objective functions are likelihood of the program given a prompt.
Using GP it's been possible for decades to generate the code that makes a test suite pass.
Nondeterminism is still the problem, not that prompts lack semantics.
"In the future programmers will only write specs and the LLMs will regenerate the code every time like a compiler."
"You don't review the compiler output, why review the LLM output?"
I've made both those arguments against LLMs, and at the time, I was assured by the pro-LLM side that those were silly! No one would ever do that! Yet, it's common for both to be true. But to the second argument, I counter with "You mean you need discipline to review the LLM output? Just like you need discipline to write safe C code?"
It seems to me that your blog post is still expressing a nondeterminism argument, just focusing on the nondeterminism of program behavior as opposed to compiler output.
Sorry, I’m not following. Can you elaborate? I didn’t say anything about making program behavior nondeterministic, only about making compiler output nondeterministic in a way that doesn’t change program behavior (according to the language semantics).
You believe the "real problem" is that LLMs make no promises about a prompt which lacks semantics. Isn't that another way of saying that the prompt does not determine the behavior of the program in a predictable way?
Absolutely, but “determine the behavior of the program in a predictable way” is not the same thing as being deterministic. As I showed in the post, it’s easy to make a deterministic LLM. Prompts to that deterministic LLM still definitely do not determine the behavior of the program in a predictable way.
I guess an extreme determinism-vs-semantics distinguisher would be «SHA512 of each 4KiB block of source, as x86-64 machine code» — absolutely deterministic, absolutely standartised, absolutely semantic-less… LLMs are less deterministic and closer to having semantics than that, and easier to get something somewhat useful, so it's semantics not determinism.
ETA: when I say standartised, I mean compiler behaviour, target is somewhat scary but mostly documented.
Are you seriously suggesting that natural languages don't have meaningful semantics? I'd say your conclusion suffers from the same flaw you drew from non-determinism.
Furthermore, it also applies to compilers. Machine languages, as well as byte-code has semantics, so why not review it?
Overall this might serve as a useful heuristic (maybe), but it's not more or less accurate than the model you are trying to replace as far as I can tell.
I would say the main reason people are compelled to review LLM generated code is (lack of) trust, and pure habit.
“Semantics”, unsurprisingly, is a word with multiple meanings. The author is clearly using it to refer to formal semantics. For example, the Java specification which they repeatedly mention in their post. Acting as though the author doesn’t know that words mean things is nitpicky and insulting.
The semantics of natural languages is physical reality. But physical reality does not fit into GPU VRAM. It follows quite neatly that, while we can say that the latent space of a probabilistic model tightly approximates the semantics of natural languages, it can't possibly be a faithful encoding!
I'm sorry but I don't follow. Physical reality does not fit my brain either so I guess my brain also approximates the semantics of natural languages. So I guess natural languages don't exist except in the mind of an abstract all-knowing god?
Of course natural languages exist as syntax. However, there is no objective semantics for any natural language, since reality is not objective. (Additionally, all-knowing gods are impossible as a consequence of Tarski's Undefinability.)
peter-leonov | a day ago
Nice mental model.
Back in the days when natural language translation was based on deterministic dictionaries and rules it was expected that with a long enough set of rules the machine would just translate ideally.
It happens that no matter how much they tried, a true language to language mapping is too subtle, ever changing and culture rooted to be created by hand. So, they gave up and built LLMs.
So fundamentally, compiler is so predictable because we learn to speak its language. While LLMs translate from one to another which is inherently lossy and, as we've just recited, solved with tech invented after the probabilistic turn.
And this is good. One more different tool for different tasks, right?
sjamaan | 23 hours ago
It's not only nondeterminism, it's also opacity. If something weird happens when you compile your code, you can dig into the compiler and figure out why it's doing that, and maybe fix it. An LLM is a complete black box, even for the people who created them.
hyperpape | 22 hours ago
While I agree with the basic thesis of this post and have said that same thing to friends, I think this line is wrong, and makes a claim that's much too strong
If I say "never use the getFoo() api", this statement has a meaning that can be studied (https://plato.stanford.edu/entries/logic-deontic/). The challenge is that even with a deterministic LLM, will I know:
This is not to deny that natural language lends itself to vagueness, and code is sometimes more suitable for the things that we want to do as developers (But not always! There is a reason that we communicate with other developers in natural language).
isaacvando | 21 hours ago
Yeah I could have definitely improved that statement. Perhaps "formal semantics" is better. But I didn't want to say that because there are plenty of languages without a formalized semantics where the expectations about what's expected are still clear.
lifepillar | a day ago
Is setting the temperature to zero enough to make an LLM deterministic? There's a lot of parallel numericalcomputation that is likely scheduled differently in each execution. Is it possible that operations accumulate different rounding errors, and that those errors compound to produce different results?
rsalmond | a day ago
Not according to this: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
isaacvando | 22 hours ago
Yeah I mention in the caveat section that the approaches I show don’t actually guarantee determinism. But I think that’s fine as creating a “deterministic LLM” is just a tool for argument.
bobpoekert | 15 hours ago
Generating programs from specifications is called Program Synthesis and it goes all the way back to the days of Alonzo Church. Program Synthesis is usually framed in terms of logic, but Genetic Programming is essentially the same problem framed in terms of optimization. A Field Guide to Genetic Programming is a good introduction to that topic. One can think of coding agents like Claude Code as GP systems that use an LLM to do guided search, and whose objective functions are likelihood of the program given a prompt.
Using GP it's been possible for decades to generate the code that makes a test suite pass.
spc476 | a day ago
Nondeterminism is still the problem, not that prompts lack semantics.
I've made both those arguments against LLMs, and at the time, I was assured by the pro-LLM side that those were silly! No one would ever do that! Yet, it's common for both to be true. But to the second argument, I counter with "You mean you need discipline to review the LLM output? Just like you need discipline to write safe C code?"
isaacvando | a day ago
Why do you think nondeterminism is still the problem?
skobes | a day ago
It seems to me that your blog post is still expressing a nondeterminism argument, just focusing on the nondeterminism of program behavior as opposed to compiler output.
isaacvando | a day ago
Sorry, I’m not following. Can you elaborate? I didn’t say anything about making program behavior nondeterministic, only about making compiler output nondeterministic in a way that doesn’t change program behavior (according to the language semantics).
skobes | a day ago
You believe the "real problem" is that LLMs make no promises about a prompt which lacks semantics. Isn't that another way of saying that the prompt does not determine the behavior of the program in a predictable way?
isaacvando | a day ago
Absolutely, but “determine the behavior of the program in a predictable way” is not the same thing as being deterministic. As I showed in the post, it’s easy to make a deterministic LLM. Prompts to that deterministic LLM still definitely do not determine the behavior of the program in a predictable way.
k749gtnc9l3w | a day ago
I guess an extreme determinism-vs-semantics distinguisher would be «SHA512 of each 4KiB block of source, as x86-64 machine code» — absolutely deterministic, absolutely standartised, absolutely semantic-less… LLMs are less deterministic and closer to having semantics than that, and easier to get something somewhat useful, so it's semantics not determinism.
ETA: when I say standartised, I mean compiler behaviour, target is somewhat scary but mostly documented.
nrdxp | 20 hours ago
Are you seriously suggesting that natural languages don't have meaningful semantics? I'd say your conclusion suffers from the same flaw you drew from non-determinism.
Furthermore, it also applies to compilers. Machine languages, as well as byte-code has semantics, so why not review it?
Overall this might serve as a useful heuristic (maybe), but it's not more or less accurate than the model you are trying to replace as far as I can tell.
I would say the main reason people are compelled to review LLM generated code is (lack of) trust, and pure habit.
josephjnk | 19 hours ago
“Semantics”, unsurprisingly, is a word with multiple meanings. The author is clearly using it to refer to formal semantics. For example, the Java specification which they repeatedly mention in their post. Acting as though the author doesn’t know that words mean things is nitpicky and insulting.
Corbin | 19 hours ago
The semantics of natural languages is physical reality. But physical reality does not fit into GPU VRAM. It follows quite neatly that, while we can say that the latent space of a probabilistic model tightly approximates the semantics of natural languages, it can't possibly be a faithful encoding!
marcecoll | 11 hours ago
I'm sorry but I don't follow. Physical reality does not fit my brain either so I guess my brain also approximates the semantics of natural languages. So I guess natural languages don't exist except in the mind of an abstract all-knowing god?
Corbin | 7 hours ago
Of course natural languages exist as syntax. However, there is no objective semantics for any natural language, since reality is not objective. (Additionally, all-knowing gods are impossible as a consequence of Tarski's Undefinability.)