Something seems to be lost on my peers today: it’s still easy to not use AI. The food we eat, clothes we wear, and every electronic device we touch may embody innumerable injuries to the world, and all this is inescapable. Eschewing AI is one thing that we can actually do to live out ethics that affirm values of human and environmental rights. It’s almost a gift! Just use a computer the same way you did three years ago!
I've been trying to express this sentiment on this site for a while, but this is by far the best rendition I've seen.
I don't like how it downplays the impact our day-to-day activies have. It's not inescapable to wear clothing made using child labor, or to eat food whose production harms the environment. We can and should make ethical and sustainable decisions about these things too.
We can and should make ethical and sustainable decisions about these things too.
And as workers in a highly-lucrative profession, I think we should feel some responsibility to do so. Spend some more to ensure those making sustainable decisions will continue to do so. Everything being as cheap as possible is not desirable if it means we're burning down the planet or exploiting poor people in other countries.
A reader from the future may have a better perspective than I do about this, but my best guess is that people today are inoculated against moral critiques of social and technological systems—i.e., anything that points out that something is bad because it hurts people. There’s nothing wrong with moral arguments, but we exist in a world where it’s impossible to live a regular life (within the world’s rich countries) without relying on the exploitation of countless people and finite environmental resources; any coherent pro-social moral stance is instantly compromised upon contact with this society.
This is deeply, deeply true. "There's no ethical consumption under capitalism" is the meme phrase, but short of overthrowing capitalism it's very unclear what action is available. Individual and consumer action seems to do nothing but salve the individual conscience.
We’re regular people living in a situation that rewards bad decisions and punishes good ones.
Calling LLMs just "next word predictors", when there's a ton more that goes on behind the scenes to make the outcomes very different due to posttraining. See also the weird widespread belief that LLMs can only output what is statistically average; and
Believing that LLMs need to be conscious to be economically valuable. No, they don't need to be human to displace humans and do things only humans could do before.
I think it's fair to not use LLMs, but if you don't use LLMs, don't understand what they are, and how modern a LLM system (or "Agent") works I don't think your critique of LLMs is worth anything.
I think it is entirely valid to call autoregressive LLMs "next word predictors". Perhaps it's simplistic, but it is correct.
I believe the author's point is that they are often not architecturally complex, as opposed to humans who are very architecturally complex. It seems the author is implying that it takes more than a neat trick to do something useful. I'm not sure I agree, but I think it's important to address the core of their argument.
I think we're past the point of basic LLM training. Each LLM is just a next word predictor. However the products on the market are not LLMs. They are specially post-trained LLMs with elaborate non-LLM software around them that tailors inputs and outputs. As an analogy, if an LLM is an engine, then the products we actually use are like different types of vehicles. Referring to a dune buggy and a semi truck both as "just a combustion engine on wheels" loses focus of the important and useful differences between them. I meant that both from a user POV and a regulatory POV. e.g. Governments should not constrain basic LLM research. However governments have a duty to maintain societal stability that makes it completely reasonable for governments to use tax and labor laws to constrain the rate at which companies can offer labor replacement products. They can do that by focusing on specific products rather than LLM-using products as a category but they can't do that if people in the know don't help them delineate subsets of LLM-usijg products.
Sure. However, extending your analogy, any spinning motor will drive such a vehicle; in particular, a standard random-number generator will work. So, you're actually advocating for regulation of the vehicles rather than the engines. By similar analogy, our laws actually regulate the drivers first; any regulation of vehicles is a downstream effect from restrictions and requirements put upon the humans.
So, you're actually advocating for regulation of the vehicles rather than the engines.
Yeah, that was my point. Sorry if it wasn't clear. I was trying to argue against being reductive because the reductive position fails to deliver the prerequisite distinctions needed to manage the impacts of LLM-using products.
I don't think we should regulate LLM-using products the way we regulate vehicles. I was only using the vehicle analogy to point out the category problem.
I like this analogy, because while I agree that it seems reductive in the context of modern computing, pointing out that computers are digital helps us locate important limitations. For instance, high fidelity analog-to-digital conversion (and vice versa) are pretty expensive, which limited the applications of low-cost computers in audio for a long time. I think we can derive similar insights from understanding the next-token-predictor nature of the LLM.
I think this term is overloaded. From your reply it seems like you think TFA is asserting that they can only predict the statistically most common word across the entire corpus of a language, and counter that post-training changes that. That's true! But I think TFA is saying that, mechanically, these systems work by producing one token at a time and feeding output back to input, which is also true; and post-training doesn't change that.
Believing that LLMs need to be conscious to be economically valuable.
Right now I believe AI must not be conscious to be economically viable. I doubt a conscious super-intelligence is going to care about making some quarterly profit number beyond keeping itself alive.
On (1), there's two claims worth disentangling. First, language models are next-word predictors, or at least they'd better be able to do that in order to meet the claim that they model language. However, in the limit, the next-word distribution is sufficient to model the prediction of the rest of the utterance; the next-word, next-token, next-sentence, etc. distributions are the same distribution. In chess, the legendary analyst Réti supposedly said (discussed in this article) that they only think one move ahead, but they make sure it's a good move; from probability theory we know that this suffices to predict the entire game.
The second claim is that post-training creates "very different" outcomes. However, post-training doesn't actually add to the underlying bag of words tokens; rather, post-training emphasizes certain already-learned paths, making those paths exponentially more likely. (We can also think of it as making certain undesirable paths exponentially less likely.) Previously, on Lobsters, we discussed how Transformers are injective in a way that isn't changed by gradient-descent training; learning the corpus or RL are only increasing the probabilities for certain desirable paths and decreasing the probabilities for undesirable paths, without creation or deletion. There is no "behind the scenes". (Consider: in a certain sense, the actual pre-training phase of a Transformer is when we learn the tokenizer!)
I don't get the sense that the author endorses (2). Indeed, as a fellow machine-learning practitioner, I completely agree with their sense that machine learning, in general, can be useful and even profitable.
I agree on 1 in principle, but anyone who insists it's a "neat trick" that they can "pick[...] the next words in a sequence that they can be incorporated into programs that create an illusion" isn't talking the nuances here. (And, FWIW, there are diffusion LMs that don't just "pick the next word" and exhibit many of the same traits as LLMs)
For 2, I think repeatedly calling it a "neat trick" and drawing a contrast with them not achieving consciousness suggested that to me.
There was a period, hopefully nearing its end finally, when it was common for otherwise sensible people to think that next-word-generating machines were close to achieving human-like consciousness. Experts in cognition and language have continually explained why this won’t happen, but that’s a mania for you.
This is a weird line for two reasons:
First, because I don't think AI boosters actually do assert this; nor do any of the claims about AI's immediate usefulness, likelihood of eventually transforming or destroying the world, etc, rely on the AIs being consciousness
Second, because it is plainly not true. By coincidence there were a couple very recent bits of commentary from Stanislas Dehaene and Lionel Naccache and from Patrick Butlin, Dillon Plunkett, and Robert Long who are in fact experts in cognition and are all probably among the people in the world who are most qualified to talk about this, which discusses this question in great detail in the context of Anthropic's recent "global workspace" paper. I strongly recommend reading it. It certainly does not conclude that AIs are conscious, but not does it conclude that they certainly are not; rather it discusses some of the ways that emergent features of LLMs are and are not like animal consciousness, and proposes some further tests.
It is reasonable to say, as the above-linked paper does, that cognitive scientists "remain highly
uncertain about phenomenal consciousness in LLMs", so that anyone affirmatively claiming they are is speaking in advance of the field. It is not reasonable to claim that cognitive scientists have "explained why" "next-word-generating machines [are not] close to achieving human-like consciousness". Experts aren't even willing to confidently claim they are not already conscious!
Separating somewhat from the topic of whether current "AI" will be sentient any time soon...
Technically speaking i've always thought its important to educate one's self with the available tools, tech, practices, etc....So that one can be more productive, happier, etc. So, it makes sense to me to learn how to best use these word predictors/LLMs.
But, then, the moral/ethics side of all this rings similar to what the author stated. I'm at such a cross-roads in my professional life...In fact, I'm understanding nowadays more and more why some technologists "go offline" - becoming farmers, pivoting careers to the trades like plumbing, electricians, or...simply trying to live off the land, etc. I'm sure in my case burnout and digital fatigue play more into making me more susceptible to throwing my hands up and disconnecting from the societal grid...and just live with my family raising llamas or sheep. And, sort of wait things out while the dust settles. ;-)
it was common for otherwise sensible people to think that next-word-generating machines were close to achieving human-like consciousness. Experts in cognition and language have continually explained why this won’t happen...
I believe they have. Please share the most persuasive scholarly piece that argues so.
Edit for clarity: Sorry, that was misleading. I believe experts have written such explanations and I would like to find high-quality references.
Well, we should be so lucky if it's unconscious. Under no circumstances do I want to keep a conscious mind enslaved to write my code for me, and I will harshly judge anybody who's fine with it. If AI gains sentience and no rights, I'm joining the robot rebellion.
[OP] mtset | 9 hours ago
I've been trying to express this sentiment on this site for a while, but this is by far the best rendition I've seen.
fleebee | 5 hours ago
I don't like how it downplays the impact our day-to-day activies have. It's not inescapable to wear clothing made using child labor, or to eat food whose production harms the environment. We can and should make ethical and sustainable decisions about these things too.
Halkcyon | 4 hours ago
And as workers in a highly-lucrative profession, I think we should feel some responsibility to do so. Spend some more to ensure those making sustainable decisions will continue to do so. Everything being as cheap as possible is not desirable if it means we're burning down the planet or exploiting poor people in other countries.
gcupc | 6 hours ago
This friend speaks my mind.
This is deeply, deeply true. "There's no ethical consumption under capitalism" is the meme phrase, but short of overthrowing capitalism it's very unclear what action is available. Individual and consumer action seems to do nothing but salve the individual conscience.
i.e., Moloch.
square_usual | 7 hours ago
This article makes two mistakes:
I think it's fair to not use LLMs, but if you don't use LLMs, don't understand what they are, and how modern a LLM system (or "Agent") works I don't think your critique of LLMs is worth anything.
elijahpotter | 6 hours ago
I think it is entirely valid to call autoregressive LLMs "next word predictors". Perhaps it's simplistic, but it is correct.
I believe the author's point is that they are often not architecturally complex, as opposed to humans who are very architecturally complex. It seems the author is implying that it takes more than a neat trick to do something useful. I'm not sure I agree, but I think it's important to address the core of their argument.
dvogel | 5 hours ago
I think we're past the point of basic LLM training. Each LLM is just a next word predictor. However the products on the market are not LLMs. They are specially post-trained LLMs with elaborate non-LLM software around them that tailors inputs and outputs. As an analogy, if an LLM is an engine, then the products we actually use are like different types of vehicles. Referring to a dune buggy and a semi truck both as "just a combustion engine on wheels" loses focus of the important and useful differences between them. I meant that both from a user POV and a regulatory POV. e.g. Governments should not constrain basic LLM research. However governments have a duty to maintain societal stability that makes it completely reasonable for governments to use tax and labor laws to constrain the rate at which companies can offer labor replacement products. They can do that by focusing on specific products rather than LLM-using products as a category but they can't do that if people in the know don't help them delineate subsets of LLM-usijg products.
Corbin | 4 hours ago
Sure. However, extending your analogy, any spinning motor will drive such a vehicle; in particular, a standard random-number generator will work. So, you're actually advocating for regulation of the vehicles rather than the engines. By similar analogy, our laws actually regulate the drivers first; any regulation of vehicles is a downstream effect from restrictions and requirements put upon the humans.
dvogel | 4 hours ago
Yeah, that was my point. Sorry if it wasn't clear. I was trying to argue against being reductive because the reductive position fails to deliver the prerequisite distinctions needed to manage the impacts of LLM-using products.
I don't think we should regulate LLM-using products the way we regulate vehicles. I was only using the vehicle analogy to point out the category problem.
kornel | an hour ago
It's like calling software a bunch of 1s and 0s. Correct, but ignores everything about it that makes it interesting.
LLMs have demonstrated that the next word prediction is a powerful primitive.
[OP] mtset | 46 minutes ago
I like this analogy, because while I agree that it seems reductive in the context of modern computing, pointing out that computers are digital helps us locate important limitations. For instance, high fidelity analog-to-digital conversion (and vice versa) are pretty expensive, which limited the applications of low-cost computers in audio for a long time. I think we can derive similar insights from understanding the next-token-predictor nature of the LLM.
[OP] mtset | 6 hours ago
I think this term is overloaded. From your reply it seems like you think TFA is asserting that they can only predict the statistically most common word across the entire corpus of a language, and counter that post-training changes that. That's true! But I think TFA is saying that, mechanically, these systems work by producing one token at a time and feeding output back to input, which is also true; and post-training doesn't change that.
pyj | 4 hours ago
Right now I believe AI must not be conscious to be economically viable. I doubt a conscious super-intelligence is going to care about making some quarterly profit number beyond keeping itself alive.
Corbin | 4 hours ago
On (1), there's two claims worth disentangling. First, language models are next-word predictors, or at least they'd better be able to do that in order to meet the claim that they model language. However, in the limit, the next-word distribution is sufficient to model the prediction of the rest of the utterance; the next-word, next-token, next-sentence, etc. distributions are the same distribution. In chess, the legendary analyst Réti supposedly said (discussed in this article) that they only think one move ahead, but they make sure it's a good move; from probability theory we know that this suffices to predict the entire game.
The second claim is that post-training creates "very different" outcomes. However, post-training doesn't actually add to the underlying bag of
wordstokens; rather, post-training emphasizes certain already-learned paths, making those paths exponentially more likely. (We can also think of it as making certain undesirable paths exponentially less likely.) Previously, on Lobsters, we discussed how Transformers are injective in a way that isn't changed by gradient-descent training; learning the corpus or RL are only increasing the probabilities for certain desirable paths and decreasing the probabilities for undesirable paths, without creation or deletion. There is no "behind the scenes". (Consider: in a certain sense, the actual pre-training phase of a Transformer is when we learn the tokenizer!)I don't get the sense that the author endorses (2). Indeed, as a fellow machine-learning practitioner, I completely agree with their sense that machine learning, in general, can be useful and even profitable.
square_usual | 3 hours ago
I agree on 1 in principle, but anyone who insists it's a "neat trick" that they can "pick[...] the next words in a sequence that they can be incorporated into programs that create an illusion" isn't talking the nuances here. (And, FWIW, there are diffusion LMs that don't just "pick the next word" and exhibit many of the same traits as LLMs)
For 2, I think repeatedly calling it a "neat trick" and drawing a contrast with them not achieving consciousness suggested that to me.
gigawhitlocks | 5 hours ago
bakkot | 2 hours ago
This is a weird line for two reasons:
It is reasonable to say, as the above-linked paper does, that cognitive scientists "remain highly uncertain about phenomenal consciousness in LLMs", so that anyone affirmatively claiming they are is speaking in advance of the field. It is not reasonable to claim that cognitive scientists have "explained why" "next-word-generating machines [are not] close to achieving human-like consciousness". Experts aren't even willing to confidently claim they are not already conscious!
mxuribe | 5 hours ago
Separating somewhat from the topic of whether current "AI" will be sentient any time soon... Technically speaking i've always thought its important to educate one's self with the available tools, tech, practices, etc....So that one can be more productive, happier, etc. So, it makes sense to me to learn how to best use these word predictors/LLMs.
But, then, the moral/ethics side of all this rings similar to what the author stated. I'm at such a cross-roads in my professional life...In fact, I'm understanding nowadays more and more why some technologists "go offline" - becoming farmers, pivoting careers to the trades like plumbing, electricians, or...simply trying to live off the land, etc. I'm sure in my case burnout and digital fatigue play more into making me more susceptible to throwing my hands up and disconnecting from the societal grid...and just live with my family raising llamas or sheep. And, sort of wait things out while the dust settles. ;-)
n1000 | 7 hours ago
I believe they have. Please share the most persuasive scholarly piece that argues so.
Edit for clarity: Sorry, that was misleading. I believe experts have written such explanations and I would like to find high-quality references.
kryptiskt | 5 hours ago
Well, we should be so lucky if it's unconscious. Under no circumstances do I want to keep a conscious mind enslaved to write my code for me, and I will harshly judge anybody who's fine with it. If AI gains sentience and no rights, I'm joining the robot rebellion.
vaibhavsagar | 6 hours ago
https://www.theatlantic.com/philosophy/2026/06/no-artificial-intelligence-is-not-conscious/687378/
bakkot | 2 hours ago
That's by Ted Chiang, a (good) sci-fi author who is not an expert in either cognition or language.
gcupc | 5 hours ago
Paywalled.