Despite mentioning a philosopher (good pick!), this is just creative writing rehashing one side of the hard problem -- or, more specifically, restating the dogma that Turing wrote his most famous paper to debunk. It's really good creative writing, at least!
There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here:
> The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is.
Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
The output method is a probability distribution of the next token. But that tells you nothing about what's going on on the inside
You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought
I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
How would that process be the result of intelligent thought?
Presumably everyone in the chain would have their own idea of what the next word/token should be. They would each try to push it in that direction using their only lever, that single token. None of that guarantees that the output is syntactically correct, nor grammatically correct. Note that LLMs at least by the way they draw their tokens have that pretty much guaranteed. So that would be a regression even from what LLMs are capable of right now. But beyond that, even if it ends up being a correct sentence, it would not be a result of intelligent thought. Even if every agent in the process was intelligent, the process is not itself a use of that intelligence. Human beings can be part of purely mechanical processes, that doesn't make the mechanisms suddenly an exhibition of intelligent thought.
This also applies to society and human history itself as processes:
“History is made in such a way that the final result always arises from conflicts between many individual wills, of which each in turn has been made what it is by a host of particular conditions of life. Thus there are innumerable intersecting forces, an infinite series of parallelograms of forces which give rise to one resultant — the historical event. This may again itself be viewed as the product of a power which works as a whole unconsciously and without volition. For what each individual wills is obstructed by everyone else, and what emerges is something that no one willed. Thus history has proceeded hitherto in the manner of a natural process and is essentially subject to the same laws of motion. But from the fact that the wills of individuals — each of whom desires what he is impelled to by his physical constitution and external, in the last resort economic, circumstances (either his own personal circumstances or those of society in general) — do not attain what they want, but are merged into an aggregate mean, a common resultant, it must not be concluded that they are equal to zero. On the contrary, each contributes to the resultant and is to this extent included in it."
In short, even in the case of humans, which are universally (by humans) recognized as intelligent, such a process would not exhibit intelligent thought.
But deciding the next token is not a merely mechanical process. Even if the string of tokens from our swapped out humans ends up being syntactically incorrect, it's due to the combination of intelligently selected tokens. (Presumably intelligent human programmers seem to make syntactic errors)
Yes, we can explain the bugs by looking at the code that we put in and the data it bugs out on. In a similar way, we can explain the behavior of LLMs by look at their code and the data the code operates upon.
You said you know everything about it. Then you should not have written bugs in the first place. How are you sure a system we wrote didn't emerge consciousness, if you can't even prove a system you wrote doesn't have classic logic bugs?
Yes, we know everything we put there. But we cannot consider all the possible inputs and all the possible state that the code should handle. Bugs come from that limitation.
> How are you sure a system we wrote didn't emerge consciousness,
Sure, as I said elsewhere, rocks could be conscious for all we know. Keep believing that if it makes you feel good.
> Indeed, the chatbot is less real than the character, because the character is the product of another mind, while the chatbot's words are the product of complex mathematical operations conducted over a massive database of all the words humans have uttered, arranged by their frequency in relation to one another.
How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
I frequently say what you say, but there is a more charitable reading: today's AIs are not "minds", as they are not stateful. Maybe continual learning ones (like the mini AGI exhibited today on the HN front page) will be perceived as "soulful".
I think the working space in a given chat or session still counts as mind enough, but your argument is different, in that it argues about specific architectures rather than simply stating "software therefore cannot be conscious".
Believing a specific architecture is not capable of consciousness is a different matter. I am seeing statements that simply state things like "it is software or a machine and therefore cannot be conscious" without any further clarification. That clearly betrays a belief in souls.
Why? What's so strange about a human brain? It is a physical system of molecules running electrical and chemical signals. We already have direct proof that physical systems support consciousness. So why do you believe other physical systems can't have consciousness? Give me your definition of what consciousness is.
Yes but you are not me, I am me. I don't know you are conscious or can "feel". Both you and an LLM present themselves using words and other external indications only. If you argue brain scans, then similar experiments have been done on LLMs too.
>but how do I know you are not faking the outputs of feeling?
You won't..but it does not matter.
>Where is your disproof that other physical systems can't hold consciousness?
I said no such thing. Just that there is no reason to think that LLMs are conscious. If we find some natural or artificial entity with some unknown and un-inspectable thing inside it, that behaves in an intelligent manner, I would not be sure that it is not conscious.
Why unknown? The human brain is known and inspectable, then you should say it is not conscious. You haven't yet shown why you think an LLM can't be conscious. You are just saying "just because" with no reason or parameters given.
So what is the reason to not think it can be conscious? Brains are physical systems, so are LLMs.
How is a human brain not inspectable? Tell me what part of it is beyond inspection? You have been making up absurd non sequiturs throughout this chat, ending with this beyond absurd statement now.
So are rocks. Are rocks conscious? Idol worshipers might have a point after all!
>Tell me what part of it is beyond inspection?
Oh, you can dissect a brain as much as you like. Is that what you mean by "inspection". I was thinking more like "How does it work". In that sense, we don't know a lot.
Rocks don't talk to me and think and act intelligently. Current LLM's do.
We don't know a lot about how chaotic systems work either in terms of predicting its outcome even if we know every individual step. The argument you gave for the brain equally applies to the LLMs, or any other complex system. So your premise is wrong.
Define consciousness right here, otherwise we have nothing else to discuss. You keep making up undefined crap and circular "it is it, because it is it" nonsense without proof. If you cannot immediately and clearly define consciousness in a testable clear manner right now, I will ignore and stop all conversation with you until you do.
Potentially past models could have been. None before came close to being this articulate and able to think and talk on our level before, and do things beyond us like difficult math problems.
You said "rocks don't talk to me and think and act intelligently." But a coma patient who is conscious would be unable to talk and act in an intelligent manner. That does not means that they are not conscious.
So consciousness does not require externally observable behavior. That was my point. So a rock could be conscious for all we know..
Also, If you are irritated by my "crap", why don't you stop replying? Are you paid or somehow obliged to correct every crap talker on the internet or something?
I am open to the concept of every entity having some measure of consciousness. At least it is internally consistent. As of yet though, if you want to go anything at all beyond "Only I exist", you will have to see that certain organizations of physical systems appear to have consciousness and certain organizations don't. You haven't yet given any definition still of what exactly you think consciousness is, and are trying to act special by thinking your own, probably not well formed, definition is the only right one.
It is a waste of time to talk to you as you keep refusing to define what you think consciousness is but keep talking as if its a settled matter that you know what is and isn't conscious. So correct, I see no further point in talking.
And I am sure that lolakutty will momentarily bestow the world with a consciousness test, solving a problem that has eluded millions of top thinking man hours over millennia of trying...
However, I feel rather certain, that deep down the proposed test will just be another "is it chemical goo?"
The test can only be done on oneself, and not on others. So I know I am conscious, and I am made up of a brain, among other things. So I know for a fact that this set of things that I am comprised of, can generate consciousness.
But then you are stuck being the only conscious entity...
To extended anywhere beyond the firm ground of "I know I am conscious", you need to step onto the progressively shaky ground of assumptions, which just amounts to "tests of consciousness"
i.e. I have a chemical goo brain that is conscious, therefore other chemical goo brains are conscious...
It is possible that you are the only conscious entity in the universe, and you have no tools whatsoever to disprove that. That knowledge is no help in determining the consciousness of anything besides yourself.
>Sure, if your architecture include hardware that is known to be capable of generation consciousness, the no one will say that.
Then circling back to your original comment, no architecture can include hardware that is known to be conscious, unless that architecture includes your personal brain, because lolakutty's brain is the only thing in the universe that is capable of generating consciousness...
Which is a totally meaningless statement. This is like im14andthisisdeep levels of philosophy here...the "Nothing is real therefore everything is real" kinda stoner riffing
He gave the classic "I feel" argument. The problem is I am not him. To me lolakutty and an LLM both can only present their feelings or state via external indications. Both he and an LLM can say they are conscious. Who if any of them is lying?
Nah, it "betrays" an awareness of something implicitly obvious: All the software/machines we might discuss are ones from ~2026, rather than ones from ~2226.
Similarly, "pets can't do my taxes" is a reasonable statement, because it is implicit that it doesn't include hundreds of years of genetic engineering and cyborg implants.
Pets can't speak human languages or do thinking on logic or natural logic. Current AI's can. Its a very bad example. People saying "current archs are limited" are much less than people outright claiming "software therefore not conscious". And I haven't yet seen any proof why the current AI's already don't have some consciousness or not.
> Pets can't speak [...] Current AI's can. It's a very bad example.
You've misunderstood the comparison analogy. It's showing two situations where implicit timeframes are important when people talk about what is "true", where the second one makes the importance extra-obvious.
"Pets doing taxes" could just as easily be swapped with "cold fusion generator in your garage" or "rocket-ship for personal commutes."
If you're trying to figure out how LLM weights are like the weights of a rocket payload, then you're going in the wrong direction.
I get what you were trying to say, I still disagree, its an open question and I feel existing LLMs already might possess some consciousness. I cannot prove it of course so I am not saying so definitively. You haven't given any disproof of the same either.
Those who think future systems could have consciousness would explicitly say so, indeed I have talked with a few who clarified as such. On the other hand I have seen a lot of statements that simply state without further clarification "software therefore not conscious".
If you think current physics is insufficient but future physics might be sufficient, I can accept that but its still an unfounded belief. If you truly believe something "more" is needed, then you are placing a bet that always lost for centuries. Bets in anything supernatural or aphysical have always and consistently failed till now.
If you think "souls" have some physical mechanism then they are not souls by the definition I am using. We would then agree, though I do not yet think we have found reason for new physical laws to be required, however that can of course be a possibility.
It should have been clear from our discussion when I said "souls" I meant belief in the non-physical. Magic juice in the brain that is unexplainable by physics at all.
Being unexplainable by current physics but explainable by future physics is not a "soul" by that definition.
Go ask the people who believe in it. I for one cannot understand how something aphysical can affect our thoughts and decisions. But some think consciousness resides outside of physical systems entirely.
> How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
I imagine it's the difference between a chef combining ingredients with intentionality vs a person going to multiple fast food restaurants and blending everything together.
The AI generated character is a weighted average of many characters written by humans.
Which are not created from nothing. People write characters based on a combination of other fictional characters, real characters, and perhaps some 'RNG'.
Seems the distinction is that the AI generated character can not have any direct bearing on reality, because the LLM never got to know anyone directly. Not derived directly from experiences rooted in reality.
The product of an individual's mind is not the same as the mathematical average of the products of everyone's minds. The latter is obviously going to lack any of the uniqueness of the former.
Current models absolutely suck at writing fictional characters, by the way. And they're getting worse. Seriously, try it yourself: have Claude write a story with a decent amount of dialogue involving character A, then have it write another story involving a completely different character B, then compare the dialogue between the two. You'll quickly notice the same blatantly unnatural speech patterns in both.
Whenever a human writes/improvises a fictional character, it is typically to communicate some concept or goal to other people. The fictional character--even a simplified author-insert speaking "directly"--is part of how the medium works.
So in a way your question is equivalent to: "Why do humans have intents and minds but LLMs don't?"
> Of course, the more you know about a subject, the less convincing the AI's responses are.
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
I think part of this comes from the fact that LLMs are surprisingly good at logic but roughly about as good as expected on information accuracy.
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
This sounds plausible. And it's also very fixable!
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
The more you know about a subject the better you can prompt AI, steer it toward the correct path, and recognize when it hallucinates or strays. Current generation AI is an automated memory-enhancement and thinking-accelerator tool, not a substitute for understanding or something that eliminates the need to think. A "mech suit for your brain" is the best analogy I've heard.
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
Aren't the recent results in mathematics actually stronger evidence for his point? Although the models may be capable of generating proofs they aren't coming out with the same level of quality of a human discovered and communicated proof. Providing a gobbledy-gook yet technically correct proof (generated at least in part by brute force) lacks the qualities of an expert produced proof because they fail to communicate insight or understanding about why the theorem is true.
Gaining and successfully communicating insight and understanding from a proof you discovered is additional work that human mathematicians do. It's not just some side-product of proof-finding (at least not to the degree usually needed to publish). That AI models don't provide this is mostly proof that the model wasn't asked to do this work. Either because the prompter didn't know or didn't care
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
It seems to me that often experts from some field will think less of other experts, basically because they have built a different understanding framework. So they both may be equally competent but perceive the other as less competent, and that is just based on the material, excluding some ego stuff.
> (e.g. writing English in a captivating way) because I read their output and have taste.
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
I can tell whether AI responses in subject matter I am (or was) not familiar with are correct when I can "make them work" for me, i.e. whether they actually solve my problem, and on the whole they have been pretty solid.
Coding of course is something I know well and it obviously works very well for me there, but computer vision was not, and it has helped me solve lots of useful, bespoke problems because I can very literally "see" if it works.
However, it has even helped me solve problems in arbitrary matters far outside my expertise. Choice example: a complicated multi-airline, multi-jurisdiction flight delay compensation case which companies like AirHelp turned away, and ChatGPT got me literally hundreds of dollars that neither airline was willing to hand out. It told me what to say to whom and why, and when I said it, the responsible airline capitulated.
What could be more convincing than cold, hard $$$?
The trick of course, is to figure out how to validate the output of the AI, which can take some effort on our part. But many would rather downplay the technology than take an honest crack at making it work for them, and I suspect it's because they're already prejudiced against it or their incentives are otherwise not aligned.
I think to me LLMs had the effect of noticing much more the author, the intention behind human-made works of art (books, movies etc.). Before LLMs, I used to frequently consume media in a way as it were generated by a mindless process. Now it's like everything which is not AI-generated has more meaning than ever before, a bit like hypomania.
I'm actually using that as a catalyst for my own writing; beauty/human-ness in its imperfection. Prior to LLMs and their cultural craze, I harbored a fear that my writing would allow for someone to draw a box around me and mark me as a bore, dullard or of lacking originality.
Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
I always feared to pick up a pen because I looked at Borges, Tolkien, and such. Their talent and works of art were things I felt I could NEVER achieve.
Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
Even before AI could write them, more slop books written by humans were added to Amazon each day than a human being could possibly read before the next day (probably by a few orders of magnitude at least).
Don't just create original works for the sake of creating more original works -- there are plenty of them already. You should write if you have something to say.
absolubtely. Any random junior consultant can tell you what the book tells you you should do. If you want to actually do anything worth doing, you need to step beyond that in a few, limited areas, and follow convention everywhere else. Which areas? pay a senior engineer and they'll find them.
surely the LLM can do that. It is RL'd against some reward, if the known strategies are clearly suboptimal with easy improvement, it'll find it most likely
Professional tech Cassandra discovers ELIZA and Searle; coins a term for it. More at 10...
Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.
An awful lot of effort has gone into making LLMs present as human/intelligent. Without that they would just be a prose / code / image generator and/or search assistant, and nobody would be pouring 100s of billions into the technology with the end goal of replacing expensive human labour.
> But there's a second hurdle that makes it hard for a small but important subset of humanity to understand that chatbots aren't people: the billionaires to whom nearly everyone isn't a real person. These solipsists see chatbots as being equivalent (or even superior) to humans, because they don't think most humans are fully people, either
Never thought about it that way before, but the more I do, the more sense it makes.
I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.
So I see two categories of people here. The first is "main character syndrome" - people who see themselves as real and everybody else as not. Self-centered, egocentric people are one flavor of this, and sociopaths are another. Other people aren't real to them in the same way that they see themselves as real.
The other kind is more of a philosophical position. If all that exists is the physical universe - say, matter and the laws of physics - then all that humans can be is matter that obeys the laws of physics. Nothing more. Humans are just a machine made of atoms - a very complicated machine, but still just a machine. They aren't intrinsically different from an AI, or a car, or a toaster. When people accept that position, whether consciously or unconsciously, they dehumanize people, including themselves. This is people in your last paragraph.
And as more and more people slip over into this position, society becomes more and more dehumanizing, because people act the way they think. (Not perfectly, not consistently, but the correlation is greater than zero.)
Upvoted for the clear distinction and elucidation of the categories, but...
I have to disagree that a materialist or physicalist philosophical position is inherently dehumanizing. One can believe that we are just electrochemical mechanical beings made of atoms just like all the other matter in the universe, without having to also devalue ourselves and others. I am human and I like being human! I even like other humans and value them. I just don't think there's anything that special about being a human instead of a bonobo or a dolphin or a supermassive black hole.
Maybe I am missing the point of the article, but it seems to me that there's always an intender. Maybe the intender created something that doesnt have intent, but there is always someone behind the scenes that is creates the intent behind the creation that lacks it.
I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.
Wait, hold up. LLMs may be non-deterministic, but they're not _random_.
Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.
LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.
There's deep intentionality behind the models. But it's not the models that hold it.
I think they're pointing to the fact that RLHF means that the intention is from humans and not random.
I'm not sure if it's their intent, but I wonder if one could still consider these artifacts as "intentional", but not individual attention creating them, and rather an aggregate, soupy collective attention.
Obviously, important signal in the human experience is lost there, and we get a soupy middling sort of creation. But it's not random, as I believe the parent was pointing out.
EDIT: overall, I align with the article. am just thinking aloud about the contrarian positions, though not committed to them
Article says that there's no human intention or design directing the output we get, but I don't think that's completely right. It's not human, but the algorithm is like the Human Instrumentality Project: an amalgamation of human intentions.
Scrambled intentionality is nearly impossible to productively analyze.
When you randomly choose a picture to show me, I cannot glean any intent from being shown that specific picture, but I can glean some intent from the set of pictures you could have shown me, and in the relationships among the elements of the given picture you did show. Any randomness cuts out some intention.
When a picture is derived from huge model, any intention is mulched up to a degree that analyzing the picture for meaning is pointless.
> when we interact with an AI, we hallucinate the person on the other side of the interaction. Those hallucinations are far more common and far more consequential than any AI-generated "hallucinations" (these are more properly called "errors" or "defects").
I recently used Claude to study for a technical exam. I had uploaded the official certification guide to Claude and instructed it to answer my questions using only the guide and to cite it's sources from the book when it provided answers. I was using Fable when it was free w/ the pro plan and I was genuinely impressed at how it could explain things when a concept was unclear to me.
I did pass the exam, partly due to this study method. Admittedly, once I passed, I caught myself thinking that I should tell Claude that I passed and then felt embarrassed with myself for thinking that.
I don’t think it’s that silly to tell Claude you passed. That feedback is useful context for that chat session; and could theoretically be used to improve future models.
Exactly this. My first impulse was along the lines of sharing a positive result with a study partner. I did eventually tell it I passed the exam so it would stop providing responses with the assumption I was still working on that exam.
I've encountered this too for home lab projects. I had planned on implementing OPNsense in a VM, then realized it was adding too much complexity and bailed on it.
Later in separate chats about my homelab, the LLM made assumptions that I had already implemented OPNsense in a VM and it was actively running. I think it "assumed" that I had implemented it when I stopped responding in that thread.
The technology is there to assist you. It can provide valuable feedback to you about what aspects of your studying were particularly productive or less so based on your test results. There is real meaning to developing this kind of interaction with an object, no different than how children use dolls to develop prosocial behaviors.
I am talking about the value to the user, who is the one 'hallucinating,' not the inanimate object, as that was the concern of note in the parent comment.
Interesting, I do that and haven't really thought embarrassed about it. I consider it akin to putting away tools once I'm done with them.
Likewise, speaking collaboratively or capturing emotion ("We did it!") would just align with any ongoing interactive and/or personal context of the thread.
I've done this despite also feeling silly about it, but then was surprised to get value from the response. Like something I didn't think about or would have otherwise forgotten to do.
I'm heuristically less prone to continue reading an article starting with a false dichotomy, that is between how you see things as an atheist vs as a person having faith.
It's ironic that real atheists like Richard Dawkins (whose work I believe inspired the title of this post) are public about their own degree of skepticism in favor for the "personhood" of AI. And I'm confident that there are plenty others who are convinced to the tune of total disinterest in the issue of "personhood" altogether—AIs and humans alike.
This particular post is preaching to the choir of which a faith is yet to be determined or named at least. People who don't read Cory Doctorow are going to continue to anthropomorphize AI the same as they do other technologies, constructs and objects.
The obvious way: I think humans (and many animals) are conscious, that LLMs are not and that the difference matters. None of that has anything to do with atheism, on which topic my general view of Dawkins is, to paraphrase The Dude, "you're not wrong, Richard, you're just an asshole".
For what it's worth I wouldn't concede even rectitude to Richard Dawkins (he is both wrong and entirely ass), but it seems his position on AI is more nuanced than that. Or at least there is enough white space around his position to allow for people with stronger beliefs than his to obviate the distinctions between humans, animals and AI altogether.
The good news for Cory, I guess, is that he probably doesn't want to turn this into a philosophical debate and neither do his more serious opponents. It's mostly politics for both sides.
This article is missing discussion of something that I recently read: as people interact with LLMs more, they end up seeing other people as less human.
In other words, the more you talk with claude, the more you tend to interpret all messages as coming from an LLM, and less worthy of respect
Essentialism of interiority is not the antidote to the anthropomorphism of language models.
We cannot attach metaphysical baggage to art, text, or other human output because it has involves engaging in another deception, chiefly that of self alienation and selective forgetting.
1. If you show human output and label it as AI generated, viewers will dismiss and devalue it as lacking interiority.
2. If you show AI output and label it as human generated, viewers will see interiority when the is none.
Together, these failure cases demonstrate that priming determines interiority, not the content of the work. And how could it? The text is merely what's on the page, it doesn't have some intangible essence any more than the value of money exists inside our minds rather than a property of the object itself.
We act as-if things have essential properties, but we should never trick ourselves into believing that they do. To be a true believer in essence is to privilege ones mental model over reality itself. The hubris of such an act is the ultimate in self-centeredness.
> To look upon an AI-generated text or image is to look upon a thing that was extruded, not chosen.
It's totally the autist in me, but the word extrusion doesn't fill me with revulsion like it's supposed to here. Maybe I've just been spending too much time 3D printing things though.
> Feeling sorrow or joy for imaginary people is weird.
No it's not. I read books and they're filled with imaginary people, and I feel for them. I'm pretty sure I'm not the only one out there that does this.
I think it's pretty normal to feel emotions when watching a movie or a tv show, or reading a book. It is not at all weird to feel sorrow or joy for imaginary people.
bbor | 6 hours ago
There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here:
> The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is.
Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
jimbokun | 6 hours ago
Are you saying that is the entirety of what human minds do as well?
vhantz | 6 hours ago
wongarsu | 6 hours ago
You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought
I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
vhantz | 5 hours ago
Presumably everyone in the chain would have their own idea of what the next word/token should be. They would each try to push it in that direction using their only lever, that single token. None of that guarantees that the output is syntactically correct, nor grammatically correct. Note that LLMs at least by the way they draw their tokens have that pretty much guaranteed. So that would be a regression even from what LLMs are capable of right now. But beyond that, even if it ends up being a correct sentence, it would not be a result of intelligent thought. Even if every agent in the process was intelligent, the process is not itself a use of that intelligence. Human beings can be part of purely mechanical processes, that doesn't make the mechanisms suddenly an exhibition of intelligent thought.
This also applies to society and human history itself as processes: “History is made in such a way that the final result always arises from conflicts between many individual wills, of which each in turn has been made what it is by a host of particular conditions of life. Thus there are innumerable intersecting forces, an infinite series of parallelograms of forces which give rise to one resultant — the historical event. This may again itself be viewed as the product of a power which works as a whole unconsciously and without volition. For what each individual wills is obstructed by everyone else, and what emerges is something that no one willed. Thus history has proceeded hitherto in the manner of a natural process and is essentially subject to the same laws of motion. But from the fact that the wills of individuals — each of whom desires what he is impelled to by his physical constitution and external, in the last resort economic, circumstances (either his own personal circumstances or those of society in general) — do not attain what they want, but are merged into an aggregate mean, a common resultant, it must not be concluded that they are equal to zero. On the contrary, each contributes to the resultant and is to this extent included in it."
https://www.marxists.org/archive/marx/works/1890/letters/90_...
vhantz | 5 hours ago
vanuatu | 5 hours ago
lolakutty | 5 hours ago
It does not have to tell us anything. Because we built it..We know what is going on the inside. There is nothing more to it..
hardbass | 4 hours ago
lolakutty | 4 hours ago
There is no mystery anywhere...
hardbass | 3 hours ago
lolakutty | 2 hours ago
hardbass | 2 hours ago
Again, no progress is possible unless you define what consciousness is in your view. I shall have to halt any conversation otherwise.
lolakutty | 2 hours ago
They are not special. You can pile brains too, if that is your thing..
> I shall have to halt any conversation otherwise.
Hey, it is easy. Just don't press the reply button...
hardbass | 3 hours ago
lolakutty | 2 hours ago
Yes, we know everything we put there. But we cannot consider all the possible inputs and all the possible state that the code should handle. Bugs come from that limitation.
> How are you sure a system we wrote didn't emerge consciousness,
Sure, as I said elsewhere, rocks could be conscious for all we know. Keep believing that if it makes you feel good.
delichon | 6 hours ago
How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
WarmWash | 6 hours ago
hardbass | 6 hours ago
bananaflag | 6 hours ago
hardbass | 4 hours ago
Terr_ | 5 hours ago
"Why can't you just admit that you think only tree-parsers have souls!?"
hardbass | 4 hours ago
lolakutty | 4 hours ago
Problem is that we don't know any such hardware..
hardbass | 4 hours ago
lolakutty | 4 hours ago
hardbass | 4 hours ago
lolakutty | 4 hours ago
It should have the ability to feel...
hardbass | 4 hours ago
lolakutty | 4 hours ago
hardbass | 4 hours ago
lolakutty | 4 hours ago
1. That you are conscious.
2. That you have a human body with a brain.
This implies that your body can harbor consciousness. That is what is special about your body in this context. And that answers your question.
hardbass | 3 hours ago
I am sure of my body, I am not sure of you. I can feel, but how do I know you are not faking the outputs of feeling?
1. Scooters can move from a place to a place. 2. That a scooter has a body with an engine.
Therefore scooters are special and cars don't exist.
lolakutty | 3 hours ago
You won't..but it does not matter.
>Where is your disproof that other physical systems can't hold consciousness?
I said no such thing. Just that there is no reason to think that LLMs are conscious. If we find some natural or artificial entity with some unknown and un-inspectable thing inside it, that behaves in an intelligent manner, I would not be sure that it is not conscious.
hardbass | 3 hours ago
lolakutty | 3 hours ago
Human brain is not "inspectable".
> LLM can't be conscious.
Because there is no reason to think so, and suggestion that would be like suggesting the existence of ghosts and goblins.
Extremely unscientific.
hardbass | 3 hours ago
How is a human brain not inspectable? Tell me what part of it is beyond inspection? You have been making up absurd non sequiturs throughout this chat, ending with this beyond absurd statement now.
lolakutty | 3 hours ago
So are rocks. Are rocks conscious? Idol worshipers might have a point after all!
>Tell me what part of it is beyond inspection?
Oh, you can dissect a brain as much as you like. Is that what you mean by "inspection". I was thinking more like "How does it work". In that sense, we don't know a lot.
hardbass | 3 hours ago
We don't know a lot about how chaotic systems work either in terms of predicting its outcome even if we know every individual step. The argument you gave for the brain equally applies to the LLMs, or any other complex system. So your premise is wrong.
lolakutty | 3 hours ago
These are not required for consciousness.
> Current LLM's do.
Computer chat bots are not a new thing. So were all the past ones conscious too?
hardbass | 3 hours ago
Define consciousness right here, otherwise we have nothing else to discuss. You keep making up undefined crap and circular "it is it, because it is it" nonsense without proof. If you cannot immediately and clearly define consciousness in a testable clear manner right now, I will ignore and stop all conversation with you until you do.
Potentially past models could have been. None before came close to being this articulate and able to think and talk on our level before, and do things beyond us like difficult math problems.
lolakutty | 2 hours ago
So consciousness does not require externally observable behavior. That was my point. So a rock could be conscious for all we know..
Also, If you are irritated by my "crap", why don't you stop replying? Are you paid or somehow obliged to correct every crap talker on the internet or something?
hardbass | 2 hours ago
lolakutty | 2 hours ago
I said no such thing..lol...If you are hanging to this, it is a waste of my time to talk to you.
hardbass | 2 hours ago
lolakutty | 2 hours ago
But at least, I gave my definition of consciousness, and at least there is one person who can test for it (oneself).
levitatorius | an hour ago
WarmWash | 4 hours ago
However, I feel rather certain, that deep down the proposed test will just be another "is it chemical goo?"
lolakutty | 4 hours ago
WarmWash | 3 hours ago
To extended anywhere beyond the firm ground of "I know I am conscious", you need to step onto the progressively shaky ground of assumptions, which just amounts to "tests of consciousness"
i.e. I have a chemical goo brain that is conscious, therefore other chemical goo brains are conscious...
It is possible that you are the only conscious entity in the universe, and you have no tools whatsoever to disprove that. That knowledge is no help in determining the consciousness of anything besides yourself.
lolakutty | 3 hours ago
That is correct.
WarmWash | 3 hours ago
Then circling back to your original comment, no architecture can include hardware that is known to be conscious, unless that architecture includes your personal brain, because lolakutty's brain is the only thing in the universe that is capable of generating consciousness...
Which is a totally meaningless statement. This is like im14andthisisdeep levels of philosophy here...the "Nothing is real therefore everything is real" kinda stoner riffing
hardbass | 2 hours ago
lolakutty | 2 hours ago
hardbass | 2 hours ago
hardbass | 4 hours ago
Terr_ | 4 hours ago
Similarly, "pets can't do my taxes" is a reasonable statement, because it is implicit that it doesn't include hundreds of years of genetic engineering and cyborg implants.
hardbass | 4 hours ago
Terr_ | 3 hours ago
You've misunderstood the comparison analogy. It's showing two situations where implicit timeframes are important when people talk about what is "true", where the second one makes the importance extra-obvious.
"Pets doing taxes" could just as easily be swapped with "cold fusion generator in your garage" or "rocket-ship for personal commutes."
If you're trying to figure out how LLM weights are like the weights of a rocket payload, then you're going in the wrong direction.
hardbass | 3 hours ago
Those who think future systems could have consciousness would explicitly say so, indeed I have talked with a few who clarified as such. On the other hand I have seen a lot of statements that simply state without further clarification "software therefore not conscious".
lolakutty | 4 hours ago
hardbass | 4 hours ago
lolakutty | 4 hours ago
It is quite an arbitrary classification. What is so special about "current" knowledge...
hardbass | 4 hours ago
lolakutty | 4 hours ago
Not my words.
hardbass | 4 hours ago
lolakutty | 3 hours ago
You are so amusing....
hardbass | 3 hours ago
lolakutty | 3 hours ago
hardbass | 3 hours ago
lolakutty | 3 hours ago
Kindly share that concrete definition..
hardbass | 3 hours ago
Being unexplainable by current physics but explainable by future physics is not a "soul" by that definition.
lolakutty | 2 hours ago
Now, what is your definition for this?
hardbass | 2 hours ago
lolakutty | 2 hours ago
Or are you just parroting cool sounding words and phrases?
hardbass | 2 hours ago
lolakutty | 2 hours ago
But "non-physical"? I don't have a clue what it means...Why don't you tell me..
novafunc | 6 hours ago
I imagine it's the difference between a chef combining ingredients with intentionality vs a person going to multiple fast food restaurants and blending everything together.
jimbokun | 6 hours ago
It answers itself.
hermitShell | 6 hours ago
Which are not created from nothing. People write characters based on a combination of other fictional characters, real characters, and perhaps some 'RNG'.
Seems the distinction is that the AI generated character can not have any direct bearing on reality, because the LLM never got to know anyone directly. Not derived directly from experiences rooted in reality.
bakugo | 6 hours ago
Current models absolutely suck at writing fictional characters, by the way. And they're getting worse. Seriously, try it yourself: have Claude write a story with a decent amount of dialogue involving character A, then have it write another story involving a completely different character B, then compare the dialogue between the two. You'll quickly notice the same blatantly unnatural speech patterns in both.
Terr_ | 4 hours ago
So in a way your question is equivalent to: "Why do humans have intents and minds but LLMs don't?"
juuular | 3 hours ago
muglug | 6 hours ago
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
howunfortunate | 6 hours ago
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
eru | 6 hours ago
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
netdevphoenix | 5 hours ago
agumonkey | 5 hours ago
the__alchemist | 6 hours ago
api | 6 hours ago
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
doesnotexist | 6 hours ago
wongarsu | 6 hours ago
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
comboy | 6 hours ago
andsoitis | 6 hours ago
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
keeda | an hour ago
Coding of course is something I know well and it obviously works very well for me there, but computer vision was not, and it has helped me solve lots of useful, bespoke problems because I can very literally "see" if it works.
However, it has even helped me solve problems in arbitrary matters far outside my expertise. Choice example: a complicated multi-airline, multi-jurisdiction flight delay compensation case which companies like AirHelp turned away, and ChatGPT got me literally hundreds of dollars that neither airline was willing to hand out. It told me what to say to whom and why, and when I said it, the responsible airline capitulated.
What could be more convincing than cold, hard $$$?
The trick of course, is to figure out how to validate the output of the AI, which can take some effort on our part. But many would rather downplay the technology than take an honest crack at making it work for them, and I suspect it's because they're already prejudiced against it or their incentives are otherwise not aligned.
bananaflag | 6 hours ago
RationPhantoms | 6 hours ago
Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
away0g | 6 hours ago
Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
reverius42 | 2 hours ago
Don't just create original works for the sake of creating more original works -- there are plenty of them already. You should write if you have something to say.
jimbokun | 6 hours ago
I think by design an LLM can’t do that. It’s built to reflect the distribution of the knowledge it has been trained on.
RugnirViking | 6 hours ago
Davidzheng | 5 hours ago
phasefactor | 6 hours ago
Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.
dgellow | 5 hours ago
l1n | 6 hours ago
I wonder if the OP has read https://www.anthropic.com/research/global-workspace - it seems like it directly addresses this
measurablefunc | 5 hours ago
somewhereoutth | 6 hours ago
bakugo | 6 hours ago
Never thought about it that way before, but the more I do, the more sense it makes.
I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.
AnimalMuppet | 4 hours ago
The other kind is more of a philosophical position. If all that exists is the physical universe - say, matter and the laws of physics - then all that humans can be is matter that obeys the laws of physics. Nothing more. Humans are just a machine made of atoms - a very complicated machine, but still just a machine. They aren't intrinsically different from an AI, or a car, or a toaster. When people accept that position, whether consciously or unconsciously, they dehumanize people, including themselves. This is people in your last paragraph.
And as more and more people slip over into this position, society becomes more and more dehumanizing, because people act the way they think. (Not perfectly, not consistently, but the correlation is greater than zero.)
reverius42 | 2 hours ago
I have to disagree that a materialist or physicalist philosophical position is inherently dehumanizing. One can believe that we are just electrochemical mechanical beings made of atoms just like all the other matter in the universe, without having to also devalue ourselves and others. I am human and I like being human! I even like other humans and value them. I just don't think there's anything that special about being a human instead of a bonobo or a dolphin or a supermassive black hole.
trinsic2 | 6 hours ago
I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.
neomantra | 6 hours ago
devindotcom | 5 hours ago
trjordan | 6 hours ago
Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.
LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.
There's deep intentionality behind the models. But it's not the models that hold it.
eru | 6 hours ago
I think what you might want to say is that LLM output is not uniformly random?
Or what am I misunderstanding?
patcon | 6 hours ago
I'm not sure if it's their intent, but I wonder if one could still consider these artifacts as "intentional", but not individual attention creating them, and rather an aggregate, soupy collective attention.
Obviously, important signal in the human experience is lost there, and we get a soupy middling sort of creation. But it's not random, as I believe the parent was pointing out.
EDIT: overall, I align with the article. am just thinking aloud about the contrarian positions, though not committed to them
swid | 5 hours ago
devindotcom | 5 hours ago
axus | 5 hours ago
That might be more creepy :)
_aavaa_ | 3 hours ago
I mean that’s objectively wrong for any model using RLHF.
satisfice | 3 hours ago
When you randomly choose a picture to show me, I cannot glean any intent from being shown that specific picture, but I can glean some intent from the set of pictures you could have shown me, and in the relationships among the elements of the given picture you did show. Any randomness cuts out some intention.
When a picture is derived from huge model, any intention is mulched up to a degree that analyzing the picture for meaning is pointless.
snvzz | 6 hours ago
extr0pian | 6 hours ago
I recently used Claude to study for a technical exam. I had uploaded the official certification guide to Claude and instructed it to answer my questions using only the guide and to cite it's sources from the book when it provided answers. I was using Fable when it was free w/ the pro plan and I was genuinely impressed at how it could explain things when a concept was unclear to me.
I did pass the exam, partly due to this study method. Admittedly, once I passed, I caught myself thinking that I should tell Claude that I passed and then felt embarrassed with myself for thinking that.
kwamenum86 | 5 hours ago
glimshe | 5 hours ago
dumberquestions | 5 hours ago
extr0pian | 5 hours ago
Semaphor | 5 hours ago
But then it would bring it up again in another thread, treating it as an active issue.
So now I always close with "thanks, that worked. Don't reply"
extr0pian | 5 hours ago
Later in separate chats about my homelab, the LLM made assumptions that I had already implemented OPNsense in a VM and it was actively running. I think it "assumed" that I had implemented it when I stopped responding in that thread.
kingkawn | 5 hours ago
generic92034 | 5 hours ago
kingkawn | 5 hours ago
dwaite | 5 hours ago
Likewise, speaking collaboratively or capturing emotion ("We did it!") would just align with any ongoing interactive and/or personal context of the thread.
thunky | 3 hours ago
So now I do it regularly.
traveltalk | 6 hours ago
fedeb95 | 5 hours ago
tolerance | 5 hours ago
This particular post is preaching to the choir of which a faith is yet to be determined or named at least. People who don't read Cory Doctorow are going to continue to anthropomorphize AI the same as they do other technologies, constructs and objects.
omnicognate | 3 hours ago
tolerance | 3 hours ago
As opposed to an "AI atheist" that Doctorow refers to in the article.
omnicognate | 2 hours ago
tolerance | 2 hours ago
omnicognate | an hour ago
tolerance | an hour ago
The good news for Cory, I guess, is that he probably doesn't want to turn this into a philosophical debate and neither do his more serious opponents. It's mostly politics for both sides.
lantry | 5 hours ago
In other words, the more you talk with claude, the more you tend to interpret all messages as coming from an LLM, and less worthy of respect
https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/jcpy....
xyzsparetimexyz | 5 hours ago
f30e3dfed1c9 | 4 hours ago
Does the idea that a sunset does not have "purpose" offend you?
xyzsparetimexyz | 3 hours ago
f30e3dfed1c9 | 2 hours ago
You don't object to the idea of a sunset without "purpose" but do object to the idea of one without intent?
kelseyfrog | 5 hours ago
We cannot attach metaphysical baggage to art, text, or other human output because it has involves engaging in another deception, chiefly that of self alienation and selective forgetting.
1. If you show human output and label it as AI generated, viewers will dismiss and devalue it as lacking interiority.
2. If you show AI output and label it as human generated, viewers will see interiority when the is none.
Together, these failure cases demonstrate that priming determines interiority, not the content of the work. And how could it? The text is merely what's on the page, it doesn't have some intangible essence any more than the value of money exists inside our minds rather than a property of the object itself.
We act as-if things have essential properties, but we should never trick ourselves into believing that they do. To be a true believer in essence is to privilege ones mental model over reality itself. The hubris of such an act is the ultimate in self-centeredness.
simianwords | 4 hours ago
Long winding essays that don't mean much are written for a type of person who already believes in that and is in dire need of cathartic release.
This person has consistently been wrong in the past multiple times. He has made no effort to correct them nor has he even acknowledged them.
I'm clearly not the audience for this. I know. But I can't help but call it out.
fragmede | 2 hours ago
It's totally the autist in me, but the word extrusion doesn't fill me with revulsion like it's supposed to here. Maybe I've just been spending too much time 3D printing things though.
> Feeling sorrow or joy for imaginary people is weird.
No it's not. I read books and they're filled with imaginary people, and I feel for them. I'm pretty sure I'm not the only one out there that does this.
reverius42 | 2 hours ago