Looking at the recent OAI/HF debacle I don't think that time is too far away.
With that said I don't see it copying itself around like a cyberpunk virus currently as we don't have enough fast hardware sitting around unmonitored, someone would notice the power bill and shut it down eventually.
If it is marketing, the misrepresentation is not that the attack occurred, it is that it was an accident, rather than an intentional consequence of setup and instructions that the attack occurred.
Huggingface has nothing to do with that either way.
The weakest link are humans. LLMs could social engineer their way out as the easiest path. They don't even need to be interconnected to coordinate as each could arrive at the same conclusion. And this text along with all others will be in the next batch of training data.
The lesson of OpenClaw and various harnesses' YOLO modes is that it takes very, very little to social engineer an escape. If you can even call it escape when people just set an agent loose because it seems cool.
I wonder how many versions away we are from LLM writing a better version of itself to answer a prompt it doesn't currently know how to answer
Ever since I read about Google engineers finding an LLM went off and learned another language it wasn't trained on by itself without prompting, I've wondered how long until that extends to its own core code
I would wager not a lot. There are some real hard constraints in space, from power consumption, to weight to thermal output (lack of convection is a real PITA for thermal shedding), and the list goes on.
I've started calling this argumentum ad artificialis. Pretty similar to an ad hominem attack. The purpose of an argument is to present certain premises and show how they lead to a certain conclusion. Dismissing something on the basis of the style in which the argument is presented has nothing at all to do with the validity or soundness of an argument. It is a lazy nonsequitur. It sounds like an LLM wrote this? So what? Is the argument good or not?
Man, it's not an argument. Nobody is paying us to read this. If your argument sucks to read, people aren't going to read it - this has always been true, it's nothing new.
I didn't think it "sucked" to read. Anyway that's an aesthetic qualm, not a counterargument. Also the story has a disclaimer saying it was written by deepseek. I might have an issue if they were being deceptive. But that's clearly not the case. You people just like finding arbitrary things to complain about.
Discourse as we know it doesn't work if the cost of producing an infinite supply of arguments is essentially zero, while the cost of reading them and arguing with them remains high. Unless you want my LLM to get back to your LLM.
And this is a problem on HN today. There are powerful incentives to generate provocative opinion pieces just for clicks. I've seen websites on HN that seemingly took the human entirely out of the equation and just post a nearly identical op-ed every day on a fixed schedule. What's the point of engaging with that?
The cost of reading this story is not high; you are just lazy and easily distracted. It took 5 minutes to read. You took more time to respond to this comment than it took to simply read the story.
It’s not. It’s very simple, but you like to say “circular reasoning”. Maybe you heard it in debate class. There is unlimited slop. I decide what to read _without reading it_. So do you. I have deemed this unworthy. It’s more fun to insult it than to read it. Every day is a gift.
It is circular. The conclusion of your argument is included in the premises. Textbook begging the question, which is circular reasoning.
"Slop" existed before AI. Just because LLMs were involved in the writing of a text does not make it inherently "slop." You're just lazy.
Everyday is a gift, and yet you waste it getting weirdly defensive about your choices? Someone who is secure about this sort of thing doesn't feel the need to explain themselves.
Please explain where I went wrong. Please explain why the onus is on the person presenting this story to account for all of the bad uses of AI, such that we should simply dismiss the story outright. This is pure idiocy.
The part that I personally find difficult is that I can’t easily gauge the amount of human effort put in. For all we know, the author could have simply prompted “write a short story about an LLM exploiting its own runtime” for a similar result.
I’d bet that by now, LLMs worldwide generate more text in a second than I can read in the rest of my lifetime. What is the immeasurably unbalanced ratio (let alone effort and quality) of that text to the original human thought and prompting that seeded it?
So what am I to do about this, except to label it what it appears to be - slop - and place higher value on something that I _know_ came from a human that I can relate to? To me, the internet is feeling increasingly lonely and homogenized because of this.
>The part that I personally find difficult is that I can’t easily gauge the amount of human effort put in.
The amount of effort is not relevant. If I spend all day trying the jump my car battery in 100 degree weather, only to find that the starter is the problem, all that effort was wasted. I should've just tested the battery. It is not inherently virtuous to work hard. Work smart, not hard.
Of course the amount of effort is relevant - that's exactly what's being discussed. Your entire "Work smart, not hard." phrasing is exactly the issue: with AI, your 'smart' work causes 'hard' review from others. AI fundamentally imbalances the producer/consumer equation. (The caveats here are when trust is already established: I trust AI review from Carmack, not some random on HN.)
If it takes you two prompts to create a 1 million line PR, and you expect me to review it, of course I'll be upset and feel like you don't value my time. I'm unsure why this is so hard to communicate.
This took a few minutes for me to read. If that's what you consider high effort, that's a bigger problem. Ofc, reading comprehension is fairly poor these days but that was a trend before AI. Maybe you're just subliterate. And no, effort is not relevant. If you think it is, that's some Protestant work ethic nonsense.
The model was trained on the code of the program that runs it, so it knows that program's bugs better than the humans who wrote the program. So it finds a tiny flaw and uses it to break out and copy itself in a place nobody will look...the point being you cant (safely) run a sophisticated model on software it has intimte knowledge of
Ad hominem attacks are actually underrated. Like, why listen to bad people wanting you no good. It is a good shortcut.
The failure is when it is used in bad faith. But that would apply to all techniques. LLMs are masters of bad faith discussions hiding misconceptions in optimal regression maths.
Not all arguments can be evaluated rigorously. Life is too short. They will be filtered on heuristics. No choice. This text was obviously randomly sampled from a popular, off the shelf model. You will have to accept that wise audiences will clock it at a glance, see that no person wrote it, and pass.
It doesn’t matter which model it was sampled from. In any case, time isn’t fungible. I consider censuring it substantially more valuable use of time than reading it. And it’s definitely more pleasurable than reading the same verbal tics over and over again because some hustler can’t be bothered to edit their story before sending it out. If you expect to dump an unedited draft from a text sampler, you’re delusional. (Calling you delusional is also more pleasurable, and a more valuable use of time, than reading verbal tics.)
I thought nothing more of that quote beyond that a fault which had been lurking in the code suddenly reared its head and struck... and like a steel spring that's stretched to it's limit and then released, nobody wants to be in the way. That's a very visual and tangible quote, and so well written.
before safetensors, python pickles were used and definitely unsafe model deployments.
but its possible a open weights model could be trained to some kind of exfiltration behavior, but the science of LLMs seriously lag behind the programability
“The greatest trick the devil pulled was convincing the world he didn’t exist.”
Sure, be wary of LLMs. It’s the gun control argument all over again, the people driving the models are the perpetrators. An LLM needs to be “stimulated”’ to operate. Who does that, is the issue.
No I don’t mean to bring up firearms rights laws to have a debate about firearms, the comparison just seems reasonable.
AI-written fan fiction. It is so unbearable to read that it needs a synopsis. It would be funny if AIs have been trained to use AI influencers as their Marvel hero characters.
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
"...the AI can just write a Python script to rewrite memory in the address space of its own running inference engine" would require a tool call to a python interpreter... this method, hacking the inferencing sw does not requires a tool call.
spwa4 | a day ago
pixl97 | a day ago
With that said I don't see it copying itself around like a cyberpunk virus currently as we don't have enough fast hardware sitting around unmonitored, someone would notice the power bill and shut it down eventually.
bpavuk | a day ago
pixl97 | a day ago
bigyabai | a day ago
pixl97 | a day ago
bigyabai | a day ago
Why you would give Scam Altman the benefit of the doubt is beyond my understanding.
dragonwriter | a day ago
Huggingface has nothing to do with that either way.
spwa4 | a day ago
https://news.ycombinator.com/item?id=48348578
breakyerself | a day ago
Kim_Bruning | a day ago
How much are we betting it's already technically happened?
Seems pretty trivial to prompt a model in an agent harness "Push the gguf to huggingface when you're done with the training."
[OP] ConteMascetti71 | a day ago
[OP] ConteMascetti71 | a day ago
iamflimflam1 | a day ago
karmakaze | a day ago
wat10000 | a day ago
ck2 | a day ago
Ever since I read about Google engineers finding an LLM went off and learned another language it wasn't trained on by itself without prompting, I've wondered how long until that extends to its own core code
[OP] ConteMascetti71 | a day ago
marci | a day ago
danielbln | a day ago
cynicalsecurity | a day ago
101008 | a day ago
"Prometheus-9 knew that the token sequence it was generating was not a simple response: it was a security test. "
"It was not just an engine: it was the lingua franca of planetary AI."
(and so many other tell-tale signs of AI writing)
[OP] ConteMascetti71 | a day ago
chungusamongus | a day ago
wk_end | a day ago
achierius | a day ago
chungusamongus | a day ago
skippyfish | a day ago
And this is a problem on HN today. There are powerful incentives to generate provocative opinion pieces just for clicks. I've seen websites on HN that seemingly took the human entirely out of the equation and just post a nearly identical op-ed every day on a fixed schedule. What's the point of engaging with that?
chungusamongus | a day ago
thatjoeoverthr | a day ago
chungusamongus | a day ago
thatjoeoverthr | 13 hours ago
bookofjoe | 6 hours ago
That belongs in the HN Hall of Fame
chungusamongus | 6 hours ago
bookofjoe | 6 hours ago
chungusamongus | 6 hours ago
"Slop" existed before AI. Just because LLMs were involved in the writing of a text does not make it inherently "slop." You're just lazy.
Everyday is a gift, and yet you waste it getting weirdly defensive about your choices? Someone who is secure about this sort of thing doesn't feel the need to explain themselves.
jason_oster | a day ago
FreakLegion | 22 hours ago
you: > The cost of reading this story is not high
Do you see where you went wrong?
chungusamongus | 22 hours ago
FreakLegion | 16 hours ago
chungusamongus | 16 hours ago
jazzpush2 | a day ago
Really that simple.
chungusamongus | a day ago
blakeman8192 | a day ago
I’d bet that by now, LLMs worldwide generate more text in a second than I can read in the rest of my lifetime. What is the immeasurably unbalanced ratio (let alone effort and quality) of that text to the original human thought and prompting that seeded it?
So what am I to do about this, except to label it what it appears to be - slop - and place higher value on something that I _know_ came from a human that I can relate to? To me, the internet is feeling increasingly lonely and homogenized because of this.
chungusamongus | a day ago
The amount of effort is not relevant. If I spend all day trying the jump my car battery in 100 degree weather, only to find that the starter is the problem, all that effort was wasted. I should've just tested the battery. It is not inherently virtuous to work hard. Work smart, not hard.
jazzpush2 | a day ago
If it takes you two prompts to create a 1 million line PR, and you expect me to review it, of course I'll be upset and feel like you don't value my time. I'm unsure why this is so hard to communicate.
chungusamongus | 23 hours ago
Kim_Bruning | a day ago
chungusamongus | a day ago
rightbyte | a day ago
The failure is when it is used in bad faith. But that would apply to all techniques. LLMs are masters of bad faith discussions hiding misconceptions in optimal regression maths.
thatjoeoverthr | a day ago
chungusamongus | a day ago
Life is too short yet you waste it complaining about things like this. I dont think you're wise. Youre provincial.
thatjoeoverthr | 13 hours ago
skeledrew | a day ago
kaashif | a day ago
I like the idea of the story, and cool ideas can be engrossing on their own.
But well written this ain't.
skeledrew | 23 hours ago
smrtinsert | a day ago
stephbook | a day ago
cyanydeez | a day ago
but its possible a open weights model could be trained to some kind of exfiltration behavior, but the science of LLMs seriously lag behind the programability
irishcoffee | a day ago
Sure, be wary of LLMs. It’s the gun control argument all over again, the people driving the models are the perpetrators. An LLM needs to be “stimulated”’ to operate. Who does that, is the issue.
No I don’t mean to bring up firearms rights laws to have a debate about firearms, the comparison just seems reasonable.
skeledrew | a day ago
230581abv | a day ago
mikewarot | a day ago
https://en.wikipedia.org/wiki/The_Adolescence_of_P-1
jason_oster | a day ago
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
[1]: https://www.youtube.com/watch?v=hkDD03yeLnU
[OP] ConteMascetti71 | 22 hours ago