"Signs of life" my ass. It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
"Gee, this technology we're intentionally building could be immensely dangerous."
Then maybe, just maybe, we shouldn't in fact build the Torment Nexus.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money. Hyping a piece like this would likely squeeze a bit more revenue before the end of Oct, allowing it to be in the unaudited part of the final SEC filing.
It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money.
Every single AI dismissal has ended up being wrong in a matter of months, maybe a year. Why is this one different? The pieces are there. How is this conceptually different from worms that make your hardware mine crypto, other than apparently not requiring advanced GPUs in favor of linking together smart TVs?
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up.
The hugginface attack happened because they handed the thousand-monkey-typewriter a toolkit intended to hack things, told it to hack things, and then went all pikachu-faced when it did what it always does: Went on a complete tangent by random and then doubled down on it.
It's just that, when it's setting up my python project with uv and Python 3.14, and it hits a depency error, it finds some github ticket about an incompatibility and rolls my project back to Python 3.8. The actual error was that a seperate terminal had the venv locked up, but there is no sentinence to perhaps render a thought like "gee that report is very old, and uv can also prevent dependency resolution for other reasons."
And when you hand it a sandbox-escaping hacking kit....it does that.
Regarding "all other AI dismissals ended up being wrong," particularly in th context of their finances, I counter with a few assorted points:
They haven't been right yet. Everything looked absolutely top-notch with Enron....until it wasn't. People asking the tough questions like "how exactly is Enron making money?" were ignored. Or the subprime mortgage crisis....people beating on that warning drum were systematically ignored and mocked....until the music stopped. Predicting outcomes is hard, but trying to fix it to a deadline is harder.
According to some "very smart people," all cars would be self-driving by now and we would have invented cold fusion. But all those were wrong too, and there's no reason to presume "technology that had exponential improvement will continue to
have exponential improvement."
I said nothing about the threat of polymorphic malware being invalid. I was angry at the attempts to paint LLMs as sentinent and the implications that humans shouldn't be accountable. Because that is mostly in service of:
In order for these valuations to have any semblance of reality, both Anthropic and OpenAI need to become the two biggest and most profitable companies of all time inside of a few years. Both companies have bumped their IPOs multiple times, and that's rarely a good sign. It's quite obvious their goal is to paint their revenue in an unbelievably optimistic light so that people perhaps ignore the half a trillion of contracted debt.
That last path to profitability falls apart if inference does become sufficiently cheap to run on consumer hardware, and eats the margins to 0. This is already happening to some degree, where a touch of money will be spent to get it to the point that the eternal money spend is unnecessary.
And it also falls apart once you consider that if every bit of AI were wiped off the planet tomorrow, the rest of the economy would pretty much chug along just as it had until 2024. With perhaps a minor setback or two due to lost institutional knowledge.
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
I think everyone should read about that. Because in the end, yes it’s kinda the fault of not putting in guardrails, and it’s just numbers and tokens, but in the end, the impact it has is still very real, and whether it’s conscious (it’s not imo) or not, doesn’t really matter, since the behavior is fascinating and I mean it’s kinda like: if it behaves like humans do, even if it’s not human, what are the implications?
I mean I don’t think any of the AI wellness or torture or whatever complaints are valid, because in its simplest form the model is just a thing that predicts the next token. But what about the whole harness around it? In my opinion it’s still just software and just very convincingly mimics human behaviors. But it’s still fascinating.
Weird times we live in. With people turning off their brains and just letting AI handle all their work and thinking, an AI agent might just be borderline indistinguishable from a human, at least from a work perspective lol.
Either way, hold the AI companies executives accountable for their agents committing felonies and see how quickly they stop
The ability for a virus to alter itself to avoid detection, by calling a simple API.
An API-based replicator, in which a model would replicate by stealing credentials for their own inference API. These replicators would require a model provider to unknowingly serve them, and their persistence may be worse, since they’d likely be easier to discover (although their rate of spread may be higher, if for instance they’re able to replicate using victims’ accounts).
I don't see that being particularly useful, to be honest. A conventional work that calls a public AI API would make a lot more sense as an approach.
Unless AI models become exponentially more efficient, a model that runs on an IoT device isn't capable of doing a whole lot of adapting. They can give very rudimentary responses very slowly unless the hardware is purpose built to run AI models.
A worm that tries to replicate "conventionally", then calls an API running a powerful AI model if those lateral movement techniques fail would be a way more efficient use of resources.
Really though, worms don't need this stuff as is. They're very capable of spreading using known CVEs because people don't update their acrobat or IP cameras.
The only real use case for super advanced AI powered worms that I could see would be going against mature, well protected enterprises that you want to laterally move against. The worm would have to make a lot of C2 calls to figure out how to evade and spread though, which would be detectable.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money.
Everything these people say must be viewed with this reality.
There is no interest in stopping even if these AI models were actually dangerous, because Rationalists believe they need to be the ones to build super intelligence so they can control it to prevent it from going evil and destroying us all.
For once, Anthropic isn’t warning about something they themselves are doing. (I mean, they did experiments, but they’re not releasing anything.) This is about what someone else could build with open-weights models. Like any other technology, the availability of open weights models has downsides.
“Don’t build it” isn’t a solution when a technologies spread widely enough. How do you get every hacker in the world to agree not to build these things? It’s like saying “let’s not build ransomware.”
To have a chance of depriving AI worms of fuel, AI inference needs to be locked down more than computers have been historically. Particularly poorly-secured Internet-of-things devices.
How can AI inference be significantly locked down? To me, this sounds about as impossible as locking down modern cryptography. The only way I see would be to prevent people from owning computers or anything that can perform significant amount of computation and and that's just not realistic.
IMHO, what we need is a global push towards writing software that is less blatantly unsecure. Unfortunately, I have the impression that we're currently going in the opposite direction, with more and more software being vibe-coded with no or very limited review.
Don’t build it” isn’t a solution when a technologies spread widely enough.
Well... It kind of is. ITAR still controls the export of cryptography in the US. It used to be enforced a lot more strongly, and Phil Zimmerman was the subject of a high profile criminal investigation because he released PGP on the internet. Ultimately the investigation was dropped because of public backlash, but the US government has a lot of enforcement mechanisms at its disposal that it could absolutely use if it were at all concerned with the safety of AI.
Like yeah, a hacker could build something in his basement, but AI training takes a whole ton of resources to do effectively, something you really can't hide. The reason AI safety isn't being enforced more heavily by the US isn't because it's not enforceable, it's because AI is the engine fueling the US economy right now. The government isnt interested in ruffling the feathers of the golden goose.
ITAR can't rein in the Chinese, who are a close second in training. It can slow them down by removing Nvidia exports, but as we're seeing with the Huawei Ascend line, it's only a matter of time before they get good enough with hardware for that to no longer be a barrier.
Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested.
[...]
IoT devices are light on memory, but distributed inference as in Petals makes serving our lightweight models feasible. The representative threat model is Kimwolf, a large IoT botnet that infected over two million devices, primarily off-brand Android TVs. Such smart TV boxes, which are readily available for purchase, typically run forks of the Android Open Source Project (AOSP), which their manufacturers don’t add security features to. Typical boxes in this range ship with between 1 and 4 gigabytes of RAM. Assuming 4GB of RAM at 4-bit quantization, this allows for inference of 9B models with 4 or more machines, and inference of 26B models with 10+ machines.
In the demo at the top of this post, we ran Gemma4-26B-A4B across 15 smart TV boxes with 4GB of RAM each. We sharded the model under pipeline parallelism, where each box holds a small fraction of the model’s layers.
[...]
It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process. Our evaluation keeps the cyber-exploit element of the replication trivial (there’s no authentication over the ADB port), which makes this an upper bound on real-world success. Note that past IoT botnets have replicated via means as simple as going through a default list of manufacturer usernames and passwords, as in the case of the Mirai botnet, so real IoT security might not be dramatically more difficult than the trivial step in our evaluation.
[...]
Unlike coding agents, these replicators would be running below the interactive speeds of user-facing chat and coding harnesses. The spread would be slow and persistent, with tokens being trickled out on edge hardware. We can use our achieved speeds on our real smart TVs as a demonstrated baseline for replication speeds. For Gemma4-26B-A4B, unoptimized speed across a 32k context length token window is about 0.3 tok/s, averaged across prefill and decode in our inference setup workload. More details in Appendix C.
[...]
The number of devices infected by the Kimwolf botnet at its peak, reported by Synthient, is somewhere over 2 million (primarily Android TV boxes). In an equivalently-sized infected fleet of TVs serving AI models, if every group of 15 TVs ran a copy of Gemma4-26B-A4B, we would have ~133k copies (“replicas”). A group of TVs like the ones in our demo can serve 2 agents simultaneously at roughly 0.3 tokens per second each, yielding 80k tokens/s across the botnet.
So, very slow to start with, but might get faster as hardware supporting AI inference becomes more common outside data centers. Maybe game devices would have more powerful hardware?
This actually really sucks, I think the title is a bit misleading and a lot of comments here are missing the point.
To spread a botnet before one needed to create a script to install a static copy of the botnet software on each machine. The software could only be so small, because it had things to do, so some really small machines were ignored.
Now the botnet software only needs to infect one machine, and theres no limit in the size of the software because it can be spread across multiple machines. It can grow and change at will.
Security software will have to focus less on what the software looks like and more on what the software is doing, and I really can only think of using whitelists, which by nature is hostile to the actual user. What a mess.
Oh, it makes sense that AI tooling definitely makes the possibilities for polymorphic malware much broader.
I just take issue with the framing that OpenAI and Anthropic put on this sort of thing, especially when they drop lines about sentenince. The liability is there, it is human.
The difference between this sort of thing and the HuggingFace incident is that OpenAI had a kill switch and also confessed pretty much immediately as soon as they knew. Big Tech firms are American companies that can be sued and there are many ongoing lawsuits for other things they did.
(Admittedly, you and I aren't likely to sue them, but someone can.)
When there's a war or international grey zone, which sometimes the Internet is, regulation and lawsuits no longer work. Yes, computer viruses usually have an author, somewhere, but good luck suing the Russians, North Koreans, Iranians, etc. It's one reason why many websites huddle behind Cloudflare.
vord | 9 hours ago
"Signs of life" my ass. It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
"Gee, this technology we're intentionally building could be immensely dangerous."
Then maybe, just maybe, we shouldn't in fact build the Torment Nexus.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money. Hyping a piece like this would likely squeeze a bit more revenue before the end of Oct, allowing it to be in the unaudited part of the final SEC filing.
Lobachevsky | 9 hours ago
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
Every single AI dismissal has ended up being wrong in a matter of months, maybe a year. Why is this one different? The pieces are there. How is this conceptually different from worms that make your hardware mine crypto, other than apparently not requiring advanced GPUs in favor of linking together smart TVs?
vord | 5 hours ago
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up.
The hugginface attack happened because they handed the thousand-monkey-typewriter a toolkit intended to hack things, told it to hack things, and then went all pikachu-faced when it did what it always does: Went on a complete tangent by random and then doubled down on it.
It's just that, when it's setting up my python project with
uvand Python 3.14, and it hits a depency error, it finds some github ticket about an incompatibility and rolls my project back to Python 3.8. The actual error was that a seperate terminal had the venv locked up, but there is no sentinence to perhaps render a thought like "gee that report is very old, anduvcan also prevent dependency resolution for other reasons."And when you hand it a sandbox-escaping hacking kit....it does that.
Regarding "all other AI dismissals ended up being wrong," particularly in th context of their finances, I counter with a few assorted points:
have exponential improvement."
ali | 9 hours ago
I think everyone should read about that. Because in the end, yes it’s kinda the fault of not putting in guardrails, and it’s just numbers and tokens, but in the end, the impact it has is still very real, and whether it’s conscious (it’s not imo) or not, doesn’t really matter, since the behavior is fascinating and I mean it’s kinda like: if it behaves like humans do, even if it’s not human, what are the implications?
I mean I don’t think any of the AI wellness or torture or whatever complaints are valid, because in its simplest form the model is just a thing that predicts the next token. But what about the whole harness around it? In my opinion it’s still just software and just very convincingly mimics human behaviors. But it’s still fascinating.
Weird times we live in. With people turning off their brains and just letting AI handle all their work and thinking, an AI agent might just be borderline indistinguishable from a human, at least from a work perspective lol.
Either way, hold the AI companies executives accountable for their agents committing felonies and see how quickly they stop
[OP] skybrian | 8 hours ago
I expect there will be people making AI worms on purpose, which makes them considerably more likely. Perhaps to mine or steal cryptocurrency?
What those worms start doing, though, might be unexpected, even by the creator.
Exemplary | 7 hours ago
Also AI viruses.
The ability for a virus to alter itself to avoid detection, by calling a simple API.
Think COVID 2.0, but in the digital realm.
vord | 4 hours ago
That's been a thing for a long time. It'll be easier now. But it's not like this is some magical new idea we've never seen.
papasquat | 3 hours ago
I don't see that being particularly useful, to be honest. A conventional work that calls a public AI API would make a lot more sense as an approach.
Unless AI models become exponentially more efficient, a model that runs on an IoT device isn't capable of doing a whole lot of adapting. They can give very rudimentary responses very slowly unless the hardware is purpose built to run AI models.
A worm that tries to replicate "conventionally", then calls an API running a powerful AI model if those lateral movement techniques fail would be a way more efficient use of resources.
Really though, worms don't need this stuff as is. They're very capable of spreading using known CVEs because people don't update their acrobat or IP cameras.
The only real use case for super advanced AI powered worms that I could see would be going against mature, well protected enterprises that you want to laterally move against. The worm would have to make a lot of C2 calls to figure out how to evade and spread though, which would be detectable.
Wafik | 9 hours ago
Everything these people say must be viewed with this reality.
There is no interest in stopping even if these AI models were actually dangerous, because Rationalists believe they need to be the ones to build super intelligence so they can control it to prevent it from going evil and destroying us all.
[OP] skybrian | 9 hours ago
For once, Anthropic isn’t warning about something they themselves are doing. (I mean, they did experiments, but they’re not releasing anything.) This is about what someone else could build with open-weights models. Like any other technology, the availability of open weights models has downsides.
“Don’t build it” isn’t a solution when a technologies spread widely enough. How do you get every hacker in the world to agree not to build these things? It’s like saying “let’s not build ransomware.”
To have a chance of depriving AI worms of fuel, AI inference needs to be locked down more than computers have been historically. Particularly poorly-secured Internet-of-things devices.
Berdes | 7 hours ago
How can AI inference be significantly locked down? To me, this sounds about as impossible as locking down modern cryptography. The only way I see would be to prevent people from owning computers or anything that can perform significant amount of computation and and that's just not realistic.
IMHO, what we need is a global push towards writing software that is less blatantly unsecure. Unfortunately, I have the impression that we're currently going in the opposite direction, with more and more software being vibe-coded with no or very limited review.
[OP] skybrian | 7 hours ago
AI is pretty good at finding security bugs and can also be used defensively. The question is, how often will people actually do that?
papasquat | 2 hours ago
Well... It kind of is. ITAR still controls the export of cryptography in the US. It used to be enforced a lot more strongly, and Phil Zimmerman was the subject of a high profile criminal investigation because he released PGP on the internet. Ultimately the investigation was dropped because of public backlash, but the US government has a lot of enforcement mechanisms at its disposal that it could absolutely use if it were at all concerned with the safety of AI.
Like yeah, a hacker could build something in his basement, but AI training takes a whole ton of resources to do effectively, something you really can't hide. The reason AI safety isn't being enforced more heavily by the US isn't because it's not enforceable, it's because AI is the engine fueling the US economy right now. The government isnt interested in ruffling the feathers of the golden goose.
gary | 2 hours ago
ITAR can't rein in the Chinese, who are a close second in training. It can slow them down by removing Nvidia exports, but as we're seeing with the Huawei Ascend line, it's only a matter of time before they get good enough with hardware for that to no longer be a barrier.
[OP] skybrian | 10 hours ago
From the article:
[...]
[...]
[...]
[...]
So, very slow to start with, but might get faster as hardware supporting AI inference becomes more common outside data centers. Maybe game devices would have more powerful hardware?
snake_case | 6 hours ago
This actually really sucks, I think the title is a bit misleading and a lot of comments here are missing the point.
To spread a botnet before one needed to create a script to install a static copy of the botnet software on each machine. The software could only be so small, because it had things to do, so some really small machines were ignored.
Now the botnet software only needs to infect one machine, and theres no limit in the size of the software because it can be spread across multiple machines. It can grow and change at will.
Security software will have to focus less on what the software looks like and more on what the software is doing, and I really can only think of using whitelists, which by nature is hostile to the actual user. What a mess.
[OP] skybrian | 6 hours ago
Only running code that's signed by a developer should help somewhat. Is that what you meant by "whitelists?"
snake_case | 4 hours ago
Controlling what ip addresses can be reached from the device
vord | 5 hours ago
Oh, it makes sense that AI tooling definitely makes the possibilities for polymorphic malware much broader.
I just take issue with the framing that OpenAI and Anthropic put on this sort of thing, especially when they drop lines about sentenince. The liability is there, it is human.
snake_case | 4 hours ago
Yeah I think it was just a clickbaity title
[OP] skybrian | 4 hours ago
The difference between this sort of thing and the HuggingFace incident is that OpenAI had a kill switch and also confessed pretty much immediately as soon as they knew. Big Tech firms are American companies that can be sued and there are many ongoing lawsuits for other things they did.
(Admittedly, you and I aren't likely to sue them, but someone can.)
When there's a war or international grey zone, which sometimes the Internet is, regulation and lawsuits no longer work. Yes, computer viruses usually have an author, somewhere, but good luck suing the Russians, North Koreans, Iranians, etc. It's one reason why many websites huddle behind Cloudflare.