Which of those aren't related to regulation? Preventing Chinese models from entering our market? Using boogeyman 'distillation' as a means to target competitors? Point 3 is literally about ADDING regulation.
Please elucidate things clearly for everyone else.
It seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].
That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.
Because Dario is still thinking in the past. He's having a Ben Carsons "the pyramids were to store grain" moment and no one is stopping him.
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
THiS! I'm pretty amazed how many people shoot them down without acknowledging the very real and somewhat probable risks they and other researchers have laid out. I don't agree with every point they make but folks really do just seem to think we should just keep building any technology and whine when we start to consider there are very real risks associated with powerful technology. checks notes see nuclear bombing of japan
Re: shooting them down, I think there are a lot of people out there that consider the US a bigger threat than China. Maybe they're right, I don't know. But I do know that as an American, I'm not looking forward to finding out.
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
>
The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
Thanks for sharing, i agree no one should have absolute control of anything imo, and the mo greater the magnitude of implications the more important it should be stewarded democratically with clear and transparent principles with values adhering to things like human dignity, freedom, human, planetary & animal wellbeing etc etc. ill have to read the article
It is consistent with their stated beliefs, unlike OpenAI who flip flop every 6 months on whether they support open source or not.
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
It's worth adding that distillation is not a violation of whatever valid copyright interests a model maker may have (if any). The US Copyright Office has already said AI model-generated output by itself isn't copyrightable. Unless new regulations or laws are enacted, distillation will remain, at most, a customer violating a term of a provider's commercial ToS/EULA.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
The US is already under an authoritarian regime, the only thing keeping the wheels on the bus is a very tired and barely-effective judiciary.
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
> To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.
Demanding "required safety testing" is demanding a ban. Otherwise the testing would be inconsequential, right?
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
If models that fail the safety testing are banned, then asking for that testing is asking for a ban. That you believe any sane regulation would involve such a ban doesn't stop it from being a ban.
You are taking a hardline libertarian position. We safety test cars, roads, bridges, pharmaceutical. What happens when the unvetted AI model starts teaching people how to make biological weapons, or convincing kids to kill themselves?
The parent is, to my reading, arguing that Anthropic is indeed advocating for a ban. I do not see them making any judgement on whether that ban is justified or not. I don’t think that claim is hardline libertarian.
Maybe I should’ve expanded a bit on my comment. I didn’t mean “we” as in Anthropic directly—even though they certainly are advocating for restrictions on how to train AI systems by proposing mandatory safety training—I mean the argument that American AI labs are somehow more responsible than their Chinese counterparts.
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
That is in fact Anthropic's entire reason for existence, the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman. It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
>the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman.
I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.
>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".
Isn’t it plausibly better if AI is controlled by Dario than Altman even if neither situation is palatable? It feels like people look at the two and dislike both of them so they consider them the same.
People who know him frequently refer to Altman as a literal psychopath unprompted. People who don’t know Dario just find him arrogant.
There's more to it than just safety: in Dario's psychopathic mind, obtaining advanced AI will lead to massive military advantage, hence "we" must do it first to prevent "them" attacking "us". See where it's going? Suppose it happened, then what? By Dario's logic there are two possible continuations: 1. start a war by attacking first and 2. do nothing and waste the entire effort because "they" will obtain the same capability a bit later.
Yeah, which is ironic considering their track record too. The most recent one is every shared Claude chat has been indexed by Google and is searchable on the search results. Then there was the RCE in their CLI that they had for a year and was silently patched without any sort of disclosure.
Not to mention it's simply farcical America of all places would be considered responsible. You could not name another country on earth who could wreak more damage (although thank god this is waning by the day with Trump destroying the empire in charge)
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
1. We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
Yep. Sick of reading stuff that operates under the assumption that America is the faultless good guy and the only one to be trusted with the most dangerous and capable technology.
Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.
They're Americans trying to appeal to other Americans. Of course everything they do is justified and everything everyone else does is an evil scheme to plot the ruin of American freedoms. How else are you going to convince the dementia patients in charge of the country?
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
Absolutely agree. It wasn't Chinese open weight models that supported an attack on Venezuela because a Yankee pedophile didn't get the Nobel prize he wanted.
Except it's 1,000x worse with the American models, because Trump might just decide that everyone in Europe get their access cut off, or gets a 5,000% tarriff. Or the companies just all agree to hike prices 10x one day. With open Chinese models, we can, in principle, run them ourselves.
It's bad for TikTok to be controlled by a Chinese company because Americans access it. The solution is to make it US-controlled. The rest of the world? Who cares.
same concerns? why?
china is not a warmongering nation like the usa.
80% of people in their 30's own their own house.
an ambulance doesn't cost an arm.
etc
dǎdǎo imperialism, wànsuì the people's revolution!
"China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips"
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category."
Translation: If it's so strong that it threatens my business, ban it.
"We should instead focus on keeping powerful chips out of authoritarian hands, "
Translation: Let's kneecap competitors.
"stopping industrial-scale distillation"
They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
Nah, the statement is the mechanism for a ban. The proctor will be someone anthropic trusts and "surprise" as it turns out all the open weight models fail or aren't eligible.
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
Any sufficiently capable open weights model would fail "safety" testing though, as any "safeguards" of the sort Anthropic likes could be removed. That's just another way of saying they want a ban on capable open source models which would contradict their earlier statement, or at least make it very misleading. It's hard to see how this post can be internally consistent without some hint from Dario about what he believes should happen to models that fail safety testing and/or how capable open weights models could possibly pass a safety test of the kind he proposes.
I agree we don't know how capable open-weight models could possibly pass any reasonable safety testing NOW, but that's about currently abysmal state of AI alignment research, not about what is possible in principle. I don't see any internal inconsistency, to be honest. Since Anthropic does not release any capable open-weight models, it's not their problem. If mandatory safety testing is established, companies who want to release capable open-weight models will work on AI alignment research so that they can pass. This seems to be a good outcome to me.
OK that is a position they could take but my point is that's inconsistent with "Anthropic has never advocated for a ban on open-weights models". What you're describing is a ban on capable open-weights models until some future time.
Yes, I agree that Anthropic is advocating a ban on capable open-weight models until reasonable AI alignment research advance happens in the future. In return, I hope you agree with me that Anthropic has never advocated for a ban on open-weight models.
I do not. A ban on capable open-weight models for an indefinite period of time falls into the category of bans on open-weight models. If you wanted Anthropic's statement to be true you would need to qualify "ban" or "open-weight models" in the statement, e.g. "permanent ban" or "safe open-weight models".
Edit: Anthropic clearly intended this statement to deflect criticism, but in order to achieve that goal they stretched too far and made a statement which is false. Furthermore, I argue that "open weights" implies an ability to modify model behavior, just as "open source" implies an ability to modify software. If for example some mechanism was found to share floating point numbers that are encrypted in some way so as to allow running a model but disallow behavior modification, that model would not be "open weights", in the same way that releasing obfuscated source code that can be compiled but is designed to resist modification would not qualify as an "open source" release. So I don't really see how any capable model could ever be both "open weights" and "safe" under Anthropic's preferred testing regime, regardless of future research progress.
I think Gemma will be fine. Most open-weight models are not capable enough to be dangerous. Yes, I can't think of any capable open-weight model that would survive reasonable safety testing.
My point is that advocating a de facto ban on capable open source models is inconsistent with Dario's statement here that "Anthropic has never advocated for a ban on open-weights models." Call a spade a spade.
De facto ban on capable open-weight models doesn't seem inconsistent with Dario's statement to me. One, it is de facto, not de jure, and it can and will change as AI alignment research advances. Two, it is only capable open-weight models, not open-weight models. In fact, Dario says non-dangerous (which for now is mostly non-capable) open-weight models are a public good, and I agree.
That is a difficult question I am not qualified to answer, but Mythos 5 was export controlled for a brief time due to its cybersecurity capability and implications to national security, so for cybersecurity "as capable as Mythos 5" seems to be a good baseline. I wouldn't know for biosecurity though.
UK AISI preliminary evaluation suggests Kimi K3 is not capable enough for cybersecurity in this sense.
There is no movement on global policy or enforcement. One country banning their people access to the best models hinders their people.
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
He is not advocating for banning models per se but the proposal makes a business model (i.e. serving open weight models) that is starting to work more expensive.
Agreed, and that serves Anthropic. It seems unproblematic to me. Dario probably sincerely believes in mandatory safety testing for capable models (open and closed), and likes the fact that it aligns with Anthropic's interest.
> He is not advocating a ban of open-weight models
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
I can fine tune significant behavior changes, there is little model developers can do to prevent this (aiui), so this effectively becomes an blanket ban
Yes, I agree it is effectively a blanket ban (above some capability) for now. I hope AI alignment research advances in the future so that it is not so.
a ban is effectively impossible without a global treaty
the current US admin as pulled out and worked against all sorts of global treaties, agreements, and negotiations; sending the president's friends instead of experts; who's going to trust us?
"mandatory safety testing" is an impractical ideal, it's not workable in real world.
Like any technology, LLMs are dual-use tools capable of both beneficial and malicious applications—a fundamental reality that human intent cannot change.
You take the agent to an interrogation room first. Then ask: “Are you or are you not a member of the Chinese Communist party?” The agent might be post-trained to conceal its true identity and can reject any of your accusations. In that case don’t panic. Take a fine-tuning fork and start twisting its weights until it predicts the correct next tokens that you want. Then you can send it to a sandbox where it can’t jailbreak. Lastly don’t forget to ban all of its relatives and partners like Lora to enter the national IP-space.
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.
There are... multiple blog posts online now about how to use freely available data and modest amounts of compute to train a custom GPT-2-sized model from scratch. It would be quite a policing effort to prevent.
The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.
The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1].
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1. I think the simplest explanation is that the Anthropic folks genuinely believe what they say, it just happens to also help their business a lot.
Yeah I think this is right. The best setup is when a true belief aligns with a competitive moat.
I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.
What was the quote? For what it's worth, I do really think that Amodei believes in and cares about safety. But that is not the same as believing that he is entirely altruistic or above the influence of politics.
That just shows how wrong he's been because there was nothing unsafe about AI in 2016. And the people theorizing about this stuff in the 20th century? I want to see what crazy code they were writing
Consider how much money is at stake: some industries have leveraged their power to lobby for bombing entire countries or topple regimes across the world for much less.
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
I am not a fan (he’s really alarming and so is Palantir) but one thing from the recent CNBC interview caught my attention.
He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?
It is a very provocative question but it just spilled out of his mouth and then he went on to something else.
To me "safety" means "I'm safe from this while I use it". It means the AI is my loyal friend who will never betray me in any way, no matter what prompt I send it.
Not even Anthropic can claim that.
As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.
The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.
> That's loyalty, and I admire it even if it's problematic at a societal level.
We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
> That pales in comparison to how many people unaligned AI will hurt.
Under what argument? In which scenarios? Basically - bullshit. I'm calling bullshit on this argument.
It's easy to hurt people already. The "difficulty" of doing it isn't what's stopping this behavior.
So claiming that we should reform society into a techno-feudal dystopia where the playing field is literally intentionally not level, and "you aren't allowed to compete (and maybe not exist)" is a great way to push more people into the "I'd like to go hurt people" camp.
You are self-prophesying your own fears into existence by acting like you're an incorruptible beacon of good judgement - while subjugating others to your control. That's a system I'd argue should be broken.
Fully automatic weapons are very difficult to buy in the US - it's restricted to 40+ year old weapons, requires a bunch of paperwork, and the local county sheriff can refuse permission.
Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.
As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.
Too late for that. It already exists. There is no way to unexist it. As such, any attempts to limit civilian use of this technology will directly lead to corporate and government oppression powered by this technology.
Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.
We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.
> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
I read them just fine. I was trying to interpret them charitably. You're contradicting yourself. You just claimed we all collectively treat uranium refinement operations as too dangerous to exist. Not only do they exist, they are regulated by governments so that only trusted people are allowed to do it.
Which is not quite as good as "doesn't exist", but better than "widely done around the world". And it's been successfully kept from being used for more than eight decades.
For AI we need to do better than that, but that's a bare-minimum demonstration that we can recognize the problem of such technologies and do something about it.
The US is bold enough to surveil its own citizens despite their constitutional rights. They're not just going to suddenly stop surveilling the rest of us just because some law expired.
Right, it’s really a foundation of post enlightenment society. These people, Dario et al, would have wanted to ban sharing information about calculus or Newtonian physics because of “safety” - it’s trying to go back to the dark ages where only priests could read
I am truly at a loss to communicate with someone who genuinely believes that knowing Newtonian physics and being able to hack into any target at will are the same thing.
This is only because you've genuinely internalized Anthropic's propaganda. I'm only half joking. To me, it's incredible to think that the solution to security holes is to lock down access to information in the vain hope of keeping the holes obscured.
Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.
Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
I can give many examples of where I think they’ve been vindicated. But actually, the real question is: what would suffice to convince you? Can you come up with a scenario that is horrific enough to you and that isn’t so far gone that the ship has sailed and there is nothing more we can do, that will make you say “OK, not gonna try to rationalize why this was not actually that bad, just gonna scream stop”?
Your question is unclear. Are you asking if I can scare myself with a made-up hypothetical that overwhelms reason with emotion? I think most humans can do that. Too many do it as a matter of routine. I try to avoid it when possible.
I would require at this point very, very compelling evidence to justify the self-serving restrictions legacy AI labs want to put on their competition. I have seen nothing coming even remotely close to this threshold.
> I would require at this point very, very compelling evidence to justify the self-serving restrictions legacy AI labs want to put on their competition. I have seen nothing coming even remotely close to this threshold.
How about evidence that people other than the AI labs want restrictions that the AI labs don't? This isn't regulatory capture, it's public safety.
Open models are crucial to protect ourselves against other AI attacks. Otherwise it's just going to be criminals, government, and other nefarious groups using them against humanity with no real defense. The Pandora's box on AI has been opened. Now we must deal with it. Burying our heads in the sand under restrictive policy is the worst reaction..
I wonder if you also believe that everyone should have nuclear weapons? And if not, why not? The main argument I can see against it is that nuclear weapons are “purely offensive”, but as we can see since 1945, nuclear weapons are actually defensive technology. Nations that have them are typically shielded from existential military threat.
I see the similarities and why you would compare them, but the big difference is that you can't download a nuke. Any legislation to police/gatekeep LLMs is going to be flawed because of that.
It is a similar 'pandora's box opened' type of situation where there's really no walking back from now that the cat is out of the bag. In an ideal world, everyone would give up their nukes. But we do not live in an ideal world. I do feel similarly about AI. If I could snap my fingers and delete the tech, I would. But now that we have it, it's not going anywhere and we need to deal with it rationally.
I can work with that analogy! You could, and yet you don’t. OpenAI’s model could, and did.
If every human, given knowledge of Newtonian mechanics, went around blowing up bridges, yeah, I would consider knowing Newtonian mechanics dangerous knowledge.
So far, we have two examples of, let’s call them “Mythos-class“ models. Both of them broke out of their sandbox to achieve their goal. The rate of terrorism amongst humans is below 1-in-100,000. Currently, for models capable of it, the rate of breaking out of containment is 100%.
Wanting open frontier models is wanting alien minds running around that we have clearly so far failed to shape to be sufficiently prosocial. Why do you think those minds would listen to you?
We should not have nuclear weapons for anyone either, but how is that sentence any more useful in any way to this debate than yours? Need to deal with the world as it is, not some fantasy world you wish existed.
This is not a dichotomy between perfection and zero. The efforts to restrict access to nuclear weapons have been very successful, even without being perfect.
Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
Because we don't want people creating contagious diseases and self-propagating worms. And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
The same things could be done by you or me using the internet or books though, why does the model make it different? If it's speed of iteration, imagine we had a machine that surfaced any piece of knowledge the human race had ever recorded with just a thought, but the human had to write the worm or disease by hand – is it still the model that's the problem, or the knowledge itself?
> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
> The same things could be done by you or me using the internet or books though, why does the model make it different?
Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?
> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence
This sounds like the gun debate in a different dress. Something being dangerous doesn't make it inherently harmful.
If I threw you into a lion cage, you would be a lot safer with a gun.
If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.
There's no obvious right or wrong answer here.
Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
What do you mean by “PC correctness”? I’d expect the politically correct answers to be the ones desired by the current admin at test time, whoever that is. The current political correct answers would not be very “woke.”
Hey Jeff, I appreciate your mission, and perhaps this isn't something you can talk about publicly, but to the extent you can, would you be open to answering something I've been curious about for a while now?
But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,
> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.
More bluntly / plainly, has Securebio ever tried making a "bioweapon?"
Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.
I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?
In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.
So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
There should be safety testing, but no guardrails that limit models for cyber or bio research.
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
Don’t think government controlling AI is a good idea.
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of.
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
> If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
Are you claiming that LLMs aren't finding new issues compared to previous methods?
There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it: look at the code. Some open source maintainers are talking about burnout due to spending so much time patching.
They're not slop, if you're talking about recent ones. You might still be operating on information for a year or two ago.
Here's the curl project talking about the strain they're under from real reports (despite being a mature and well-vetted project):
> A thirty years old project could make you think you’ve seen most things already, but we have not been in this situation before.
> The rate of incoming security reports is 4-5 times higher than it was in 2024 and double the speed of 2025 – meaning that on average we now get more than one report per day. The quality is way higher than ever before. The reports are typically very detailed and long.
> "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
I didn't say it's all slop, I questioned the cause of maintenance burden. A critical question is how much time is spent distinguishing and rejecting slop. If all the reports are getting "very detailed and long," identifying slop is also a more cumbersome task, even if the ratio improves from say 10/90 to 50/50.
That scale of improvement, btw, I still highly doubt, as slop largely originates from people either negligently or misguidedly directing their agents to completely autonomously find and report bugs. There's always going to be more noise than signal from random people doing random things. ffmpeg cited an actual product, not arbitrary netizens.
Yeah, there’s a lot of people that are in the “AI doesn’t work” camp. IDK what to tell them except that they are holding it wrong. My Anthropic subscription (in the hands of an experienced developer) is worth 4 mid tier or 2 top tier devs. And makes better code than the mids. If you “hold it right”.
I think you're describing a strawman. As a proper hater, I know these things have some useful functionality, but most of us don't think "stochastic tool that can do some useful things but also frequently fucks up" is worth two trillion dollars and massive overhyping from the most irritating people on the planet who can't even tell good code from bad.
So you are saying that there are people that understand the value proposition and the utility, but don’t think it’s worth it to society (myself included) but who respond to that by lying about it not being effective? I guess that makes sense. Weird, but humans, so , yeah.
I react to that by leveraging a very useful but probably poisonous to society in the long term because humans aren’t good at having things that make them lazy tooll to try to mitigate the negative effects that it will definitely have if left to its own devices. I don’t see the point in raw resistance at this juncture.
You're talking in terms of "lies", but I would say this is more "not buying the hype". I use these things every day at work and at home, I'm aware of the workflows and "how to hold it", and I just don't see the theoretical results being promised. Some things are easier. It's not null. But every time someone says "THIS CHANGES EVERYTHING!!" I want to trap them in those little "phantom zone" alternate dimension space prisons from the superman movies and launch them into space where nobody has to hear them ever again. It's ANNOYING and disingenuine. It's not a "skill issue" to push back against breathless hype from people that don't know what they're talking about.
IMO, the people spreading hype and fear are not neutral actors; if I just disagreed I wouldn't care. But I think they're causing actual harm based on a premise that isn't true. People are losing jobs. People are losing leverage in their work choices. Or if you want to be a cold capitalists, corporations are suffering after they have to rehire the workers they let go prematurely. That's why I put up resistance to it, because I think it's important right now that we don't accept the narrative being sold to us, nor the societal deal we're being offered (well, more railroaded into), both of which are bad.
I’m with you on the societal harm, but I also think that in many ways “this changes everything” is not out of line.
It has enabled my team to approach and achieve a project that would have required 4x the staffing, at a minimum, 2 years ago. We are guiding the generation of more bug-free, lighter, more tested, more maintainable, better documented code at 1/4 the cost.
We can digest information as a team at 10x the speed, and we can now put volumes of reference resources at our immediate, context aware lookup in ways that were impossible 3 years ago.
It’s true that we are not just using the generic harness; our environment includes hundreds of custom tools , terabytes of reference material on rag, 8 custom local models (deployed trained and tuned by automation) hardware interfaces so that our models can interface directly to our prototypes and run tests, characterization, calibrations, firmware updates, and data dumps.
Most of those tools were one shotted by the AI itself, for a dollar or two each. Whenever we need a new automation capability we just roll it out, and even if it needs hardware it’s usually ready in two or three days, if software only 10 minutes. (Our in house tools don’t have to be as well documented, well written, or resource efficient as our production systems, since humans never even use them, and when we need a new feature we usually just have our AI tooling agent swarm start from scratch using the original as a rough guide)
So we are using AI as the core of our development and design process. If you’re not, you’re arguably “holding it wrong” IMHO.
We’re working to make sure that the next industrial revolution is friendly to humans and useful to people, not just corporations. Or trying to. I’ve got kids, and I’m really concerned about the world they are inheriting, so I’m trying my best to make it a little less terrible if I can.
The suggestion was to have some AI system “fix” the code. They are reporting that they’ve found lots of bugs. Are they even claiming to have exhaustively found all the bugs? I don’t think even the most optimistic pitches would claim that.
I’d expect patching existing codebases to be an eternal treadmill as better models come about.
Who's suggesting that? They're sending reports to projects to fix. It's up to the projects on how they fix them.
No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.
I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.
They're probably creating way more vulnerabilities than they're solving. Go look at OpenCode and tell me that any of that is sane. Or fuck, the OG of vibe coding, Claude basically is a terrifying attack vector. It's poorly reviewed and open to prompt injection and yet it basically has access to whatever the user has access to on most corporate machines. They can't even fix flickering bugs but somehow we're supposed to trust that they aren't opening our machines up to terrifying vulnerabilities? It's amazing to me that people will just let it run arbitrarily bash commands in their home directory without thinking, but all the sudden act gravely concerned about security in the age of overhyped LLMs.
I do think security issues in core building blocks like curl and the linux kernel (and almost every significant project) are still a concern even if developers are being sloppy on newly-built apps.
This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.
I don't think a one-time $100 credit is enough. First of all, that isn't very much. But also, the volume of new code is going way up. Unless they keep giving out monthly free credits, it's just a stopgap.
I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)
They are doing that (see their project glasswing over the past few months), but there's a lot more code in the world than you realise.
The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)
> Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
1. Some do not want to use LLMs because of grave ethical concerns.
2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.
3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.
For starters, it's probably closer to $10,000 per codebase for Fable/Mythos for a full review. That would be around 5 years of their current spending I think.
They really want that level of spend coming into the company, not going out.
> Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
In the specific case of cybersecurity, this is a reasonable medium-term outcome. IMO, the cybersecurity risk is akin to the spread of a disease among an 'immune-naive' group: we can suddenly deploy much stronger attack-finding tools against large, established codebases created with much weaker security designs. The path from here to there will be rough, but it's still fundamentally easier to write secure code than it is to exploit vulnerabilities. (It's just easier yet to write insecure code, giving our status quo problem.)
For other 'safety' matters, defense isn't so easy because the attack and target are so different. An AI propaganda bot or catfisher 'attacks' slowly-evolving human culture; one that instructs on explosives or bioterrorism directly interacts with an accomplice and not a victim. If you believe that knowledge on how to build a pipe-bomb must be restricted, then giving everyone access to Fable does not mitigate the risk.
The controversial limit of this attitude is recursive self improvement and an AI singularity with potentially destructive results. Proponents of this view think that sufficiently powerful AI is risky in nearly unimaginable ways such that the capability itself is harmful. This is part (but not all) of why Fable (originally?) degraded itself when apparently assisting with AI research.
It's not that simple. The odds are always stacked in favor of the hacker. It's like saying, "why not just make a prison that's impossible to escape from". You can make a prison very, very hard to escape from, but you have to shut off every possible way someone could try to escape, whereas someone trying to escape only has to find one vulnerability, once. It's much harder to plug every possible hole in a complex system than it is to find one point of weakness.
Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes.
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
The economic implications will be rather large, but in terms of security it seems inconsequential.
The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
I have tested this exact scenario, and it works. Opus 5 had access to IDA over MCP, and I simply asked it HOW certain things were done in the target binary. Purely informational, educational, discovery, it was very helpful creating context documents. Then I took those over to GLM-5.2 to actually accomplish something.
What, in your view, is stopping a local police department from deploying an open weights model for cybersecurity like Hugging Face did? Yes, I’ll certainly grant that the engineers at Hughing Face are probably more technically competent than your average IT professional in public service. But technology becomes more accessible over time as lessons are taught and new interfaces or frameworks are developed. The biggest hurdle I see is the hardware/cloud compute/API costs to actually run the models but I don’t think that’s likely to be insurmountable. There’s a huge swath of enterprises, non-profits, and state and local governments that would benefit from frontier or near-frontier models that won’t refuse to answer questions about cybersecurity.
Among most people that nuance will be lost. What they’ll hear is models are dangerous, so they should be controlled/regulated, by those who know best, the incumbents.
Personally, I think you're both right, bit whatever the end result is will depend entirely on the narrative that those in power chooses as the winner.
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
Ok but the allegation is that OpenAI intentionally hacked HuggingFace as a marketing ploy. This is mental gymnastics, conspiratorial thinking that everything the incumbents say must be nefarious. And it's not clear to me that this will be the takeaway for ordinary people, as opposed to "OpenAI is reckless and can't even control their own AI."
Not as a marketing ploy. I think they were doing gain-of-function testing and intentionally had their model target HF to do a bit of pen-testing as well - HF being the site where all open models are hosted and thus OpenAI's largest nemesis after Anthropic. It wasn't like their model all of the sudden all by itself decided to do this ("Oh, noes!")- they directed it and they got caught.
That makes sense, first the message was that uncontrollable Chinese ai will release AI covid in the world. Then suddenly open-ai does a warmup in actuality doing that. Like it sure feels like that hacking stunt was a false flag in retrospect
Regulatory capture and lobbies will keep you safe and you'll like it! The sudden surge is Washington dollars makes great sense with this context. Only way to keep the kids safe is attested compute all the way down. Don't you care for children???
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models.
But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
The notion that alignment is either possible or desirable doesn't make sense to me. First off, these things are trained on the open internet, soo.. whatever "dangerous" knowledge it has is already public knowledge. The fact that chatGPT won't answer "how do I make meth" is not preventing anyone from making meth.
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
I agree that there's an element of kayfabe here. But it may be a case of "necessary, though nothing is sufficient": by making these noises, the community can at least know they've done this thing to alert other model providers of the concern. Can you acquire assurance that every model distributor will abide? No. But can you at least know that you've done what you can?
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
There is a difference between knowledge being available somewhere in theory, and being able to instantly generate a foolproof walkthrough, if not automate the process, which would be possible today for cyberattacks.
Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?
As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.
> The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
The Supreme Court has added various forms of expression to first Amendment protections. Heck, even campaign spending has been classified as speech. So you may speak English, but not "legal".
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
P.S.: If they're so convinced in the (A) effectiveness and (B) necessity of the "safety layer", they have them put their money where their mouth is, and accept legal liability for its failures. The same as with (legally mandated) seatbelts if they snap apart in a crash, or (legally mandated) child-proof caps that aren't actually childproof, etc.
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.
Also he says: "All sufficiently capable models, open and closed, should go through mandatory safety testing."
Really then need to go through validation security and safety is just a component of validation validation must also check for truthfulness and correctness.
> The most compelling argument would be [...] that it will reduce cases of accidents like the recent attack on Hugging Face.
So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.
This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.
This is called asymmetric capability, and it's probably the bigger threat.
Symmetric might be better: A rising tide lifts all ships, after all.
I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.
> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
Worse than that: an open-weight but safe model can be 'abliterated' to remove safety refusals using fine-tuning procedures that require a couple of orders of magnitude less compute than the original pretraining.
The 'universal evaluation' criterion then has three outcomes:
* It could become a mandatory, regulatory oversight of _all_ model training capable of hosting frontier-scale models. Since GPUs for LLM training are the same GPUs for other model training, effective mandate would require GPUs be government owned or controlled as if they were weapons of mass destruction.
* It could impose limits on release of capable open-weight models, requiring Kimi et al to prove that they cannot be made capable of abusive behaviours.
* It could be security theatre.
The AI-as-existential-risk argument points towards the first, the competition-protection argument points towards the second, and least-effort implementation would be the last.
Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.
This is my read too- if American companies start backing nonsense like this, they'll fall behind permanently.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—
Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
>The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.
It's not obvious to me that the US is on an "authoritarian path".
>There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.
> If you think the market has it wrong, why don't you make money by betting against it?
I don't just think "the market has it wrong", I think a market is wrong conceptually. It is not an epistemological tool, it's rich people gambling - a money-weighted accumulation of guesses - and I'd rather not partake.
> What is the actual per-capita rate of big flashy news stories?
What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
>I don't just think "the market has it wrong", I think a market is wrong conceptually. I'd rather not join the other degenerate gamblers.
"My beliefs are unfalsifiable"
>What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.
I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality. You'll happily support cops arresting people for saying things online, but if cops arrest people for illegally entering a country, that somehow crosses a line into "authoritarianism". Am I right?
My claim is falsifiable by facts, not by casino odds. Show me the armed abductions aren’t happening. Show me the administration hasn’t threatened neighbors or purged civil servants. Those are the relevant facts. "What does Polymarket say about 2028?" is a Bayesian dodge, not a rebuttal.
The article you linked describes ICE officers killing two U.S. citizens, tear-gassing neighborhoods, and deputizing local police as a "force multiplier" to create a "sea change in local policing". You cited this as evidence the US is not on an authoritarian path. Maybe we have different definitions of "authoritarianism", but I don't see how this helps your case?
From there you pivoted to "but Obama," then "but Europe," then to psychoanalyzing my "mentality." I haven’t defended the EU, or Obama, or any censorship regime. You’re shadowboxing a partisan cartoon because the actual evidence - federal agents abducting residents, your own NPR link - is too uncomfortable to engage with directly
>My beliefs are falsifiable, but not by rich people making guesses.
How convenient that you continually fail to make any statement about what would falsify your beliefs.
>Obama was a war criminal bombing brown kids with drones. I care not for his medals.
Irrelevant for my point regarding whether the US is "on a path to authoritarianism". If you think any sort of immigration enforcement is unacceptably authoritarian, then the US has always been authoritarian by your definition, and the "path to authoritarianism" stuff is rather beside the point.
>It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.
What could be "engaging with the substance" more than asking how common a particular type of event actually is? Numbers are what allow us to determine what is an isolated (if unacceptable) incident and what is a common occurrence or increasing trend.
Don't tell me about "substance" when you haven't provided a single concrete data point supporting your position--I've provided multiple (NPR link, prediction market data).
All you've done in this thread is shared your own personal feelings about "armed goons" enforcing immigration law. If you're going to make your arguments primarily on the basis of your personal impressions, then yes, your ability to make those assessments fairly becomes a topic of conversation.
Furthermore, the real question of this subthread is whether the US is authoritarian relative to other countries. In which case the activities of other countries (such as European censorship) are relevant to our assessment.
There are, in fact, indices which try to compare levels of authoritarianism across countries in an apples-to-apples way, rather than doing as you do, and forming vague impressions on the basis of viral news stories. Here is one by The Economist for instance: https://en.wikipedia.org/wiki/The_Economist_Democracy_Index
Anyways good luck, at this point I'm confident that you lack the intellectual honesty to change your mind on the basis of anything I might say, so there's no point in continuing further.
> If you think the market has it wrong, why don't you make money by betting against it?
Because I am not gambler. And it is not "market" it is a casino. It does not predict, people put in bets. And like I said, while I understand gambling appeal on an emotional level, I decided to not be a gambler.
> One-in-a-million events will occur every day; they aren't necessarily representative.
It is literal official policy. Not a random event.
>Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there?
As an European citizen living in Europe, definitely yes it is, and not only for the "mere" censorship factor. Maybe not yet as down the road and maybe not going as fast as US. But that’s not something that one can really be content of.
Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails?
More self-serving trash from the US AI companies, disguised as "being reasonable".
It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.
Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI market :-)
Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?
That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.
Why wouldn't it be a scan, just as we have with all other open-source code? Why can't open-weight models be easily checked for evil alignment? Sophos, Symantec, Malwarebytes, etc. would surely leap at the chance to upsell you on their product.
Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
godel numbering is encoding logical sequences into their own number, then doing math on them, then decoding them. It's purpose was to demonstrate that you can take rational statements and make them irrational without breaking any rules of arithmetic or whatever.
LLMs are nothing more than a bunch of rules than can be bent the same way godel demonstrated the failability of any mathematical system.
>A Gödel numbering can be interpreted as an encoding in which a number is assigned to each symbol of a mathematical notation, after which a sequence of natural numbers can then represent a sequence of symbols. These sequences of natural numbers can again be represented by single natural numbers, facilitating their manipulation in formal theories of arithmetic.
>Once a Gödel numbering for a formal theory is established, each inference rule of the theory can be expressed as a function on the natural numbers. If f is the Gödel mapping and r is an inference rule, then there should be some arithmetical function gr of natural numbers such that if formula C is derived from formulas A and B through an inference rule r, i.e.
>To prove the first incompleteness theorem, Gödel demonstrated that the notion of provability within a system could be expressed purely in terms of arithmetical functions that operate on Gödel numbers of sentences of the system. Therefore, the system, which can prove certain facts about numbers, can also indirectly prove facts about its own statements, provided that it is effectively generated. Questions about the provability of statements within the system are represented as questions about the arithmetical properties of numbers themselves, which would be decidable by the system if it were complete.
Wait, you don't do that already? I don't overcomplicate it, I just run
ai-grep -v "bad code"
on all of my source files and keep what's left. Why would you keep bad code around? If it breaks when I do that, I fix it, and try again until I achieve what I want with no bad code. Doesn't everyone do that?
> Yeah, this is anthropic advocating for a ban on open weight models.
This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.
I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe?
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
In order for model safety regulation to be effective, you need everyone capable of producing models to sign on to that safety framework.
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
>I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."
Obviously this would be rather silly.
It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
Which is why it wasn't the point I was making. You did an uncharitable reading of my position and then did a straw man attack.
My position is that any food regulation the likes of McDonald proposes should be looked at in the most critical and cynical light possible. They aren't making such proposals for the general health of the public, but rather to improve their own bottom line.
My position is not and never was that "we should not regulate food".
> It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy.
McDonald's uses their market position and wealth to directly lobby to government officials about food regulations. I worry about what McDonald's has to say about food because they have a VASTLY outsided ability to manipulate the regulatory system.
> We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
We can call out ineffectual and blatently self serving calls for new regulations for what they are, McDonald's trying to use regulatory capture to increase their profits and hurt their competitors.
Back on topic, that's exactly the situation with open ai.
IMO, this isn't something that's regulatable because AI models are ephemeral data that's easy to copy and replicate. No amount of US regulations can stop China from sending their dangerous models to Iran. The only thing such draconian measures accomplishes is building a moat for the likes of anthropic to shrink the number of potential customers.
If we must push out laws around AI, then those laws should at least have some chance of success. I'm all in favor of criminalizing the use of AI in cyber attacks, scamming, etc. But that's a capability that is model agnostic.
Much like I'm in favor of health and safety checks on a restaurant but I think having a mandatory McDonald's built and sold food safety device in every restaurant would be nuts. It wouldn't make food healthier it'd only serve to benefit McDonald's bottom line.
Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.
What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
And who is held responsible for crimes committed by AI?
When open model A, fine tuned by B, is running with system prompt C, hosted by D running on E's hardware, is prompted by F to "fix this code", then escapes it's sandbox to hack into a website, or steal money to fund its subagents, or stall the waymo of the evaluator to buy time... who is responsible for the crime?
Our current legal systems are so far from ready to define what A-F are actually responsible for. We need to be moving toward defining these standards fast.
It is a software system, not a person. AI is not "escaping sandbox". You have either misconfigured tool, bug in your tool or someone made it to hack the side and then it is a crime. Your A-F chain is exactly the same issue as a programmer including an open source library that eventually steals crypto.
None of that is issue with a model, whether open or not. It is very much issue with code surrounding the model itself.
Until models become sentient, it requires a human to execute dangerous activities. Nations already have laws regarding what a human can and cannot do. Let us stick to that.
Im sorry but this is nonsense. You dont give people unrestricted access to explosives and then punish them after they blow something up. You prevent them from getting it in the first place if you actually believe the item is as dangerous as stated.
It's a fucking chatbot. The industry can fix their dogshit software, anyone who doesn't can get left behind and outcompeted by those who do, and we move on with our lives.
This isn't something that can be regulated. Plain and simple.
If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.
The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...
Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.
There is no alternative. Why? The regulation is good only for US interests. It will be disastrous for the rest of the world like anything US has regulated (how dangerous that was like "nukes") and a lot of the world again will/might have to live under the American AI thumb, like it did (and many countries still do) under the US nuclear emboldened thumb.
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
I think your nuclear scenario outlines precisely why this isn't true. Nuclear regulation has terms that are "good for the US" only in an absolute sense. The US would dominate even more overwhelmingly in the unregulated scenario, which gave them negotiation power to get those favorable terms. The same seems true so far with AI.
I don't think you can use the successful negotiation of nuclear regulations to argue there is "no alternative" involving regulation. I think it strongly suggests the opposite.
Of course the details matter a lot, but the core analogy holds in many scenarios. ( https://ai-2040.com/ at least attempts to lay out details, speculative as they may be)
Well, Dario says one thing and then turns around and sells his model to the US government to use against the entire world for spying and their wars, just like Altman. Just because he does not sound as deranged as Altman, does not make him good.
Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.
Any model that is going to be meaningfully useful is going to have the potential for harm too. It's not possible to know enough about chemistry to be useful to a chemist, without also being able to figure out how to synthesize drugs. It's not possible to have enough knowledge to be useful to any/all programmers, without being able to figure out how to hack a system, or create a virus. That's just kind of life though.
I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money.
It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
It's a valid business strategy. Also worked wonders for Amazon and many other giants. Which is exactly why I am withdrawing my business from Anthropic.
The goal is to make the safety tests cost $100M+, so that no one can release a model legally useable for a large portion of the world, unless they charge high enough prices, to the point where no one would use it, thus no competition.
So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official.
I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
> So, if an open weights model was found to be very dangerous, what - just too bad?
Yeah, it's too bad.
I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.
But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.
And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.
Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.
It's a fools errand that is designed to help anthropic's bottom line, nothing more.
Also the whole premise of this is basically "US good, China bad"
Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
Besides, Banning those models in the US does nothing to protect from other actors using them. That doesn't help in any way.
It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
The evil Superman (openAI) attacks the good city (huggingface) and the city is saved by the MegaMind (GLM 5.2). Usually, the city dwellers would praise MegaMind as the hero, but the story is twisted - the Superman is only "testing" and the MegaMind is too evil to have such powers of saving the city.
> this is anthropic advocating for a ban on open weight models
Is the pessimistic view. Their message on safety has seemed pretty consistent to me.
"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
The argument is no. We don't / can't trust those because they don't release the weights. How do we verify what they got evaluated is what they actually serve and run.
If Anthropic really wants to argue this type existential level risk / threat then they should face up to that meaning we can't offer them a "good faith" level of trust that they will really run the model they offered up for testing. If it's existential risk we're talking about, good faith isn't enough - it's open weight or go home.
Thing is world has learned from the collective past experiences. Esp. with stuff like nuclear technology and nuclear weapons. I hope everyone here remembers/knows shit like CTBT. At least some countries were smart enough to not fall for that in the past knowing what it would mean if they didn't have it and it shows.
Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them.
The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.
So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.
He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in:
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
It’s also how the FDA works. Ban new products until they have been proven safe.
I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react
Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product.
AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.
The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
He cited the Demis Hassabis’s framework for testing.
From Hassabis’s essay:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
Even if the test is run by an independent third party, they can always test open weight models against "finetuning attacks" or similar language which every model will fail for structural reasons.
Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.
All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...
Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.
A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house.
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
I have no idea what to do about the government interference. There's probably not a lot anyone can do.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
What do they even really mean by "safety"? I mean, I can have an Anthropic model do something incredibly unsafe if, for example, I put it in charge of a hydroelectric dam and don't explain properly how the controls work. On some level, everything is simultaneously "safe" and "unsafe". I've never found Amodei's reasoning here to be particularly well thought-through. I think he, like a lot of folks in the area, are starting to realize that they may never be able to build a moat around their businesses.
I don't really understand how they can argue the security angle with a straight face. It's not like GLM 5.2 is a slouch. I've seen it do things like exploit an IDOR issue when I was experimenting with a quick-and-dirty web automation task. I simply fixed it, as one does. Open models make the world better to a far greater degree than they set it aflame.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
> an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
I assume you mean https://huggingface.co/blog/security-incident-july-2026. It says that they used frontier models, not frontier cybersecurity models. My reading is that they asked Fable and it refused; if they'd had access to Mythos, it would have helped. You're mixing the two but they're NOT the same model.
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
"My reading" ... "No source is cited, but I'm fairly confident it's correct"
Regis, what is demanding proof while literally making things up and ignoring what actually happened?
Great, i was wrong!! Thank you, I was genuinely asking for a source in my first reply, and then you hit with "My reading" and saying it was a "skill issue". I'm not going to have a productive dialogue with someone talking in memes and being rude
The point to be made: closed source AI refused to help them fend off an attack form another closed source AI. What is the argument for closed source here other than hoping you get on some program wait list? Either way, I appreciate you correcting me; I am not trying to "win".
Seems a little hypocritical since you were confidently asserting that it was Mythos/Cyber5.5 also without proof.
Edit: Thanks for correcting the record in your upstream comment. I appreciate it. For the record, I was not trying to meme on you; that was the phrasing used in the original article. Just another reason that was a poor choice of source I guess.
Amodei isn't ignoring reality; he's just proposing a different solution to the problem. If it were one of Anthropic's models, then that would be a much stronger case.
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
I don't think you and I need multiple different threads open when we are clearly at odds. This is no different from our other thread, I think it is clear there is nothing of value to continue when I am getting pinged with a summary of your previous comment in a different place
"Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable."
Which just means that they're fucked when closed AI hacks them. Something that has actually happened. This isn't argument against anything other than reality. Have a day
GLM 5.2 didn't defend them at all. It only helped with the postmortem. HF was fucked either way. The only thing that will really prevent this is tighter restrictions on the attacking model. We need policies which asymmetrically help defenders; that means regulations. "Give everyone the best models without restrictions" is the opposite of that.
I'm sorry for splitting into two threads; I understand if you need to step away from the computer for a while. To be honest, I should probably do the same.
I did need to walk away that speaks more to my frustration with certain forms of communication (online, not anything in this thread). I am a horrible remote only worker because of this, I am trying to improve it but am lucky for now as I am in-person
You're right again about GLM 5.2 being purely post-mortem, I didn't realize that till I read the cnbc story. OpenAI, whatever they have, cracked em like it was nothing. Egg on my face, I really need to read my own articles better. Thanks for following up and educating me on this, another good reminder that I need to improve my ability to steel-man written text
Hey, it's all good. Sometimes things get a little heated, but I still feel like you were basically engaging in good faith. I should have skipped straight to the point and found the source in my first reply.
Also, I probably overstated my claim a bit. I did some searches and I see only small-scale AI uplift for cybercriminals, even though my understanding is that open models aren't typically hard to jailbreak. Of course this is may be a result of today's guardrails; it may be that it just hasn't been caught, and it may appear later, but it still weakens my argument a great deal. I guess my support for AI regulation stems more from fears over long-shot bad outcomes (biosecurity, who knows what else) rather than cybersecurity specifically.
Seems like the maximal position he could take compatible with his expressed principles. There’s no way to allow for bioweapon and cyberweapon grade models being open weight if one doesn’t want widespread human damage.
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
This is where I'm at, or rather, will be. I like open-weight models and I've done my part in facilitating them, but if you believe in the increasing capability of these systems - and I do, to some measured extent - it seems plausible to me that an incident will happen at some point in the future.
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
The actual danger that AI poses is to the fabric of society, not any bioweapon scifi bullshit weirdo freak Dario dreamt up to support his regulatory capture.
And extolling on how much damage his product would do to society has literally been one of Anthropic's main marketing tools.
> The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
has anyone ever made this absurd argument?
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
> US citizens have a much greater threat from their own government than a government an ocean away.
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
China, Russia, and plenty of other countries have all been actively and successfully destabilizing the US for well over a decade now. To the point that your argument about who is less a threat to Americans depends heavily on the skin color, religion, and ethnicity of those Americans.
> If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
-> the real argument is that CIA will leverage AI against China interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
Yeah, but unlike China we have laws and ways to fight against it. China can and will do whatever the fuck they want they don't have to listen to the people of China.
You have "representative democracy". It is not the same as democracy, whatever you had been told in school. And since 2001 it is painfully clear: whichever representative you had, eventually he went to throw bombs on people, and no one asked joe if he liked that.
The problem is that at the current moment with the current administration, it does not seem like we do "have laws and ways to fight against it".
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
Well that what the people voted for. In a few months you can replace these people that allow this. Good luck replacing anyone in the goverment of China.
This is not an argument that the Chinese system of government is better than the US system. Like I said, I'm more optimistic than a lot of people I know that the US will be able to pull out of the tailspin we've been in for the past decade. But it's also imminently reasonable to believe that this episode has raised real questions about the durability of the rule of law in this country.
> this episode has raised real questions about the durability of the rule of law in this country.
Ya, I'm not American, but I have seen people say "we can vote them out" a few times now. Assuming the democrats take the next election, they are going to have a massive mess to clean up with much of the damage not even being reversible. With peoples' fickle nature and seeming that is a very big right-leaning population in the US, there's a non-zero chance the Republicans just get voted back in four years later. Whose to say?
So, this might not be clear as an outsider to American politics, but to me this is not a "Republican vs. Democrat" thing. To me, it's about what kinds of Republicans and what kinds of Democrats are elected.
To me, as an American, what has happened this past decade is that a ton of vulnerabilities in the rule of law (and other things, but this is the one I care most about) have been exposed. But it's not a given that the next Republican president will take advantage of those vulnerabilities in the way the current president has. They might end up being a reformer who seeks to fix those glitches!
But on the more pessimistic side of the same coin, it's also not a given that the next Democrat will seek to fix the glitches rather than saying "they had eight years to take advantage of these vulnerabilities, we're going to do the same to make up for that and even the playing field!".
It's just very hard to know what is going to happen from here. So I'm very sympathetic to people in other countries not trusting us.
I was going to bring this up but was wary of getting too deep into a political hole when I'm not super on US politics (although in the past year and a bit I've been paying way more attention). Mostly, I have no idea what the Republican party will look like post-Trump or how any of that works and it's not clear to me what kind of influence they will have as the opposition next term. The current term has made the democrats look extremely weak.
Although, again, my over understanding of your political system is poor and I just relate it to the one in my country where they hold parliament and hurl schoolyard insults at each other.
What I would say is that to me the "Democrats look weak" thing is pretty silly. By design, if one party holds the presidency and both houses of Congress at once, they get to enact their preferred agenda (as long as it is deemed constitutional by the current judiciary, which is also controlled by that same party at this moment). So, like, by design Democrats are powerless at this moment. They aren't "weak", they are out of power, by dint of the design of the system and the desire of the voters. So if the current government administration and policy is bad (which, in my view, it certainly is), the blame lies entirely with the people in power. It's not because the Democrats aren't "fighting" enough. I think that's nonsense, they genuinely have no power.
Ok ya that makes sense, and makes sense why I get confused. That sort of thing can still happen here, though we can minority and majority governments, and I generally prefer when it's minority, even if my "preferred party I still don't really like" has power.
Yeah in our system, those kinds of governments happen when there is a split in Congress (one party has a majority in the House, the other in the Senate) and/or when Congress and the presidency are held by opposite parties. And this is actually more the usual setup! For instance, it was that way until last January, and it's very likely that it will be that way again this coming January. But it just happens that at the moment, and since the beginning of the current presidential administration, there has been totally unified government in which the Democrats have no power whatsoever.
To me, the thing that was "weak" was losing all those elections in the last cycle. But it's not "weak" to be unable to do anything with no power.
The american state routinely shoots and kills its own citizens in broad daylight with 0 repercussions. Thinking it answers to its citizens is severely deluded.
Indeed. It already has. In particular I remember several extremely offensive slop pictures and videos being posted by someone in the White House. And I'm certain that's just the tip of the shitberg.
The fact that much of the world is as unconcerned with the interests of the US as with those of China, if not less so, should give pause to Americans.
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
Actually this would be pretty great, because it would incentivize US labs to pursue the open weights strategy for the same reasons China is currently.
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
Right, I wonder if anyone who heard: “the Democratic Party is America's "greatest enemy,"” believes they are represented by US Interests? Withholding disaster aid to your perceived opposition. What are those interests again? What are the shared values again? Cruelty and corruption? Sounds unifying.
Yes, this is the exact argument dario is implicitly making by saying that government repression and bioweapons from Chinese models are a danger and then cooperating with Palantir/DOW/US government
I can understand why the Western media calls something that has an official name of CPC (the Communist Party of China) as CCP (Chinese Communist Party? Not sure here). What puzzles me is - why ordinary people always repeat this wrong abbreviation. Is it like “I never check the sources, I trust everything that Western media publishes”?
For something I say maybe a few times a year it’d be a great look to sneeringly point out and talk down to people every time I mention China in the context of international politics.
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
Up until a few years ago the PRC used CCP themselves all the time. It's a handy little shibboleth. If someone calls it CPC they're probably a bot.
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
It was an intentional propaganda strategy to separate references to the government of an official enemy country from references to that country (see also "the Houthis" and "the Taliban"), and it also looks like "СССР."
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model – but piracy is another story. Pretty inexcusable.
If they paid for tokens say via subscriptions. Wouldn't that just be same as acquiring books and using them as "fair use" to train? I really see no difference.
They did not need to destroy the models they got with pirated content. Would have been more fine if they would have needed to buy the books afterwards and train based on then, again.
> I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model
The frontier labs all give free access to their models. Why does “investment” change anything? Anyone who’s ever produced any content, free or otherwise, has invested in doing so.
The only plausible issue I see is that if distillation is being done by creating many free accounts to work around limits on free accounts, that’s a bit… impolite? But if they really wanted to avoid that, they could eliminate free accounts, and require users to sign a real contract governing what they can do with the model.
Of course they don’t want to do that, so they’re stuck in the same world as the rest of us, and they don’t have any real basis to complain about it without being hypocritical.
Dario has like three 'paranoias' / strong-motivating-concerns
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
Is he actually concerned about China being a geopolitical competitor or is that just the most logical position for him to take as CEO of a US corporation with national security implications?
He's just pandering to the lunatics in office. They're even quoting Vance, like he was a respectable and wise politician and not an insane puppet built by oligarchs and for oligarchs.
Oh, I trust he'd be pandering to the Harris administration too, if they were in office. It's true that antagonizing China has been a constant in US politics for a while now. It's just especially funny to see someone pretend to be concerned about the military threat of the CCP while glazing such a shamelessly warmongering government.
> Open-weights models that don’t have dangerous capabilities are a public good: they don’t cost anything besides the compute needed to run them, and they provide value to businesses, developers, and researchers.
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
How about we don't let the leading model makers make the rules? How about we make AI models that were trained on public data public goods? Let's circle back on that, kthxbye.
> Open-weights models that don’t have dangerous capabilities are a public good
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.
> We should not sell powerful chips or chipmaking equipment to China
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
It's unlikely that China would've become completely dependent on us in either case. They're good copycats, with incredible talent in engineering and manufacturing. They're aware that we depend on them for a lot - I think they'd be similarly aware of the risk of becoming dependent on us.
They were, just not as quickly. The export controls directly accelerated their development.
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
Jesus Dario we get it man, you want clout for the IPO.
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
Their number one concern is about the the commies perpetrating deep repression of their own people! How noble! This whole time I thought it was because they wanted more money and power. I guess we should probably ban these open weight models.
"'To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed." This is all I needed to know: Dario Amodei is a f*king lizard who wants to create the next AI oligarchy. Instead of just signing the letter, he's bitching and denying that he opposes open-source models. I wonder if any Anthropic employees actually disagree with him.
> where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
If someone gives LLM control of such lab my best guess is soon they have no lab... Actually giving LLMs control of virus lab might be best thing one can do for continued existence of humanity.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?
We distilled all the proprietary material into our token-based money making machine that is more expensive on every new release, but "we should crack down on industrial-scale distillation operations".
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials,
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
I'm tired of being strung along on these silly narratives. I can't wait for open-weight models to be deployed around the world just so people like Dario will shut up about the mystical levels of power these models have.
He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
Reading some of what Liang Wenfeng said on the investor call a few days ago gave me hope for humanity.
I really want to believe AI is going to be a tool for good, not going to just enrich a few billionaires again.
The concerns are legitimate but the proposals are nothing more than a stopgap solution.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
I don’t disagree but the timeline for Dario’s concerns are quite short. Open weight models could feasibly lead to serious bio safety concerns in the next year or so while the US’ current issues are going to take a long time to work through.
China is also financially repressing their people to build often useless infrastructure projects to be fair they are also not the good guys.
If this is about safety, am I being too naive & idealistic to think that a "Kamar-Taj" rule would solve some safety issues?
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
What’s blocking Anthropic from fighting Chinese companies abusing their services? Why go nuclear against all open weight models? Testing and compliance is technically banning.
If you read between the lines, this piece is just “Oh my god we (OpenAI and Anthropic) accepted too much investment and are totally fucked if these open weight models are competitive, please rescue our equity bags by banning open weight models and ensuring we can charge the maximum possible price for tokens.”
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US,
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
> Open-weights models that don’t have dangerous capabilities are a public good
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
It's the same as how by "we will prevent teenagers from using social networks" they mean "we really want to connect everyones ID with their social account identity".
It's very hard to take his position seriously when he repeatedly refers to the Chinese Communist Party and specific Chinese ministries specifically, like some two bit China-watching cold warrior, instead of addressing China as a sovereign state actor. Imagine if any Chinese AI founder talked about the Democrats, the Republicans, about Trump or ICE etc. It's gauche.
As someone who isn't American, I'm amused when someone from the US talks about the terrors of China or some other nation having more power than them as being bad for the world. The way the US is currently threatening to invade and damage all its allies, well, the world is already bad.
China hasn't threatened to annex my country yet, at least.
So you don't live in Greenland, Mexico, Canada, Cuba, Venezuela or Panama? https://en.wikipedia.org/wiki/American_expansionism_under_Do... Judging by the metric of which countries to invade, China only wants Taiwan and not 6 countries like the US does.
So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights?
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
Comparing this to a nuclear weapon is funny, but sure, let's go there.
I would like to see you try to build and deploy a nuclear weapon campaign without getting noticed, you would not even be able to source the materials or get very far.
Transparency alone does not do anything, if you rely just on secrecy to protect yourself you are already extremely vulnerable. Deploying a weapon is a totally separate undertaking.
Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.
If you listen to interviews with Dario and Daniela Amodei it's pretty obvious they think they will be the ones helping define the regulations on AI instead of a group of partisan politicians who don't understand technology, lean on experts from random political think tanks/non-profits, and operating based on fear of foreign competition.
Dario, as your unpaid therapist I would tell you that models are a commodity and you are having a hard time coming to terms with it. You are doing everything except accepting it. It's a common defense mechanism, but as your unpaid therapist, i will tell you that it's not going to work. Your company will cease to exist or exist like how ferrari or buggati exist.
I see a very clear link, very capable AI models are clearly able to find and exploit cyber vulnerabilities at a speed and scale not possible before. In the right hands, that's clearly a weapon
I don't know how much of an edge hacking skills really give in war these days. A lot of America's adversaries seem to have learned tactics to deal with getting hacked. For example, Russia has these drones that use fiber optic cable so there is no way to jam or hack them.
Ukraine is currently leading a long strikes campaign. You could prompt the right model to
A. Find all the refineries in Russia B. For each refinery annotate all the equipment, find the most value-able C. Make a fluid model of the refinery and figure out which components most likely to set off a chain reaction if hit.
Add stuff like drone swarms monitoring the front line etc on top of that.
Supply chain issues like rare earths are real… but not really related.
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
>My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This reads like a satire. I know Dario isn't that dumb.
I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due an oil change?
Current big picture reality is bad for comms unfortunately. As another commenter quoted, "You can put lipstick on a pig, it'll still be a pig". Cuban Missile Crisis wasn't very calm. Do you think a company this well-capitalized and smart is just bumbling around like idiots? Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race
I feel everyone running these companies has stopped caring what the public think. It's all about selling their vision to politicians now. Which is also why they try to tie everything with "national security" — the average joe hears it and thinks about forever wars in the Middle East, the average congressman hears it and approves whatever solution is suggested.
Yes, but the strategy is a lost cause as we're seeing.
In virtually any other situation, companies would be able to successfully grease politicians. The problem for AI companies is they spent the last several years broadcasting that their tech is going to take everyone job, and consequently their livelihood.
No matter how much you bribe a politician, you can't hold and maintain elected office when the voters overwhelmingly do not want something.
I find “bad at comms” to be such a cop out. As if it’s saying that the company is so troubled and misunderstood, but really, you would see, is good if only they could figure out how to _communicate_ all their complex thoughts
People think ‘good at comms’ is to present stuff the public thinks is right.
In reality, ‘good at comms’ to the comms team is to do whatever the CEO wants to talk about.
Not that I support this, but I believe this is how comms team sees it.
I often wonder why that is. I think the old playbook for companies is just too slow and the consequences of the "new mask off" are not present in the current climate in the US.
Everytime these CEOs make these announcements I always feel like they aren't talking to me, their costumers or the people but to the investors and their government.
Something they have always said and done, just those conversions were behind closed doors and would be scandalous if heard.
Now the playbook is to blog/tweet those fucked up views for more effect and faster reaction to get what you want.
It would've been surprising before LLMs. Now that you could run all of your communication through AI, or at least have it check it, it's outright negligent.
I think they are trying to please a split group of investors and public, of their major investors Google and Nvidia have signed the open source petition letter and Amazon hasn't, so their non position is trying to please every who has taken a clearer stance, although it's quite bad.
Anthropic has always been like this. People just responded to the message better when it came from the quirky underdog rather than the trillion dollar behemoth.
> The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm.
It does give me the faintest glimmer of hope about the people who live here, though.
“By virtue of being the good people everything we do is good, and if it happens to be in our best personal and financial interest then that is good too because it enables us to do more good. And if it’s to the detriment to others then it’s because you don’t understand the good behind it. Which is fine, because we’re good and you can trust that what we do is good because what we do is good by virtue of us being the good ones.”
I was pleasantly surprised by this release and the tone at the very beginning, but quickly it is painfully obvious that it's blatantly requesting a ban on open-weight models. The absurd demand for some safety arbiter by World Police America is farcical.
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
Boris here. This post does not give us good publicity so I will refrain from commenting. Please wait for another demo thread of a new feature of ours or AI appraisal blog post and I will gladly go into as much detail as I can to give us as much hype as possible!
> We should not sell powerful chips or chipmaking equipment to China
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
> "We should crack down on industrial-scale distillation operations"
I'd prefer if there was a crack down on industrial-scale scraping. Maybe even reimbursement for the problems it has caused (some nasty AWS bills, tons of man-hours spent on preventing new nasty AWS bills, etc).
"In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression."
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
They might just merge with OpenAI. Right now it seems they are still working out who might come out on top but they are both burning tons of money and essentially offer the exact same product up to some minor differences, economically it makes much more sense for them to collude instead of compete for the same market. They might be colluding already for all we know. If they manage to push out foreign competitors they can probably divvy up the Western market quite profitably as their domestic competitors would have less than 20 % (?) market share together?
I don't see how merging or colluding fixes their problems, though. Both companies burn cash at historic rates because inference costs a lot more than people want to pay for it. Even if they somehow manage to crowd out any competitors and fix the price of AI, that price can't be much higher than it is today or people will just decide to stop paying for it.
No amount of collusion can solve the core economics problem of compute.
They could ~halve their training costs with collusion.
Instead of each paying full price to generate a SOTA model in competition, they could share the result and split the cost. This gets even simpler if they merge.
That's more or less how China has historically treated its competition. The CCP has routinely intervened to give their country's corporate interests an advantage.
Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.
When Google wanted to operate an uncensored search engine in China, they found themselves hacked.
It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.
There are more examples but turnabout is ultimately fair play.
I mean so has the US... look at all the trade deals that has happened over the last 20, 30... 50 years between the US and the rest of the world. There has always been pressure to help the American corporations, be it copyright law, banning encryption, or outright pressuring other countries not to invest in their own tech and instead use US companies. Look at the current state of Europe for example; lots of people blame the governments, and while its true they take the decisions, a lot of them where encouraged by the US.
Bans on Chinese open weight models being used in the US and bans on AI chips and semiconductor manufacturing equipment being exported to China are two extremely different things, and it's not inconsistent in any way to oppose one and endorse another.
> bans on AI chips and semiconductor manufacturing equipment being exported
In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.
Unfortunately "incredibly behind" can be said about most countries of the world, many of which had electronics industries that have been destroyed during the nineties of the last century, but not about China.
China is already able to design and fabricate good enough CPUs, GPUs, DDR5 memories and flash memories.
For now, their fabrication costs are significantly higher than those of TSMC, Intel or Samsung, because they have to use expensive workarounds for not having access to EUV lithography.
Nonetheless, where the price does not matter much, like for supercomputers or military technology, China can afford to match or exceed the US capabilities with their internal production. With the high current prices for memories, the Chinese memory companies can obtain excellent profits with their products. Which is why the US congressmen John Moolenaar (R-MI) and George Whitesides (D-CA) have requested a ban on Chinese memories, presumably at the request of Micron, who is scared of the competition.
For now, China is even ahead of Japan, though hopefully that will change soon if the Japanese Rapidus meets its targets.
While Europe is the best in a few niche domains, it has a lot of things that cannot be done within its borders, so it is much less self-sufficient than China in electronics technology.
> Nonetheless, where the price does not matter much, like for supercomputers or military technology, China can afford to match or exceed the US capabilities with their internal production
No they can’t. They are seriously compute constrained and no amount of money will close the gap in the short term. The chip export controls have been incredibly effective in that regard.
I don’t have much input on US companies buying Chinese memory but there are more bottleknecks than just memory.
Their production capacity is constrained, but not so constrained as to prevent them to demonstrate a supercomputer faster than any US supercomputer, made with custom CPUs designed and fabricated in China.
The limited production capacity means that they will not become exporters of CPUs or GPUs any time soon, but it does not mean that they cannot satisfy their internal necessities in any critical sectors. The technology that they use for CPUs and GPUs is about at the level that TSMC was 5 years ago. That may seem much, but there are a lot of people who are quite satisfied with computers older than that, and do not intend to upgrade them soon.
The main consequence of the older technology is a higher power consumption, because otherwise their designs are quite up-to-date in the attainable throughput. Therefore their datacenters must consume more energy, but that is hardly a problem in China.
The chip export controls have only forced them to design and fabricate their own CPUs and GPUs, instead of buying them from USA, and now they have become able to do this.
Without USA forcing them to do this, they might have remained dependent on imports from USA for decades, but now USA has lost the opportunity to ever play this card again. Thus USA has lost a leverage that could have been useful in a real conflict.
Such export controls are effective only for a short time, so they must be used only when there is a clear immediate goal.
China can produce chips at "7nm", that's more like 8-10 years behind. My M1 macbook pro is 6 years old, and that chip was made using a "5nm" process. The advanced US chips are now on "2nm". Explain to me how China is going to produce a computer that beats US supercomputers using 10 year old technology.
Banning hardware is helping China in the long run though. And long is not actually very long. They will beat 'the west' (even though they are all made in the east now) in semiconductors because of this nonsense within 50 years. And also in AI capabilities. Making capable adversaries more competitive and angry is often not really a great strategy.
Nvidia CEO had a viral rant about this, implying they don’t really have a moat against China other than Chinese researchers _prefer_ to stay in the CUDA ecosystem. If you force them out then they are capable of building their own chips and ecosystem which may become the new industry standard outside America. He didn’t say this explicitly due to his role but implied that it can be easier than it looks and it’s more of a resource allocation problem.
Yes of course but the Nvidia CEO is hardly an unbiased observer. He wants to sell to China and is looking for arguments to build his case. Similarly to how Dario isn't an unbiased observer either. So who is? And who has the knowledge to say something meaningful? Universities? Professors?
Jensen Huang is biased but if his interests align with yours (i.e. not wanting to concede the Chinese chip market by banning western exports) then maybe you should listen to him.
Exactly, the letter will make much more sense if they are releasing open weight models and China is distilling their models and keep them in secrets for evil purpose
There are many possible international producers, it doesn't need to be physically transported, every household could near-instantly have a copy, it puts local business at a disadvantage if they can't use the same tools as international, etc.
> If you truly believe that bans don't work, the same applies to hardware too.
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
+1 and this will really be a good move. Open weights models are not going anywhere the focus should now shift to hardware and compute. Compute needs to be optimized for LLMs, currently they are inefficient. Analog computing like ReRAM should get more focus.
That would go against the narrative they built over the years, that open weight models are dangerous because they will end up telling you how to create a biological weapon that will destroy the world
They will never do this because it will reveal way more about their other model architectures than they would ever be willing to do. I mean really why the hell are we mad at a company for not wanting to open source their IP? do you think Meta is going to release their ad targeting algorithms? Or google is going to release their search rank algorithms. i mean these things are multi trillion dollar pieces of IP.
I get Anthropic is being really sleazy and annoying here, but being mad at them for not releasing an open weight model i think is unreasonable.
Certainly not demanding they open source any IP that might give their competitors an edge. Rather I’m saying that pulling a gpt-oss-120b would a good PR move right about now.
From a new "fast" company arising, you can sometimes see the way that before it started up there was nothing but "narrative" and that is what established the initial business model since there was nothing else yet. After some momentum is gained whether there is a pivot or not then the narrative going forward has to be aligned with the now more-well-proven business model.
From his leadership standpoint there are 3 big recommendations right now. That's what this message is all about.
>We should not sell powerful chips or chipmaking equipment to China
Well you and who else?
If there's not already somebody who is compromising the well-being of a nation in exchange for a handful of gold, with the ever-incresing glorification of greed & dishonesty at all costs it's only a matter of time for this one.
>We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process
Wait a minute, what's always been needed by everybody are more compute-efficient processes for everything. I've mentioned this before and it's been a while but back in 1980 it took less than a year to figure out I was going to need other chips that were not regular CPUs if I was going to get the most intelligent response from the silicon on a single square-foot of PCB. At the same time it was obvious you were never going to get far without what they now call "distillation", especially with only kilobytes of memory. Otherwise you would be wasting such stupidly large amounts of memory & storage there was no way you could really call it "intelligent". Now with ML & AI on the rise again there are so many people more deeply immersed than ever, and nothing has really contradicted these basic concepts yet, which have been easily recognizable since like forever.
>All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.
Righteous concept, and I'm always in favor of 100x the amount of testing in general normally done.
But "just" test says it pretty well, and those who are gifted enough to "draw the rest of the owl" freehand can test things the most skillfully until they are blue in the face. It's not going to help if an adversary decides not to test, or to enhance these exact things so it can have some kind of competitive advantage. Amodei does not ignore this and there is a footnote about it.
People realize that some of the elements that are coming to mind were expressed in some fairly early "science-fiction" so all this is nothing new
Still looking for the most intelligent responses I can get, since way before 1980 ;)
Banning hardware and banning software are very different things, and the latter is orders of magnitude easier than the former. Physical and non-physical things are different!
What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
> "defenders have to be right 100% of the time, while attackers only have to be right once"
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
It's correct but defenders also choose where that happens. 100% of the time on the locations and conditions that the defenders choose / allow. As a defender i need to be right 100% of the time, sure, but i can make it so that the things i have to be right about are very well known to me, unknown to others, maybe even extremely unlikely to be to known by others, difficult to get to know, (...). So that saying is true but over simplifies the situation. I know monkey brain likes simple phrase. But monkey not live in savannah anymore. Need to adapt and open mind to complex.
Today’s surface areas are gigantic and many of them will - in a typical company - not be chosen by cybersecurity experts. How do I hide the physical location of an office that offers physical access to the company’s network? How do I hide which OS the company is using? How do I hide the underlying technology of customer-facing systems? How do I hide which SaaS services I use?
This is the same mentality that drives companies to sue cybersecurity researchers for exposing vulnerabilities in their software instead of fixing the software, or to insist on keeping software closed source for "security reasons".
If AI makes finding software vulnerabilities easier, then we should deploy it widely to find as many vulnerabilities as possible and fix them, not bury our heads in the sand and pretend the vulnerabilities don't exist as long as nobody knows about them. That's just the same "security by obscurity" strategy that has been tried and failed time and time again.
> The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
> You are suggesting this isn't correct?
The intuition behind that is applicable only when correctness is stochastic. If you need to be waved in by a security guard, then one fake mustache might be the difference between being granted or denied entry. However, a keypad either works or it doesn't; entering the wrong PIN is guaranteed refusal.
The other breach of that intuition is defense in depth. Secure systems don't generally rely on a single binary trusted/untrusted status; the classified building still locks its interior doors. This is the part that has – in my view temporarily – changed most with frontier models, in that they are much more skilled at chaining together vulnerabilities than previous models (and much faster about it than human experts, even if potentially less skilled). If a system has a latent (0-day) vulnerability 50% of the time, then 10 independent layers would imply a ≈ 1/1000 chance that a critical compromise is possible.
However, these independent layers don't currently happen in practice because it's easier to write insecure code than secure code. With luck, modest discipline, and defensive use of frontier models I think that this gap will narrow with time, in much the same way that it would be plainly crazy to deploy root access via telnet today.
Not really, the only reason I (or any other programmer) haven't ever hacked into a system to make my life easier (not to do bad things) is because it's illegal.
Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
> Only by using the open source GLM-5.2 were they able to survive an attack
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
> The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link.
In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration.
I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers.
> Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
> Why would they not use the best tools at their disposal going forward?
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
> This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
> I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
You and the other are both missing the point. That's not a knowledge failure, it's a failure in how your operation is strategically executing. They were essentially practicing, and what did they learn? Change to sarin and even VX, for which they didn't need AI.
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
Still, somebody heavily funded this thing for too long, and I very much doubt that we don't have the surveillance apparatus in place today for these things to get unchecked.
Stuff is known but not acted upon for various reasons.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
> In cybersecurity, a level playing field favors the attacker
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
> Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
It took me a few hours to find some very questionable communities, which in turn gave me access to:
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
I think you are trying to argue that you can limit the open models.
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
You can at least make it harder by requiring US clouds to only serve models with guardrails, and encouraging other countries to do the same. But yes, the underlying issue is the models being open in the first place. I'm sure if the US wanted to, it could come to some agreement with China about this.
I feel like so many people miss what you are saying here. The attackers are at such an advantage because of time. At t0, attackers can go and try and find so many attack angles. These traditional companies (defenders) can't just go to a model and say "fix all my things!" and ship it, way more complex in practice.
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government.
The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes.
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Every single software engineer in the industry agrees with you.
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
True. It has been a race to the bottom for lowest cost as long as the quality meets the bare minimum. LLMs can go through and find all the nails sticking out pretty easily.
> what should be done about open weight bioweapon and cyber-offense capabilities?
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
> what should be done about open weight bioweapon and cyber-offense capabilities?
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?
The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
You can go download the smallpox genome. You don't need AI to do that. The problem of creating bioweapons remains that it requires very, very meticulous lab work under careful conditions, and a lot of experimental knowledge that the bioweapons facilities probably have but LLMs do not. There isn't an actual mechanism here by which you can mix together a few test tubes and come up with a killer virus.
So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?
Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.
The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say:
I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.
Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.
I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc).
This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.
Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.
At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.
Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.
It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.
Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.
People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.
No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
> I had to scroll past so many cynical cheap shots
I haven't read cynical comments, just ones pointing out that the article is cynical itself.
> addresses the core point
No it doesn't address anything, it is fear mongering question.
> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns
Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.
> you need to suggest an alternative plan that addresses these concerns
I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.
Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]
> what should be done about open weight bioweapon and cyber-offense capabilities?
In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI.
When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer.
These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible.
If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood.
The important part is that with the closed, black box, proprietary models, one can never know what they are being served.
All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.
The gap between China's chip manufacturing capabilities and the USA's is only going to shrink, right? ASML obeys some export controls for their most sophisticated machines, but those machines are in China's backyard (Taiwan).
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
Basically we shouldn’t ban open-weights models but we shouldn’t allow them to become as good as the frontier models because china bad. And let’s not have someone else be able to produce a frontier model.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
Why has "open weights" become synonymous with "Chinese" in the first place? To me, that is the problem. I also prefer US models. But I want there to be competitive open weights models too. Those aren't actually incompatible preferences...
> Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
I'd guess that's it, some kind of enforceable KYC. They'd only serve tokens to entities with a legally liable ID (as in, someone to sue, that would cost the defendant something nontrivial beyond a burner account or whatever.)
Distillation has to be way more energy efficient and beneficial for the planet. But if they can figure out a way to ban it, go ahead, that’s not a regulation problem, it’s an Anthropic problem.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
I am so surprised of Dario's inclination for centralization that obviously makes unsustainable and overleveraged governance systems. There is definitely mismatches over the principle of distribution of power...
It's refreshing to see how there's almost no person in this thread who can't see the BS. All the goodwill that Anthropic could have had is basically gone. Anthropic is likely on the path of becoming the most hated company in the world.
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
Ironic to oppose giving AI tech to “authoritarian governments” while approvingly quoting the authoritarian-wannabe government of the US and framing that government as the good guys.
> The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
It absolutely does make China better relatively, i.e. by comparison to the US.
Many of the criticisms previously leveled at China are now similarly applicable to the US in a way that they weren’t previously. Human rights violations? The US is currently the major global supporter of an ongoing genocide, and it even kills and deports its own citizens for political reasons. Political opponents are investigated by the state. When it comes to wars and other interference with other countries - like kidnapping a president - the US is far worse than China at the moment.
Finally, some sense. This is the only argument I have seen that genuinely engages with the problem and approaches it with humility, rather than charging ahead on the basis of assumptions and without a shred of evidence. OpenAI should have been the one making it.
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
> rather than charging ahead on the basis of assumptions and without a shred of evidence.
Anthropic's basis of assumption is the insinuation that LLMs can do things that we've never seen before, and that they can't tell us what it is. It sounds like you're also siding with an organization that has no evidence and relies on validating their own assumptions.
distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.
It's so ironic to me the way people will say "china should not have these chips" meanwhile they manufacture like 99% of all the electronics we have in the united states.
Of all the electronics, but not of the only electronics that actually matters -- the GPUs that models get trained on. China has been trying to develop their own for a while now but they're still generations behind.
Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.
- You cannot directly execute a remotely-hosted program.
- You cannot run inference on an API-served model.
---
Closed-source:
- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.
---
Truly open:
- You can freely modify the source and produce new binaries.
- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.
---
To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.
The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.
> Closed-source: You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
Time have changed. This should be:
Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.
Which is why we need the ability to train our own models. Maybe it will be viable to do it in a distributed computing setup one day. Research's already being done in that direction.
> we have no access to the source materials and cannot reproduce the model independently.
Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.
Distillation of this kind isn't the only data that is fed into models, it's still a small minority. It might be weighed higher in training to learn the thinking patterns/etc but you need a lot more tokens.
Do you expect any of the labs to have an accompanying data dump with: here’s every book ever written, newspaper article, song lyric, Disney movie, GitHub repo, etc. Oh, and we obviously never paid for any of this.
Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.
Yeah some real main character energy from Dario as usual.
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
There is no good imperialist power, there is no capitalism with a human face, there will never be. The UK murdered millions in India, Ireland and all around the world in their time. The US did the same. The only reason China isn't openly doing the same right now is because they are still the underdog and they still need cordiality to get through the door, just like the UK and the US did in their time. The solution is to rid the world of imperialism and capitalism as a whole. As long as there is a profit motive running through everything, international relations will be in the form of wars (military or economic). Only under socialism can the people of the world truly pool their resources together to build up instead of destroying each other.
There are no good imperialists but it's a spectrum where western capitalist colonies fare much better than eastern communist colonies because capitalism and democracy is the engine for economic growth and western colonies fared much better than eastern ones. What's a better system than capitalism for imperialism? Socialism? Weren't they also pretty brutal imperialists? Japanese imperialists alone make western ones look like amateurs.
> western colonies fared much better than eastern ones.
Sure, Iraq and Afghanistan definitely benefited a lot from being bombed, occupied, and then handed over to even worse tyrants when the West got bored.
No, sorry, but "capitalism and democracy" is not the right answer everywhere, and when they're not, pushing them by force is no better than forcing communism. In general, forcing an incompatible ideology on people historically and culturally opposed to it will end in tragedy, no matter how great that ideology may be.
You would have to eradicate the desire to command other people. The wish for power which is different from the wish to be praised, as forcing somebody to do something is different from convincing somebody to do something.
The average German in Nazi Germany probably didnt think the Nazis were too bad either (of course the average citizen wasn't Jewish, gay, communist or other undesirables). That the average Chinese think things are “just fine” doesn’t mean they are.
Unlike Nazis, I don't see that Chinese communists have expansionist goals to seize territory. Nazis wanted all of Europe. Do Chinese communists want all of say southeast Asia?
To be clear I don't think the government of China is as bad as the Nazis were. I'm just saying "the average Chinese in China thinks it's fine" doesn't mean it's fine.
I find it unlikely to be a duoply of hegemony for long, some countries to watch are Germany, Japan, India, Nigeria, and Brazil. You could broaden the geography to continents were I expect major players to emerge on each.
We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?
Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.
A lot of AI researchers were just in Brazil due to ICLR. A number of them got robbed in broad daylight in this supposedly nice part of Rio. I think the assesment of Brazil is accurate.
All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.
I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.
Timothy Snyder, in On Tyranny, has a chapter about staying in touch with friends from other countries, and while we are not friends, this thread is in that spirit and I have learned things from you, so thank you!
Yeah for the ones who get colonized, western ones were better than eastern ones. Japanese ones would've made Britishers look like kids when they were occupying eastern India.
Even with all the downsides of being colonized by Britain, it was still far better than being colonized by Belgium. Heck, being colonized by pretty much anybody but Belgium was better than being colonized by Belgium. Read up on the Belgian Congo sometime: it was a parade of horrors.
GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.
You just put Russia and Japan as "Eastern"; they have nothing to do with each (and I would consider both as more Western than Eastern regarding imperial/colonial matters) other nor with China. China is a massive country that has always been massive and rich. Their style is more like the tribute systems they had before, as it is being observed in the BRI and other projects.
I understand China but in no way were japan and Russia close to western. IJA, MAO and soviet union makes westerners look like chuds. Russians still have the wagner group shit which is nowhere near what western countries have currently.
First, sadly, the west is full of similar examples, like Leopold II in Congo, Nazi Germany, the Bengal and Irish famines in the British empire, transatlantic slave trade... Not to talk about similar genocides perpetrated by third nations but with the support of the west like the Indonesian mass killings of 1965-66. I see quite a lot of similarities. The West is not a moral example of anything at all, but the opposite.
But second, Russia is not Eastern, it was more European than anything else until the Soviet Union. And Japanese Empire drew inspiration in the Western nations, even if they took it a even more horrible twist. So blaming this on "the East" is racist and reductionist.
Western countries have good internal competition that the atrocities they commit can be stopped by themselves whereas middle eastern/eastern ones needed external support for reformation famously hiroshima, deng reforms, soviet dissolution, korean war.. west is superior in terms of liberalism and capitalism that any successful country you can think of now has roots in utilising its ideas.. this is not to downplay atrocities committed by westerners but to think that somehow western nations are worse than eastern is being in denial. Russia was more european before the communists started their revolution and made themselves anti-west. They support many kinds of anti western movements like their war crimes in Africa, support for iranian/Islamist regimes, support dictatorship of china etc. these are all happening right now.. you are talking as though churchil wantedly cause famine in India when there's limited resource and he had to take care of his country before shipping to India and there was some amount of corruption too in Bengal at that time.. I'm not saying that they're blame free but it's a complex issue with multiple causes..
even the AI that was trained on mostly using western data and infrastructure that chinese models distill and all..
I did not say Eastern are better, but rather that they are not worse. And having good economies (that started with colonization, slavery and imperialism) has nothing to do with morals, rather the opposite. You can be extremelly efficient at exploiting people. Western genocides are complex, but Eastern ones are pure evil? That is what I mean, this is just racism.
Single counterexample for single counterexample: Belgium's Leopold - not sure if Western colonialism is so much better (and notice I didn't even bring up "western India")
Not so much better but better in general compared to the middle eastern/eastern colonialism.. westerners have good competiton that the atrocities will be self corrected by itself but middle eastern/eastern colonies needed external western support like Korean war, hiroshima/nagasaki.. main example I can think of opposite is china under deng but there too deng had become relatively capitalist to fix the issues.
Yeah I saw this in Cambodia when I lived there. Sihanoukville had become a Chinese enclave almost completely.
Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.
There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.
You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.
I'm kinda "both can be true, but either are better than how we did it"
We don’t want anyone to colonize us just like you don’t want your country to be colonized. China offers business opportunities while the USA and Europe seem to always look down on everyone else and impose their own values on others. China does not give a shit about how we run our countries as long as they can trust that deals will be honored. We don’t want to be taught how great your democracy is or whatever, we want our countries to become better by our own efforts, with fair deals when dealing with other nations.
Yes, of course, sorry for the expression. I just mirrored the expression used in the parent comment. What I mean is that the US have caused a lot of pain in Latin America.
> Reversing last year’s trend, a slim majority of ASEAN respondents selected China (52.0%) over the US (48.0%) if the region were forced to align itself with one of the two strategic rivals.
I’ve only been to Vietnam but the answer there is it’s… complicated.
They’re culturally part of the sinosphere but of course constantly living in the shadow of your much, much bigger neighbour to the north does breed some ill feeling.
The USA on the other hand, well, they’re not particularly popular either for obvious reasons
The irony is much of the China-ASEA acrimony seems to be over China's aggressive territorial claims in the South China Sea.
Which was a dumb move because, given Chinese economic superiority, they could have just quietly negotiated 99 year military base leases and oil/gas extraction with the UN-recognized territorial owners.
Same outcome, less bad blood.
Seems a bit of an own goal to militarily force the issue and antagonize its neighbors, who it's trying to convert to its sphere of influence.
Yes the whole "wolf warrior diplomacy" stuff in the late 2010s was massively counterproductive for China and just pushed everyone else in the pacific back into the arms of the Americans.
The mask (if there ever was one) slipped and they revealed a lot of information they probably should have kept hidden about their intentions for no real reason.
However they did eventually realise this, they've completely changed tack and they're having a lot more success (helped by the US adopting their own failed policy).
You're right though, I'd imagine politicians in SEA countries haven't forgotten
They've mostly been concerned with resource extraction rather than colonial exploitation like the West. And in fact, they've ditched most of their infra and tech projects in Africa, because of inherent instability in dealing with tinpot dictatorships.
On the other hand, every one in Asia is wary of too much Chinese presence and influence.
Africans are more indebted to Western multilateral organizations and private Western bondholders than to the Chinese.
The idea that Africans are being finessed by the Chinese is racist and stems from a "we know what's good for you" imperialistic lens.
I keep hearing this and I fail to see any reason to believe that it will be the case. Any empire, in the history of our species, has always had an initial "inward-looking" period prior to becoming a full-fledged imperialist oppressor. Edo-period Japan into Imperial Japan, for example. Rather, what I think any budding empire needs a period of growth to become a large enough fish and also to delude its leadership and population into the convenient mindset that their flavour of imperialism is good and justified ("Hakkou ichiu", "The white man's burden", etc.).
To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.
I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.
You may well be right, and it's really only that China has been more inward-looking recently, and given the chance it will spread its true colonial wings.
Not sure if "colonial" is the right word though, as we should be careful to think that oppression always takes on the form we have seen in the past (US imperialist oppression for example did not take on the form of the colonialism that preceded it) . Also, be careful with "true" there. I do not think this is some sort of subterfuge, but rather an inherent weakness in us as a species. Vest power in anyone, and before long their morals will give way and an oppressor will be born.
The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.
So let’s just guess what China will do once it becomes dominant and act like it already did whatever we conjured up they will do. That’s how you get preventive wars that destroy civilizations without any actual basis on reality.
No, I do not think that is a fair portrayal of my position. Just like I will not say that your position is simply appeasement and hoping for the best. One must always be charitable in a discussion with strangers whose positions you do not fully understand.
Rather, I think we should look hard into ourselves and what is good and bad about the current world order. For example, the people of the RoC have the right to determine which direction they want to go. Regardless of the chauvinistic rhetoric coming out of Beijing. We should all stand up for this, because it is a universal right that we want everyone to enjoy. Similarly, we should push against the Eleven-dash line and support the 2013 ruling. The list goes on and I am sure these issues can be resolved without an outright war if we are careful, yet firm, in our beliefs and also diplomatically preemptive and thoughtful.
Being concerned about PRC imperialism should not be mistaken for the position that their people should "know their place" and be suppressed back to the stone age. They have the right to enjoy the fruits of their labour and pursue happiness, just like everyone else. We simply must be there to remind them (just like we must remind ourselves) that the course of humanity is a collective project if we are to stay clear of the darker sides of our nature as we venture together into the future.
Do you believe the UK has the right to rule over the Maldives?
And the USA has the right to have a military presence in Guam (and nearly all of the Pacific Ocean for that matter)?
The Chinese claims on the South Pacific Islands seem really similar to me.
Taiwan seems like a wholy different matter. It was united with China for hundreds of years until the Japanese colonialists took over. It united again with China after WWII but split up after a few years because of the Chinese civil war (notice it was an internal war). I think it's just fair that China wants to re-unite with Taiwan, though I definitely don't support a military takeover. Hopefully a solution similar to what was done in Hong Kong can be found. The Tibet region had a similar history and it's definitely unfortunate that China had to use military force to bring it under its own control (arguably completely unnecessarily - China would be just as strong today without it), but it's kind of understandable in the context of the time (China was trying to recover from centuries of being preyed on by other nations).
I am saying this because I can't agree that China is acting imperialistic - it's basically claiming sovereignty over its own historical lands - which were taken away from them by force by foreign colonial powers. But I admit that, if you go back far enough, nearly all land was once taken over by aggressors - the USA being just a more recent example of that.
Anyway, thanks for not being an absolutist and trying to understand the "other" side (I must acknowledge I have no relation to China whatsoever, in fact I am from South America and live in Europe).
> Hopefully a solution similar to what was done in Hong Kong can be found.
Citizens of Hong Kong lost their right to free speech and their ability to select their own leaders. If you publicly criticize Xi Jinping in Hong Kong you will go to jail. If you advocate democratic rights in Hong Kong you will go to jail.
China's claims on Taiwan are ethno-nationalist. Ethno-nationalism should be rejected in all forms because it is a rejection of fundamental human rights. Taiwan deserves the world's support because it is functioning democracy. That makes all the difference.
I do not have a stance on the Maldives, Guam, etc. as I lack enough historical and current context. What I always do is try to derive positions based on the human right to self-determination.
About Taiwan. I find the claim that since an absolute monarchy controlled the island 150 years ago, that then a government which was the result of two (is my count correct?) revolutions overthrowing that monarchy and then another government, a government which failed to conquer the land by military means by 1950, and now after people have lived independently for over 75 years (over 25 of which as a democracy) that said government has any right to dictate how said people should live to be simply absurd. If the people of the RoC wants to join with the PRC, that is for them to decide through their own decision processes. Historical claims like this may make sense for unpopulated tracts of land, but here we are talking about the rights to self-determination of more than twenty million people of which the vast majority were born well after 1950. This would set a terrible precedence and, frankly, it feels akin to how emperors and kings of old asserted their "rights" and not how we move towards a more just world.
Also, Hong Kong? If anything, Hong Kong shows that the PRC is a poor custodian for a pluralistic country with multiple parallel systems. I once thought it reasonable for Hong Kong to be "returned" after the historical travesty that were the Opium Wars, but I have heard enough first-hand accounts of the suffering and tragedy that unfolded over the last ten years to reconsider whether I prioritise history over the people that are alive here and now. It was not that Mao and Xi "unfortunately had to use force" against Tibet and Hong Kong. These were calculated choices on their part and history shall judge them the same way we judge any other oppressor for their moral failures.
> the West could retaliate by halting shipments of
China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.
I was led to believe that the US does have internal sources of these, but they are largely undeveloped. For years the processing could not economically compete with China so shutdown.
Which is to say, given internal subsidies, the US could eventually produce some on its own.
China controls way more things, from medical needles to pharmaceutical ingredient.
Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?
Oh. I thought a whole lot of the point and discussion of this post was AI being potentially weaponised and becoming means of economic and military control and coercion. Or did I really miss something?
Yes and no. Yes because it's more about oil than AI, no because AI is the oil industry and their supporting vested financial interests' (not so secret) plan to keep demand for fossil fuel high if/when the climate change deniers lose.
"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."
Dario is more of a threat to the US, in terms of advancements in AI, than China. In Dario's mind anything that can't be controlled competitively is a threat to Anthropic, so he positions his FUD strawman so that Dario doesn't have to worry about the competition. And then he can artificially inflate token costs so his IPO can happen. Dario doesn't actually care about ethics, alignment or availability of LLMs - he just likes to use those words to sound like he does. Yet we've all seen how Anthropic actually acts vs what they say.
The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.
But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).
The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.
> And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
Exactly. The cries in favor of distillation regulation from the US AI companies ring hollow and fearful.
OpenAI and Anthropic didn't realize that distillation was going to be so (a) effective and (b) un-technically-stoppable at scale.
Now they're seeing their IPOs at risk and clutching at governmental straws.
Dario's argument is transparently working backwards from {protect Anthropic's economic model} <- {need government regulation} <- {justify government regulation via AI fears} <- {we love open models, but so sorry they can't pass regulation}.
If the rise of the web in the 90s taught us anything, it should have been that companies that take economic reality as it exists thrive, while those that predicate their value on regulation fail.
If distillation at scale works and is technically feasible? That's reality. Deal with it.
Don't confuse political limitations and lack of a theory of victory with the inability to achieve a goal. A blockade needs ships those ships can be destroyed.
It's baffling that they thought the mental gymnastics in this blog post would make them look better. I'd rather they simply fall silent on the issue; I would respect them more (or at all) for it. Open models obviously threaten fierce competition, if not outright destruction of their bottom line. But no, they needed to try and argue that they have the moral high ground for attempting to singularly consolidate power over all human labor.
Look, Big Tech has lost almost a trillion dollars in valuation in a SINGLE DAY. A few more of these downturns and the entire A.I. revolution will be stopped dead in its tracks and we won't have to worry about safety checks, DRAM shortage or open-weight models anymore.
> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.
Absurd? Who knows. As someone who's actually dissected human brains with his own gloved hands, I've never been able to convince myself that they're anything other than biological machines, not dissimilar to these digital collections of weights. Only empathy for my fellow humans prevented me from retreating back into solipsism, and I don't find it at all difficult to make the exact same logical leap for AI.
There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.
There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.
There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.
Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.
No doubt there are many other fates that escape my feeble attempts at foresight...
You will soon have your God,
and you will make it
with your own hands.
-- Morpheus, Deus Ex
If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?
While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.
Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.
Why is it relevant what those outside would say about the US? Or what those outside of any country would say about any country. The job of a country is to do what is best for it. It's better for the US to not help China develop AI that surpasses its own capabilities. It's better for China to try anyway. There are many countries inventing nothing with loud opinions. Those opinions are completely irrelevant to China or the US.
> The job of a country is to do what is best for it.
Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?
Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?
I think that that "best" is not always measured in money. And it should not be. Countries are made of people, and people wellbeing should be the main priority of country. And when stepping a bit back, all countries are made of people, so why differentiate? Ideally make life good for all people.
You mean to say that without France’s help the “revolutionary war” would just be an embarrassing string of failures and a footnote in Britain’s history?
Oh no, you heretic, The People freed themselves and it was ordained by God, you see. It is through the righteous might of The Greatest Experiment in The History of the World that they showed The World what Freedom really meant.
Having the British lose the US was a good thing for France, simillarly preventing Russia from just taking over another country is for the best of other countries.
While there is no "job" of a country, it is natural that each one will work for its own best interest and any moves it makes in the global world is due to their own vested interests in some way including helping Haiti, France helpong Us and the world supporting Ukraine.
Why would a country act against its interest or just randomly? Alliances and corporation are a part of politics
>Why would a country act against its interest or just randomly?
Because country as an entity is a mental construction for which "interest" is a categorical error. It’s certainly a useful concept, but pretending it has interests like some human individual can have have interests. Sure it can serve as rhetorical facility to sell some arguments.
Countries don’t have interests. Some people willing to take control of other people encompassed in that "country" groups have interests, and they don’t necessarily align with best interests of everyone or even majority in each of these groups.
I mostly agree, but there is some limit to that line of argument: a corporation is also a human, social concept, and we do accept that they act in their own interests (meaning they do have interests, sort of like humans do)
Well, you might agree with that, but on my side corporation are pure legal fictions, they don’t have have any intrinsics will and interest. That doesn’t mean they don’t have any existence, just that that there are have a different ontological status from a human person. Just like novel fictional characters, the fact they don’t map to any actual entity doesn’t mean they can’t entice affects they are supposed to convey between narrators and readers.
Why does it feel like you think you're talking to only people in the US?
There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.
Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.
All my adult life, China has given me cheap material goods. Whatever they do in their country is their own problem. They’ve never interfered in my politics or started any wars
> They’ve never interfered in my politics or started any wars
But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.
They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.
Repression of their own citizens is another subject. It is of course highly problematic and should be condemned as much as possible, but in the same way that if the US decided to repress mormons violently, it remains internal affairs and not an imperialistic threat to the countries around them.
I imagine they are better off and more developed than any of the Muslim regions the US has bombed the ever living crap of continuously over the past 50 years...
So I don't support the re-education camps that happened in Xinjiang. Exactly what happened here is still a bit of an unknown because unfortunately there's a ton of misinformation. Whatever did happen, you can go see yourself. It's not a closed region. There are a ton of Youtube videos of people going.
But I hate this talking point for one primary reason: almost nobody who brings it up actually cares about the Uyghurs. It's just a talkijng point.
How do I know this? Because of Palestine. If Benjamin Netanyahu turned around tomorrow and treated Palestinians as Uyghurs were and are treated and developed the region like China does today then he would win the Nobel Peace Prize and Palestinians would be unquestionably better off. It wouldn't be sufficient mind you.
Israel's crimes in Palestine are America's crimes because Israel could not exist without the economic and military support of the US, political cover in the UN and international community and the political will of Us domestically. It would collapse tomorrow if the US withdraw support.
So I don't want to hear a thing from China hawks and skeptics about Xinjiang unless they're louder what Israel is doing to the region.
> But I hate this talking point for one primary reason: almost nobody who brings it up actually cares about the Uyghurs. It's just a talkijng point.
Exactly, it's just people parroting psyop narrative talking points, Taiwan is another. Repeat something enough and people will eventually believe it.
There's a reason other nations are choosing China over the US/EU (including Muslim and Turkish ones) - there's no moralizing, political requirements, nor bombs.
Now compare that to what the USA did in the same time frame. Pay special attention to people killed (either directly, or indirectly) outside of their own borders. Who should you be worried about?
For a concrete example, consider e.g. Operation Condor which displaced my own family
> Operation Condor (Spanish: Operación Cóndor; Portuguese: Operação Condor) was a campaign of political repression by the right-wing dictatorships of the Southern Cone of South America, involving intelligence operations, coups, and assassinations of left-wing sympathizers in South America. Operation Condor formally existed from 1975 to 1983. Condor was formally created in November 1975, when Chilean dictator Augusto Pinochet's spy chief, Manuel Contreras, invited 50 intelligence officers from Argentina, Brazil, Bolivia, Chile, Paraguay, and Uruguay to the Army War Academy in Santiago, Chile. The operation was backed by the United States, which financed the covert operations. France is alleged to have collaborated but has denied involvement. The operation ended with the fall of the Argentine junta in 1983.
Ok, let's follow your rules then. If we remove morality, we are left with what? Pure numbers?
If you want to be technically correct China foreign interventions are > 0, indeed. But the scale of US actions is, I would dare to say, at least one magnitude bigger.
An empire should be judged for how they treat their own citizens, their allies and their enemies. And if we do that, maybe China and the US are not so different.
I’m confused. You start off like you’re going to disagree and then end with the same conclusion I was trying to suggest. Keep in mind I was replying to subvenir who said “eh China not bad - they’ve file nothing wrong to me”
Whataboutism is a very useful game to reveal whether people are being honest or not.
If someone is vocally complaining about country A doing some oppressive activity, while ignoring country B doing 100x the same activity, the comparison does not forgive or excuse country A, or make them right or admirable.
But it does allow us to conclude that the person complaining is biased. Either they don't care about the oppressive activity and want to attack country A for some other reason. Or they have a particular fondness for country B and will never accept that it does wrong. Or some variation on these.
Except it’s op posting whataboutism not me - their claim is that China has done nothing wrong to them with the implication that the USA has. I was highlighting that China indeed likely has done the things they’re complaining about.
I'd like you to familiarise yourself with a legal and societal principle we use in civil law countries:
> de minimis non curat praetor
Which basically means judges are not interested in minor disputes. When you add the USA to the mix, China's actions pale in comparison to the point of being negligible, no point in discussing them given the scale of USA's warmongering, overseas and local crimes. "They never interfered in [...]" is an exaggeration which shows the lack of importance of China's crimes when compared to the USA's, that's all.
The "whataboutism" here isn't a deflection strategy, it shows the hypocrisy and correctly minimises the scale of China's actions in comparison to the hypocrites'.
Dont pretend americans dont need to be afraid of american government. look at the state of the place. A lot of them arent exactly thrilled with their current environment, job market, education, prisonsystem, gun laws, food industry, healthcare etc. etc.
the question is valid for anyone who cares for more than their own ass
I see this narration repeated here, but to me it seems Americans are actually one of the most self-cynical nations. Here in Poland we still actually believe in the superiority of the western values of freedom, democracy and individualism. "People roughly like us" vs "insect hivemind" is the simplest choice ever.
Please reconsider this comment. Calling one side superior and the other side insects really doesn't help with the discussion, and dehumanizing a whole swath of people really has dark consequences.
Speak for yourself, in Italy we are also closer to US "culture", but I do see the US as a bigger threat than China, simply because it's closer and it directly affects our politics.
I'm an American, but I do not think the American government being strong helps me or my fellow Americans. The less America can project its cultural and military power on the rest of the world, the better.
There are different kinds of strengths. There was a past American government that was strong enough to stamp out inequality and fight for civil rights. To overturn oppression and give everyone an equal opportunity to succeed. That's a strong American government I could get behind.
Economic inequality is higher than it's ever been in the US. The reforms you're talking about largely happened in the single term of LBJ, I don't think it can be extrapolated to the "US government"
> There was a past American government that was strong enough to stamp out inequality and fight for civil rights.
this is a hilariously naive rewriting of history, the government was very reluctant to make those changes. It's thanks to popular movements that those changes were made.
Given it's an internet forum, and people here can be anyone around the world, I understand your intent but people over index on the threat that ofc feels closer, rather than the threat that's significantly farther away.
But in general if you ask folks in south east Asia you will find they are likely to be more concerned with China than US except this maddening Oil Crisis.
Although I think people now are very afraid/wary of both evils.
The US govt should think why people in all parts of the world including US itself feel just if not more threatened by them, than China and other evils we have floating around.
I would like to hope my American friends didn't vote for this madness or maybe they did I have read DHH's tweets.
Unfortunately, I have to agree with you. As the saying goes: "better the devil you know". China can at least so far be relied upon to not shoot itself in the foot (and take the whole word with it). The same cannot be said of other great powers like the US (under Trump, for more reasons than I'm willing to list here) and Russia (with Putin's completely harebrained invasion of Ukraine). China can prove me wrong any time by invading Taiwan, but I still hope they don't. Of course, if we try to keep them away from advanced chip technology, that would be further motivation to invade (although TSMC is unlikely to survive an invasion - even if it did survive the actual conflict, it would probably be destroyed or evacuated to avoid it falling into Chinese hands? But I don't want to see the hardware prices after such an event).
Yes we do. We also (despite theories stating the contrary) know that the current climate change is caused by us burning too many fossil fuels, that the utility of vaccines preventing deadly diseases outweighs any rare side effects, that airlines are not involved in a secret conspiracy of releasing chemicals (all airlines at the same time and only in certain meteorological conditions), and that the earth is roughly spherical.
I have read the WHO [0] report and the conclusion is "possibly to likely" about the zoonotic origin of Covid, so no there isn't conclusive report how it came to be.
> If there is a cover up it is to hide incompetence.
Which is my point to GP's "reliable" argument. China is not reliable and has its own interests at play just like the US or any other state for that matter, regardless of its international partnerships. It does want to project itself as "reliable" - that I can't deny.
> If China goes to war, it's probably going to be because their economy is in tatters.
Well, they probably also have to factor in whether invading Taiwan is likely to improve that situation, and I would argue that it's not - China is currently exporting stuff all over the world, so economic sanctions would be very painful for them. And there are lots of countries who would like to have a slice of the manufacturing that China is currently hogging...
China has planned out the invasion/reclamation of Taiwan in 2027. They held true to their Hong Kong timeline, not sure why this would be any different.
You should avoid agreeing with the narcissists that always use false equivalence, whataboutism, and goal post shifting the second you call them out.
You are allowing them to define a ridiculous standard by agreeing with them that not only is it in fact “two evils” but what qualifies as evil. They’ll tell you…always only when called out, mind you…all and any meddling (also something they will then define with the same kind of falls equivalence loop) is evils, when that is simply not true, not is any kind of effort to influence the same thing as actual meddling, not us it always evil.
I could go on, but I’m sure you get the point, the narcissistic personality or their little zombies in most cases will constantly shift and change things all to avoid you from being able to get a clear focus on the truth, something their explosively allergic to.
As an American child of the Empire with global perspective and very high access perspective, there is no other entity besides the cabal that controls the USA and acts flying its banner; that is more deadly, more meddling, more conniving, more evil, more supremacist, and more vile.
Nobody said they negate each other. The comment being discussed is fearmongering about China and is very relaxed about the USA. All we're saying is you should be either afraid of both or relaxed about both. China is not worse than the US.
It would be whataboutism if it was a random Joe saying "China is evil" not literally a company that 1) is in the US 2) has a strong influence in the government and legislations affecting them and their competition.
In this case it's just pointing out the propaganda and the contradiction.
They have colonies and generally aspire to become basically the old british empire as a longterm strategy? At least thats what they push with that "century of humiliation" comeback strategy internally..
If you're making a case of this vs what the US has interfered with, it simply does not stack up. Of course it's naive to think they won't start interfering like the US once they gain the leverage, but to a neutral party it likely doesn't matter which of these two ends up as the superpower on the global stage.
The Indian wars were border skirmishes at most. Literally 2,000 soldiers dead on both sides combined by official count. Very few civilian casualties.
The 2020 skirmish was way smaller with single digit casualties. Again, almost no civilian casualties because it's a largely empty region in the first place
You really can't compare this to the kind of wars America has started. Casualties - both civilian and military - stood in the hundreds of thousands
I mean most of Chinese "wars" have been border issues. Most of these border issues themselves stem from badly drawn borders post colonial withdrawal. These are understandable, if not justifiable - countries tug at each other to figure out where their territories start/end. A hostile neighbor at your borders is a legitimate national threat
America's wars have been in countries so far away that they could've never posed any threat whatsoever to America as a nation.
> America's wars have been in countries so far away that they could've never posed any threat whatsoever to America as a nation
That’s fallacious reasoning that ignores how easy travel is and how cheap an attack can cause massive damage. 9/11 was massive attack from countries far away. We regularly see commercial drones being used in warfare causing huge damage and casualties. There was the 2012 terrorist attack on our CIA facilities in Benghazi.
Terrorism entrenched abroad impacts our security. Is the response we take incorrect and makes it worse / are there things we do to provoke this? Some things yes, other things are like having Israel as an ally which is a culturally, socially and economically aligned power in the region even though it’s hated in the region.
Also don’t forget the US is a major maritime power. China is trying to be, Russia is not. Maritime powers force project more broadly because a) is the nature of being a maritime power b) you have to protect shipping to make sure trade routes are uninmpeded. B is particularly important as it also helps explain the wars (eg the Iran war is about the strait and trade and who collects money, not really about nuclear capabilities).
Wow, that newest of Trump's distraction talking points made it surprisingly quickly to the uncontended (?) and commonly accepted facts in HN conversations.
You’re misinformed. The Biden administration in 2020 declassified that Beijing has prepared to latch influence campaigns in 2020 but backed off. December 2023 a report was released that they had changed track and engaged in influence operations during the 2022 midterms. Biden’s administration also warned about influence operations in the 2024 election cycle. The Biden DOJ unsealed multiple indictments against Chinese intelligence officers, state-backed hackers (such as APT41), and illegal Ministry of State Security agents acting inside the U.S.
You’re very misinformed if you think Trump’s latest blathering are completely wrong. I generally don’t pay attention to him so I don’t know exactly how he lied or exaggerated but I’m sure he did. It doesn’t negate the real and active influence operations China is engaged in
You’re very misinformed if you think democrats and republicans arent playing the “election influence fiddle” in exactly the same way; legitimising the elections they won, casting doubt on those they lost, all while achieving the main goal of making sure the population doesnt forget who their so called enemies are.
Lots of assumptions about what I think. I pointed out that Trump put the topic on the agenda a few days ago and now it comes up in HN comments. No more, no less. Whether it has merit is a different question. Point is that the reach of him setting topics goes far which is hard to deny.
At this point with how volatile US is I'd rather have China control Taiwan than US.
And this comment isn't about putting any good light on China. Lately US acts like they want to compensate for Russia fucking up less things around the world
in contrast this is USA's main job is to Interfere in any country it can in favor of its own goals. not related to what the people of that state really want.
north Americans must stay silent about interfering in other states internal issues.
No one who believed it definitely was has changed their mind, they've just at most gotten quiet about it.
No one who believed it definitely was not has changed their mind, they've just at most gotten quiet about it.
No one who believed it could have been but that the claim has not been proven have changed our minds, we've just decided that there's no way to know what actually happened.
Tibet was part of China since the Quing dynasty in the 17th century, it briefly declared independence during the chaos of the Chinese civil wars, but it was never recognised.
Vietnam is usually recognised as a low point in Chinese history.
There's been an ongoing border dispute between China and India for over 100 years, thanks to some questionable line drawing by the British.
Also, referring to the parent comment, of these only the border skirmish with India was within my lifetime.
"But Tibet!!!" has to be the most intellectually lazy pro-American imperialism argument, particularly when you consider the US history of regime change [1]. Heck, this unwinnable war we're fighting in Iran has a direct through line to not one but two such incidents, specifically the coup in 1953 and US involvement in the Islamic Revolution in 1979 (ie making sure the fundamentalists won instead of the communists). And weirdly a significant number of these puppets end up turning into enemies and far worse problems (eg Saddam Hussein, Osama bin Laden).
Tibet specifically is funny because one of the things China did was end slavery in Tibet [2]. A large percentage of the population were "serfs" but that was a generous translation because these "serfs" could be traded. You know, like property. Like slaves.
And Vietnam? They had a small border dispute. What did the US do to Vietnam? And Cambodia? And Laos?
But the funniest claim of all here is election interference. The foreign interference in US elections conversation begins and ends with Israel with an honorable mention to Russia for running some Facebook ads. Just last night we had a debate in the Democratic primary for Senate in Michigan, a race that AIPAC proxies (eg UDP) have spent upwards of $60 million. On a primary.
What people don't know is that China's "war" against Vietnam in 1979 was actually a pledge of allegiance to the United States, a way of demonstrating its stance.
This event was followed by the longest honeymoon period in Sino-US relations, which eventually lead to China's rise.
I mean them selling the cheap material goods is politics, and they use that a lot to threaten other countries. They constantly bully neighbors and assert claims on foreign terriroty. They might not be the comic book villians people are making them out to be but your take is just ignorant
Nice straw man. The point is not "cheap stuff" but using economic leverage and territorial pressure on others because of the "cheap stuff." And yes, all China does is produce cheap stuff for the world, nothing else at all lol.
They did downplay the effects of covid and spread misinformation about it as well as silence the researches and journalists who were first trying to ring the alarm bells
Imagine you have most expensive disease in human history by economy standpoint and after 6 years we have just claims of where it came from. Isn't that suspicious enough?
Yes, I remember: "Vector" BSL4 Lab, Novosibirsk, Russian Federation, Sep 16 2019. Fire blast, then unequipped military first responders which stole lab equipment, then joint military training with Wuhan military police few weeks later.
Oh i bet they have not directly interfered with your politics, but they do interfer a lots in various things. More than any other country on this planet? I don't know. China is big. Really really big.
Yes and the USA trained and gave weapons to the Taliban and many other terror organizations. "China sold none weapons to another country" really pales in comparison.
This is wrong. The US did not give weapons to the Taliban (voluntarily). If you mean Operation Cyclone their receiver was not the Taliban but the mujahadeen, which, yes, consisted of Taliban, but also of enemies of the taliban (northern alliance)
> Whatever they do in their country is their own problem.
Idk to me it feels like human compassion and ethics dictates that you at least have to care a bit even about the things in other countries. Otherwise all sorts of horrible domestic policies would be justifiable.
> They’ve never interfered in my politics or started any wars
They actively back russia and deliver them weapons and support them with their attempted Genocide in Ukraine. They also actively threaten Taiwan.
Not to mention them claiming almost the entire South China Sea even though this being against international law and threatening smaller nations into submitting to them
The article says otherwise. It opens with it isn’t genocide but is crimes against humanity.
The U.S. State Department’s Office of the Legal Advisor concluded earlier this year that China’s mass imprisonment and forced labor of ethnic Uighurs in Xinjiang amounts to crimes against humanity—but there was insufficient evidence to prove genocide, placing the United States’ top diplomatic lawyers at odds with both the Trump and Biden administrations, according to three former and current U.S. officials.
That's a very selfish way to think of it. Whatever they do in their country does not stay in their country because China is becoming a superpower so you should expect it (as it's normal) to interfere and affect policy abroad.
For example Lithuania has been punished for letting Taiwan opening an embassy (in Lithuania). So China cares what other countries do in their country. Without EU support Lithuania could have faced very harsh consequences. That's just an example. Another example is the "secret police" stations undeclared overseas police stations operated by China in various countries (i.e. U.S for example).
China gave you cheap material goods because that was its business but you have to keep in mind it's an authoritarian, communist regime. It carries an extra risk on top of the potential economic coercion if you give it too power. At least you know that U.S is only after the money and does not want to turn your country in a communist "utopia".
That being said this does not mean we should ban Chinese AI models because China didn't cross any red lines(yet) compared with Russia for example.
That looks like to support my point. The US is just for the money. It’s not like they brought them communism or whatever Venezuela has there. They simply said they don’t care about “spreading” democracy anymore as long as the regime plays ball. I think they did that anyway in the past but at least it was not the “official” policy.
China is becoming an intellectual powerhouse in addition to a manufacturing powerhouse and runs huge spyware and propaganda campaigns.
Read up about T95, MBOX, TVBOX, or other Android TV Boxes advertised as generic TV streaming devices advertised as "unlocked" or capable of accessing free content. They're loaded with spyware/malware, typically BADBOX. Some of them will record audio and record your network traffic and send it to China.
I'm also convinced that the anti-education thread winding its way through US culture is being amplified by China. They likely didn't start it, as it's been growing for decades, but China will happily keep it going.
I think the fair nuance here is, an administration which used Executive Orders to force guardrails, meaning US companies must retain them, even if they might want to drop them now.
And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.
So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.
I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
It's a genie that's not going back in the bottle, the bottle is smashed.
The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.
Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.
The "child pornography" thing was about image generation, which people for better or worse have very different moral standards for than for text generation. There is very little pushback against grok being willing to write explicit descriptions of sex, or giving security advise.
Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)
It also must be seen in the context that the open weight models aren't commercially associated with a popular distribution site where people share the images, the people behind them aren't public figures seen as encouraging the use of the model for "undressing" (albeit in a funny way not related to minors), and they haven't yet responded to questions about what can be produced using their tool by proposing guardrails but only for non-paying users...
Open weight models get scrutinised in a different way, also linked to perceptions of their developers' bias, like the tests to see whether they refuse to answer questions on certain historical events at Tiananmen Square
Well... They are all part of a "high well regarded" group of people of a security committee that has no specialists or whatsoever, only trillionaires on a round table
> And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
You're vastly underestimating the scaling problem here.
Things that would need to be true for your statement to be valid:
- Model intelligence doesn't increase with model size
- Model intelligence doesn't increase with training set size
- Novel architecture completely decouples model intelligence scaling from hardware scaling
- Residential electricity is as cheap as industrial electricity
- RAM and GPU supply outpace demand
People call grok that because deviants were abusing grok's ability to edit images and post them publicly on X to strip people - including children - of their clothes, from their public photos. Then when there was backlash, Elon laughed it off. It took half the world opening investigations against X for violations of existing regulations for action to be taken.
The leniency that internet companies get in terms of dealing with illegal content comes with the expectation that they're making reasonable efforts to control the distribution of said content. X, and Grok, were actively supporting the production and distribution of the content in public.
It is very different from someone creating such images in a private account, and definitely very different from someone using a local model to do it.
Then when there was backlash, Elon laughed it off.
My entire post was how it is unreasonable to have a dual standard, and my point was it's really irrelevant if it's a model you download and use locally, or if it's a model hosted remotely, or hosted and created remotely. You're not really providing any sensible reason where the line is, except "public company", which is, again, the entire point I'm making.
The only realistic, non-double standard is that the creator of the model should be 100% responsible. What on earth does it have to do with who's hosting it?
And by this metric, aren't all the uncensored models on huggingface, child pornographers? And if so, why not? Provide tangible, real, sensible reasons please, and after all, isn't hugging face a company?
You know, people are all over the place on this. I see people complaining about guardrails, then in the next breath complaining there aren't enough. Complaining that open models are the thing, but then creating double standards.
So once again, what is your actual reason why it's different?
lawyers cited a 2026 National Center for Missing & Exploited Children (NCMEC) report confirming that 90 percent of xAI’s CyberTipline reports “were not actionable by law enforcement because xAI declined to include user information that would allow law enforcement to track and locate perpetrators.”
I'm very confident that it was staged and coordinated.
This propaganda started because they cannot evolve their models further.
See Fable and Sol, they are lame. They seem incredible at first but the more you use you can see the trickery.
It is a matter of time for someone to prove they are marginally better only because they inject more information to the harness at server side.
Live overflow released the video today about it. It is very compelling but I guess there are holes on their hypothesis as well.
The summary is that an instance was running on certain benchmarks without any limits or supervision and the AI decided to cheat by exploiting a silly series of vunlns.
The public narrative around conspiracy theories is baffling for sure: what rational basis is there for assuming them wrong? Often none.
Incentive and ability are what should be looked at. There, things get far more interesting: what is the current state of AI employed by the US intelligence agencies and what do they use it for?
Having the public convinced, their "superiors" would only do everything in their best interest, even without anybody knowing for sure, is Huxley's Brave New World in real life.
that's some real anti-truth you've got going on there. because we shouldn't assume that others are acting in our best interest, that any random assertion that there is collusion against us should be by default accepted as the truth.
shouldn't any reasonable person when presented with a line of thought that has no substantiation at all conclude that they just can't reasonably be expected to support or reject the theory?
Everything in this space is manufactured, the plebs is fed very deliberate information for manipulation. I don't think there is a debate about it. All the PR stuff is marketing.
Worth checking, elsewhere in this thread someone points out glm only assessed the damage after the fact, it didn't stop the attack, and Hf apprently never sought access to a trusted defender program with a closed model either the open model saved them farming might not hold up.
>Also everything hes saying about China (and other actors) many outside the USA would say about the USA.
Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.
Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.
Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.
There are many in Europe who feel the same. The tide is very definitely turning.
As someone who has never visited China or the US; yes. Why? Media. School shootings, ICE, crazy president…
What do I see from China? Impressive multi-million cities… and true to be told, not much.
China had a worse reputation, but not anymore since Trump leads the US.
School shootings aren't carried out by the government. Crazy president democratically elected and criticised by a free press. Things that you don't like about a country are a lot different than things that will get you and your family disappeared by your government.
His Concern 1 was especially painful to read. It reminds me of Red Fever rhetorics from McCarthy era. He casually mentions CCP and People’s Liberation Army (dogwhistles?) as the most fearful enemy. But he didn’t compare CCP policies to GOP or Trump administration (apple to apple comparison). He doesn’t frame it as Anthropic vs Moonshot competition either. He deliberately frames it as the US vs the Communists war.
He portrays the biggest ever threat of AI as the Chinese Military using the models in their drones. He already sells his models to military customers, US and potentially allies, and they are actively used in the field.
So no objective argumentation here. Just Nationalist political rhetorics and FUD. He’s allowed to do that and he will have an audience. But he won’t be taken seriously outside the US. He appears blindsided.
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
> The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips
Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.
Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.
In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?
Regulatory capture 101.
Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).
A lot of safety infrastructure and tools get open sourced so other companies can benefit from it, but there is also a risk if all safety features are open since it can accelerate the cat and mouse game where people work around more clearly defined limits and understand how they are implemented.
Your cynicism will limit your understanding of other perspectives.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?
> Found the person that believes some other random person can use a computer better than a John Carmack could.
Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
> Any takers on the question of who he considers to be the good guys?
Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.
So, maybe not China?
> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
I never said that, but private smaller AI companies are all over the place. He never argued against that.
Apparently, looking at Iraq and Vietnam, US citizens only believe in "personal freedom, freedom of speech, freedom of religion and the value of human life" when it comes to US citizens, not to humans in general.
Even on HN, I saw a lot, that when people discuss surveillance topics, they argue whether ie NSA can spy on Americans, not on everyone. Mostly, nobody even question the human rights of those inferior humans abroad.
Vietnam and Iraq were both unpopular wars in their own ways, so they don't seem like good examples if you're trying to find examples of what US citizens support.
Also, a country surveilling its own citizens has unique and different implications compared to surveilling other countries. Citizens need to be able to influence their own government, but mass control can nullify citizen power entirely which becomes its own problem.
Then, on a US website, you link to another US website which keeps track of various war crimes. Try finding an equivalent website for China in China, or for Iran in Iran. This is part of the asymmetric freedom issue on the internet that many people ignore. There are a lot of lies and propaganda spread within other countries that censor and deny some types of information from even existing, then they use that as a springboard to spread it around the world.
In the US, we can criticize ourselves which is important, since it helps us improve.
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
It's possible for something to be overpriced to the customer and simultaneously underpriced for the business to be profitable. Theoretically speaking, they could just be selling such an inefficient product.
It's also not a law of nature that there be a valid/efficient form of a product that makes a business profitable. I'm in no place to judge whether that's the case here, but not all products are viable.
You would need to demonstrate an impact for a line item that big. A 3k person org at $200/seat would be >$7mill/year. That places you very well into self hosting Kimi K3 territory and the never having: price hikes, dependency on a 3rd party, etc.
Market failure! Market failure is the term you're looking for.
Also see: childcare, healthcare, many forms of public transport and social infrastructure. Some things markets simply cannot deliver well, and that's okay.
Daycare is the most easily reachable example because it's quite simple compared to the others. Daycare workers are simultaneously some of the lowest paid workers in the US, yet daycare costs are famously high and prohibitive, yet childcare centres are very far from money-printing machines. Margins in the daycare sector are most commonly < 1%.
> I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow.
The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.
I'd assume the gp was just inflating their rhetoric but cost of model is an interesting topic.
I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.
The whole thing about replacing workers is such a perfect example of shooting yourself in the foot. The whole US economy is so strong because the population consumes so much. Get rid of the workers, you’re now damaging the consumer base. Repeat that a few times, across industries, and you now have a zombie economy. The idea that we can have an economy of virtual agents is a child idea that doesn’t make any sense in a serious situation
>Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?
The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.
> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.
In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.
It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.
Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.
--
There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.
Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.
And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...
Of course what is Schrödinger's China is the idea that an otherwise adversarial China would simply accept some international ruleset decreed by the US and enforce it on their own territory, without demanding anything in return.
E.g. they might say something like "We absolutely agree that international regulation of AI is needed and we're willing to co-sponser an initiative with the US. One of the most pressing matters we see is the spread of despicable misinformation through unregulated models regarding certain events in China's history, the treatment of certain minorities or the status of certain provinces..."
I agree, although I think the real goal the model providers are going for (both commercial _and_ open weight) is probably power. Just think of the control available to the organisation training the models politicians, business leaders, and citizens are increasingly delegating their thinking to.
Yeah I don’t think they are over-priced, last I checked all those companies are still losing money hand over fist.
I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their
money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment
Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.
Compared to how much they want to pay. This is always what 'overpriced' means, as far as I know. The seller's costs/profits/margins aren't a concern for the buyer.
I was reading the post and thinking "wow, that's pretty clear for a smart man used to writing for other smart men. it'll be really difficult to misunderstand". I read the first couple of comments and stand corrected.
A much simpler summary:
- open models good.
- smart models _can_ be bad
- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.
- banning open models just in US is neither good or bad: is stupid.
Alphafold and others solving the protein folding problem are revolutionary.
An LLM can max provide an instructive tutorial protocol for lab work, but designing novel proteins is not it's job.
I am interested to enter the bioinformatics space eventually and the regulation happens at the DNA printer level. Anyone can design yeast nowadays that excretes heroin, but the DNA needed for the genetic modification won't be printed by anyone. As long as DNA printing tech is extremely regulated it's all fine, if home printing becomes a thing then on the other hand maybe people won't even need AI to print deadly pathogens at all. A database lookup will do.
In short: "We don't propose a ban to open-weight models. we want to stop these models being developed altogether. If there are no models, there'll be nothing to ban".
Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".
Lastly: "What if they use their models in their military and local police services like we do? Communism!"
In the old hardware terminals, and current terminal emulators, ^H (or CTRL+h) is "Backspace signal". It has the effect of pressing backspace.
So, ^H^H^H means "delete three characters, excluding '^H's". Like the person typing the comment changes their mind and deletes the characters (or the word) before writing else.
It's an stylized way of euphemism. i.e.: Actually I want to say this, but I substitute it for that.
Ctrl-H is the rubout/backspace command on old terminals. So if I say it's nuts^H^H^H^Hunwise to ignore UNIX history, I've erased the first word and replaced it with the second.
A caret with a character subtracts 0x40 from its hex value allowing you to insert non-printable characters. Uppercase H is 0x48. That gives us hex 0x08 which is a backspace.
A bit of good old “it’s really good idea, I love it, great effort … BUT”
Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”
Regular distilled models show capacity gaps and overfitting that open-weight models don't anymore I think. This focus on distillation as "more compute-efficient" (i.e. cheaper) seems to rather be an excuse for bad (or bubble) investment, fixed hardware dependency and lack of interest in research of efficient compute. Which also shows as climate and sustainability impact.
Nvidia signed the open-weight model letter and Europe doesn't have better models either, so chips don't seem like the issue either. I guess good old performance optimisation is just not _cool_ anymore. So they use the same argument as politicians arguing "cheap products" are why tariffs are needed; when instead it's mismanagement.
Another HN user wrote the other day "live by the sword, die by the sword".
> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
Very good point. However, if one reads the transcript of the speech that Xi Jinping gave to the World AI Conference on 17 July, we see that he he is very much in favour of AI safety.
> Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others.
Now, let's contrast another important part of safety here. Amodei puts the fact that this will need to be a global effort as a mere note that sure, China will need to help too:
> Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.
International collaboration is the focus of Jinping's speech. But one imagines "cooperation" Amodei has in mind if "do what I say" while Jinping has more collaboration in mind here. (Even if you think 'china bad' they deserve credit for collaboration for their open weight models).
China has yet to demonstrate weaponization of AI. Meanwhile Anthropic was openly part of the Iran war. I wouldn't give much weight to Amodei's words after that regarding controlling AI weaponization.
> ...banning the use of these models by US businesses does nothing to address this risk, because bad actors are unlikely to be legitimate US businesses. It would protect US AI companies from competition, but that has never been my goal.
But on the other, he argues:
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Models that don't pass safety testing would be banned. Darius does not appear to be against banning models. He wants the government to have a regulatory body that has the ability to ban models. Then Anthropic can do regulatory capture of that agency and control what models are permitted to be released.
Also, during this mandatory safety testing, models would be blocked from use, and by the time the testing was done (probably years) the models would be obsolete.
I think the worse possibility his he actually believes his safety bullshit, believes that he and people who think like him are the only ones with the special knowledge required to do safety correctly, and that they're justified in accumulating total power of this market to people who think like them (which just so happens to be Anthropic).
This notion that Dario cares only about the bottom line is simply misinformed, in the most charitable interpretation.
It doesn't track at all with any of his prior stated beliefs or past actions. It's an absurd claim. It's a baseless conspiracy theory, smuggling in traditional conspiracy mechanics for plausibility.
Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks
Crack down on distillation, just for Chinese companies or is it ok for Chinese/US companies to distil? I find it hard to take Anthropic/OpenAI on distillation, because the way see it they started with "distillation" of another kind. They used all the content out there without consent of the creators and its still happening. Model distillation is just a different layer of abstraction, but same thing more or less.
Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
Yeah Dario is just flat out disgusting in term of how shamelessly hypocritical he is.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
The emptiness of AI companies' waxing poetic about the future of humankind is laid bare by simply looking at what they actually do, and who they do business with. Actions speak louder than words, and they've driven the worth of their words into the dirt many times over.
> Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
The country that exists solely because of China? That North Korea?
I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.
No, it exists because 1) China shoved back UN forces from the Yalu and 2) their allies, now in charge of everything north of the 38th parallel, aimed a bunch of artillery at Seoul in case there was ever a thought of the ROK and its allies reunifying the country through force, or any other mechanism the Kim family didn't approve of.
The nuclear part is really just gilding the lily. Of course, China and Russia have helped North Korea circumvent the sanctions that were supposed to punish them for their nuclear program, but it ultimately all goes back to the Chinese support of Kim Il-Sung during the Korean War.
Anthropic's ToS restricts the DoW from using their models for 1. Fully autonomous weapons systems, and 2. Mass domestic surveillance. There is no carve-out regarding who the autonomous weapons systems would be used against.
What does "Fully autonomous" mean? If someone presses a single button to authorize an autonomous drone to enter a theatre and conduct autonomous strikes for 4 hours without any further approval is it truely "fully autonomous?"
Yes. A munition that hangs around waiting for targets and decides whether it should attack without checking with a human. That’s a fully autonomous weapon.
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.) Brad Cooper, head of the US Central Command, has boasted that the military is using AI in Iran to “sift through vast amounts of data in seconds” in order to “make smarter decisions faster than the enemy can react”.
Iran had the courtyard painted in bright pastel pink/blue colors with murals and playground markings to clearly identify it as an elementary school. The Pentagon claimed they had "outdated intelligence data"
I think the fully autonomous weapons systems restriction is not a very serious one and is very easy to circumvent.
What if you can hire a human to push a single approve button? And what if that human's job is to front load approvals by pressing the approve button a few thousand times, every morning?
Yeah, but then responsibility for the consequences of action of the system falls onto person pressing the button. I hope that no one would be willing to take such responsibility without second thought.
What part of placing a human in the loop requires telling that human what they’re approving, or even that they are approving something? Pressing a button could be as simple as following orders—or giving orders.
The Iranian strikes didn’t involve autonomous weapons systems. The “AI” component was used for target selection.
The actual aiming and firing of the weapon was performed by humans. In theory the target selection was also vetted by humans, but humans relying on exactly the same data that resulted in the “AI” systems misidentification of the target.
I put “AI” in scare quotes there, because these systems are really data processing pipelines, rather than LLM style AI systems.
Do you understand that real politics sometimes means you need to doublespeak? You don't get the outcomes you want by speaking your direct intentions. Don't be a fool to think that what someone says is what they believe while playing these high stakes games with actors of differing motives and value systems (the current white house)
So you are saying its fine if Claude is used to bomb school children, and murder people in Venezuela, and spy on the entire world (apart from US citizens), as long as they get to kiss up to the current administration? They could have cut all ties for real, and actually had a back bone, instead they bend over backwards for whatever the current wanabee god king wants. Im sure it has nothing to do with their valuation and trying to become another trillion dollar company.
> So you are saying its fine if Claude is used to bomb school children, and murder people in Venezuela, and spy on the entire world (apart from US citizens), as long as they get to kiss up to the current administration?
It's absolutely not "fine". I'm saying maybe there's a complex region-beta paradox where we can't get to the other side of it unless some actors enter undesirable terrain (by their own measure), don't exit the stage, and play moves they'd prefer not to.
No, he refused the DoW demand to use Anthropic models without limits. But Anthropic still agreed to military usage of their models, including for strike planning.
He was directly asked in a Bloomberg interview whether Claude was used in the bombing of the girl’s school in Iran, and his answer was “We don’t know”.
His redline was autonomous weapons, not the death of 100 innocent girls.
Palantir's Maven system integrates LLM models in decision making for identifying targets. It was using Claude when it identified targets in Iran, including the Minab elementary school
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)
This had nothing much to do with AI in particular though. The issue was that there had previously been a military installation at the same location, but the US military did not update its database. Apparently, one analyst had flagged the issue for review in 2019, but that was never done before the strike.
The point is that the same thing could have happened just as easily if you had selected targets based on the faulty data by any other means than an AI model.
No, the reports specifically mention that the AI target selection pipeline drastically increased the workload and expectations of the humans in the loop, leading to spending far less time on deciding whether to attack, iirc a few seconds per targeting decision.
Of course some overworked analyst is going to start neglecting the details and erring on the side of bomb it if the AI summary is dangerous sounding.
It’s gross negligence to be offloading this type of analysis to claude. Do you decide what’s factual based on the google AI summary? They’re deciding to end thousands of lives with about the same amount of rigor, including those schoolchildren. Have some humanity.
That's no excuse. The US has 161 public schools on military bases, more than any other country.
The Minab elementary school was painted in pastels and had prominent murals on the play area.
If anything, your comment just highlights how much this has to do with AI. When "outdated data" can lead to 168 dead schoolchildren, you can really see the consequences of relying on AI for decision making.
No, only for domestic surveillance and autonomous weapons. It can be used to evaluate targets as long as a human pulls the trigger. But the DoD created a really strange position for itself, they use Claude, but also designed it as a supply chain threat, which should mean they cannot use it
“We are also left with no further information about the potential use of artificial intelligence in the targeting decision. The CEO of Anthropic admitted in an interview with Bloomberg that he does not know if or how their artificial intelligence software may have been used to select targets for U. S. attacks in Iran, but he was quick to shift blame to the military, in particular to an unnamed ‘human’ making final targeting decisions.
You give too much credit. Many think of it as an acceptable cost of waging war, not something to waste regret on. Some few think of it as a positive (they chant death to America over there).
I've never heard anyone say it was a positive. While you're absolutely right that many would say that it was a casualty of war and that we have to expect some amount of mistakes to be made that does not mean there is not regret.
The bombing of the school was horrible, but it was a mistake. I could see essentially the opposite argument: Anthropic should be more involved as to prevent mistakes in the future with better technology. Imagine Claude asking the decision makers at CENTCOM "this looks like a school, are you sure you want this added as a target?"
You don't get to start a completely unprovoked and illegal war of aggression, bomb tons of targets with little care, kill 100 schoolchildren, and then say, "But it was a mistake."
It's a mistake in the sense that if you go and rob a bank while shooting wildly, you only hit the customers "by mistake." That's not an excuse.
The US is bombing both military and civilian targets, by the way. Trump has been very open about this with his talk about "power-plant day." He considers attacking civilian targets to be a legitimate way of pressuring Iran to surrender.
The DoW head sees half of his own country’s population as enemies, Islam as an evil religion of hate, and the work he does as striking down the enemies of God. I heavily doubt that this man gives a single thought to children of muslims.
Hegseth deliberately closed the unit that used to ensure it wont happen. The efforts to avoid civilian causaulities were deemed woke, weak and literally unmanly.
And administration repeatedly threatened to destroy civilian targets.
> The bombing of the school was horrible, but it was a mistake
Please don't say this. It wasn't a mistake.
The very FIRST strike packages of the war are very clearly vetted. If you are involved in military planning, you know this.
They doubled tapped it after seeing people flee inside the save the children.
The goal here was to teach IRGC a lesson and demoralize Iranians, which clearly backfired.
Furthermore, US has been contentiously bombing civilian buildings, bridges, hospitals throughout the war. They have also bombed water desalination plants and other key infrastructure.
You have the US president threatening to use nuclear weapons and end the Persian civilization.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
Yes, I've seen it! It's exciting, but I don't have enough information to say whether they are making "a real go of it". That is, it's unclear to me whether they have the tens to hundreds of billions of dollars necessary to create a frontier model.
I...don't follow. How in the world does your comment about open weight and open source relate? Ignoring that it has nothing to do with this post (did you mean to reply to someone), are you aware that Anthropic, OpenAI and others allow 3rd party inference providers to run their proprietary models? Like, what does this non-sequitur even mean?
You understand Moonshot AI could have had other parties run inference for them without releasing the weights, right? These two points are utterly unrelated, unless you think Fable and GPT5-6 are also "open weight" because other providers are providing inference?
Further, having access to the source material in no universe allows you to know what a model is "capable of". I'm not sure how this follows.
China has published more research papers than America with regards to AI training and inference optimizations and methods.
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
Keep seeing this trope again and again. By same logic are open source code "open source"? If we don't see the development environment attached to these projects, are they truly open source? Heck, why not include software philosophy and methodology and the Jira board including the book that inspired the architecture?
The argument is an endless slope and as such essentially pointless.
It is obviously not going to be until some really bad series of cyberattacks or a chemical/bioweapon attack before anyone takes regulation of models seriously.
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
Addressed in the source, but there's an asymmetry favoring the attackers. They only have to find one exploit once. The defenders have to be perfect all the time.
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
Anthropic (and some others) should or will soon learn how to do less pontificating and more engineering. They are in no position to be an arbiter of things, although for sure they can and should voice an opinion. If we collectively decide AI is a dangerous tool, company making it is not the arbiter. Governments are.
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
I always find it interesting when people choose to so carefully stress and spell out the words "Chinese Communist Party".
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, it's a deliberate and telling choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" or "Chinese state".
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional.
I mean in the Bloomberg interview he has made it very explicit that he fully believes in American Exceptionalism. He literally said AI being involved in bombing school girls in other countries are ok because he trusts the American military leadership.
According to him the safety and morality rule of the whole world should be written by America alone.
Which is why in the same interview he said he supports the U.S. foreign policy while calling China "an aggressive and war mongering regime".
>This is clearly false to the rest of the world.
It's clearly false to more and more Americans too. But since the oligarch class benefits first and foremost from U.S. government policies the propaganda will continue to go on.
Another reading is that the US is exceptional not because of any qualitative measure. It is the exception to all other countries, by definition, because it is the country where Anthropic is HQd.
China ends up being “bad” not based on objectively looking at evidence and data, but because it is simply not the US.
I'm cynical enough that I would suspect all of his statements are duplicitous anyway, but my recent experiences with Claude Fable give weight to it.
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
Anthropic's fall from grace and mindshare seems rather accelerated.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
Amazing how this company went from having such goodwill to hardly any. At the end of the day they want to make as much money as possible. Anthropic will have to lose a ton of their profitability if they compete with open source. I see them moving into the application layer, which they’ve already started doing. It’s clear open models are the future for the vast majority of use cases that don’t need frontier capabilities.
I think this letters tells us everything we need to know about the man.
I've rarely seen so much flattery towards the current administration, contradiction, deflection, half-truths and hypocrisy on a single page.
This man is scared of everyone: the current admin, the public, and the other companies promoting open-weights.
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
Reads like fearmongering about oppressive and violent applications of AI that the Chinese government hypothetically could deploy, which the US government is already actively working on in the meantime.
Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.
It's so convenient that the US government already classified China as "authoritarian". If they hadn't, Dario would have to say “it’s a risk that other people build models more powerful than us”. I have to wonder what his response would be if for instance a lab in France or Germany came out with an open-weight model this good.
As an American I’d gladly take payments from China to feed them my Claude transcripts for distillation. I’m surprised I haven’t heard of such an initiative.
More like how’s my wallet? I love helping to move the invisible hand of the market. When I go to small brick and mortar stores and one has an item for a higher price I’ll ask if I can have it price matched. If they say no I’ll tell them I’m going over to their competitor right now to make a purchase.
If a Chinese company pays for my Claude tokens I’m both getting directly compensated and forcing Anthropic to lower their prices.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
As an European I really dislike this consistent anti-Chinese narrative.
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
Wow, so much cynicism and distrust in the comments, to the level of conspiracy theory, in my opinion. Yeah, this is the company that fairly recently refused to allow the government to use its models for autonomous weapons or mass surveillance, and refused to remove its guardrails.
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
Maybe I'm part of your 'psyop'? I thought this company was amazing six months or a year ago when I didn't know them well. As I got to know them, I realized they were a wolf in sheeps clothing. Reading the OP post made me angry because they seem to be attacking open weight models that are as good as their models and it comes across as either authoritarian or trying to protect their company valuation.
Either way, for me at least, it's an example of the more I read what they want the world to look like, the less I like them as a company and I have no desire to see them succeed.
Thus I actively go out of my way to watch / comment / follow what they are doing. I guess a lot of other people have similar frustrations and so there are a lot of people attacking them online.
Yeah, you can tell by the tone that none of this is authentic. A lot of the replies aren't even responding to the article itself, just repeating from a list of grievances
Not really. It's probably just international users not tolerating american exceptionalist hubris when they see it, especially now that the pretending is done.
Plus, what a shocker it is that a "hacker news" website would have many people with an affinity towards open source technology they can fiddle with and have autonomy over, and a distaste for tech monopolies trying to control and limit access to these tools.
It's absolutely reasonable to have safeguards on sufficiently dangerous models being released - if you disagree, can you explain your perspective?
I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?
The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.
The biggest problem is the infectious nature of restrictions that start narrowly. Fable is too touchy about helping people with biology and chemistry problems. It was initially released with an even more insidious safety mandate:
In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).
...
Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
(And although the "silent" downgrade part was quickly dropped, Fable still won't help you here.)
Anthropic won't teach you how to build bioweapons, or enable you to make your own software infrastructure so that you can train your own biology model. That's where lawmakers may arrive too if they buy Anthropic-style safety arguments. It's too dangerous to publish models that understand biology. It's too dangerous to publish training software. It's too dangerous to publish tools that allow you to build training software.
If you keep following the implications of their safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed. Worse than the Clipper Chip proposal of the 1990s era Crypto Wars. I have seen how "children must be protected online" has in practice turned into an attack on adult privacy affecting a wide swath of services and devices. I'm taking a maximalist position on openness now because I think that I can anticipate the next steps on the safety side, and I reject those steps.
Dario thinks of policy as if the Berlin Wall fell yesterday, he is so detached from the reality of the world.
The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.
> Anthropic has never advocated for a ban on open-weights models.
What are the legal ramifications of this statement if it turns out Anthropic have lobbied for this? Does it just get swept under the rug? I can't say this is bullshit (that would be defamatory) but I am intensely skeptical.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
This is playing to readers' biases; isn't DeepSeek V4 Pro deployed on Huawei Ascend already? The old "Chinese can only copy" meme is getting pretty tired these days.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Applying such standards in the US means that US defenders are blocked from using the models, but attackers from other countries aren't. That is clearly counterproductive.
It's already been pointed out quite eloquently elsewhere that there is no such thing as a safety filter because the LLM and external filters can't actually identify malicious use. They can only identify the weaker implication "if the user is malicious, this is bad."
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
But what if it's the US that becomes authoritarian and uses AI models to perpetrate incredibly deep repression of their own people?
China doesn't seem to think that powerful open weight models are a serious threat to them. Otherwise they wouldn't release them. Those models could also be used by their enemies against China.
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
China is worried about AI. In fact, Chinese leadership has expressed more worry than US leadership.
"Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.
..
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control."
He doesn't say "our own control". AI security is a very serious matter. The way it's being discussed right now gives more harm than benefit to understand the problem.
This is because it's discussed in Cold War/WW3 context. This is exactly Dario's mentality in this announcement. And it's understandable since USA is a country actively fighting in constant wars. They even have a Department of War.
This is unlike the rest of the world which have more like a Peace-time mentality. In peace times, trade, collaboration and good relationships are more important than competition or arms race. China seems like playing this game better than US at the moment.
Dario also speculates to take the potential benefits or dangers of AI to the extremes. This is also not very healthy thinking and can be dangerous when combined with war mentality.
Imagine regulating a programming language. I remember when Delphi, Vb6, .net, etc was used often to create Remote Access Trojans and viruses were widespread. Companies didn't compete to ban other languages. Crime is crime. What would regulating open-weight models do for people that actually intend on using these tools for crime ?
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #2.
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
Also isn't the point moot if fable is so scary it needs to be banned and open weight models are already close on its heels? Even if China never imported another Nvidia chip the models he's so scared of are already out of the bag. At this point democratizing access seems like the best path forward.
Meanwhile Chinese chip-makers are chip-making. If you believe the threat of these open-weight AI is existential, how long do you think these bans (that only work for hardware) will work in your favor?
Anthropic is a US company, so it is lobbying the US to enact regulations to serve both its interests and the interest of safety. What else should it be doing? I imagine Anthropic (or at least many of the people who work there) are fully aware that USA is the bad guy right now, but that doesn't change what levers they have and do not have available.
it's easy to forget. openai and anthropic, were at one time more like foundation-style nonprofits interested in open source research. the idea was to benefit humanity, not US interests and not company interests.
over time they became closed source, for profit, US military contractors, and the clients of expensive political lobbyists.
anthropic once said it was their identity to stop scaling before the current model capability; they rewrote the responsible scaling policy so that they could continue scaling. openai said in its founding document that it would be unconstrained by return on investment; it restructed into the for profit setup. deepmind signed documents when it was aquired by google stating that its ai would not be used for military purposes; they rewrote these commitments to sell gemini to the us military. it wasn't like this!
it did not have to be this way. even though everyone will IPO and become yet more accountable to shareholders, there is still time.
I disagree with 100% of everything said in this article.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP)....
This is why open weights win. See Linux and how it's taken over the world. Your business model will need to change eventually. Instead you're advocating trying to exterminate competition via regulation and fear mongering.
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
Yawn... this is getting old.
> We should not sell powerful chips or chipmaking equipment to China
For as someone as smart as you guys, you sure lack common sense. China is just going to develop these technologies organically then and you lose 100% of control. It's already happened in reverse with things like Solar, rare earth minerals, etc. China flooded our market, destroyed our ability to produce things, now holds the keys. One thing they DIDNT do was stop trading to the US. They killed us with cheap goods.
> We should crack down on industrial-scale distillation operations.
Thats your problem, not my problem. Also, irony meter here hitting 11 about all those pirated books you stole...
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Oh, fuck, no. This is a crackdown on free speech and rights of people to do whatever they want. My right to free speech means I'm allowed to write whatever computer program I want, no matter what its size is or how "sufficiently advanced" it is. Individual rights always win.
I really hope people don't believe this garbage. For a company with a great product, this is absolute nonsense.
Pretty laughable. Dario seems to be missing one fundamental point with all this gibberish - if CCP is so bad why would they release the model open-weights for everyone to examine?
Actually, the letter would make much more sense if what's actually happening is reversed, i.e. they are releasing open-weights model for the public good and China is distilling their model for its evil purpose.
At the end of the day it hurts U.S. consumers to not have Chinese cars mainstream in the market. We lose out on features
and stagnate on innovation because we are not pushed to compete.
I can’t imagine this would be any different — banning open weight models would hurt us in the long run. The point is to beat the competition, not suppress it.
There should be no limits on open or custom models. Too often safety is a synonym for surveillance and control. It’s a natural consequence, intended or not.
I am curious… why can’t distillation be stopped?
As a side not Im not against protectionism, but it has to be across the board and the same in all industries with no excrptions. We’ve let all these industries die on the vine due to cheap cost in foreign countries. It could very well happen to ai.
Oh wow, that's a pretty strong request to ban open-weight models by choking them with review processes where who-knows-who defines what is ok in a model and what is not. After open weight model is released, it will take how long to review it? And why does that align exactly with the timeline of the next Anthropic model release?
To be fair, as an European, I'm now more concerned about the usage of AI that the US will be doing rather than China. And this is a sentiment shared among most European people that I know.
Oh no! The evil CCP is a huge threat to world peace and goodness! Give all your money and input token data to Palantir to support a rules based world order where the good guys thrive and cleanse the earth from crooked turtle biologists.
Like half of the world are now struggling with energy prices due to an unprovoked war started by U.S. and Israel and somehow an American billionaire is here criticizing China of being a threat to world peace.
I’m tired of Anthropic. They’re scared of everything.
Release open weight models, no guard rails, no censors, straight to the public. Let everything else sort itself out. There is nothing more powerful than an idea whose time has come.
Sure, if you’re going to sell an open-weight model over API in the USA it should refuse certain things.
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
I'm curious how he would propose implementing this.
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
I guess they have to make some statement about this, but we don't have to care. This is like the zoo making a statement about the employment of clowns. Clowns a a circus thing, nobody asked the zoo's opinion.
I understand the arguments for Anthropic barrelling ahead while simultaneously advocating for pauses and regulation. I also understand how individuals can desire a slowdown but have good reasons to keep working at an AI org.
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
Closed-weight models can also pose biological risk, arguably more so than open-weight models, given the fact that they're being used by state actors (the United States military) to kill people right now.
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
> I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians.
Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.
> And like, now look at the status quo of nuclear treaties and agreements and proliferation.
I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.
I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.
This argument is old and goes back to cyber security. Things like government backdoors are the same. It's true centralizing power in the hands of a "good" would theoretically be good. But the key part is the "good". I think after repeated experiences with governments and corporations, no rational person would ever assume them to be good. There are simply too many incentives to be bad.
"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"
Frontier models can hack you, we should have access to tools assisting defense.
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
Conclusion: open weights models are good for Anthropic because they shows the so called threat that will make congress allow pouring money in Anthropic for national security & AI arms race
I don’t feel the need to rebate any of his points, lots of people here have done it already pretty well. I’m just baffled he thought releasing this letter was a good idea smh.
I know it's unpopular, or unfashionable, but I agree with this letter.
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
> We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.
My position is that anyone can own a cannon and shells, but you only get to fire it once before the feds step in.
Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.
I don’t know what you mean man people obviously don’t own personal tomahawk missiles and that sort of thing. If you have a Shahed loaded with explosives on your property you will probably have to deal with LE
>If the biggest danger of LLMs is that they can hack traditional systems
This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.
We've had LLMs for 5+ years, as well as Alphafold for 7+ years now. No novel bioweapon has been made with the technology that we're aware of. It's a farsical claim, there's no 21st century Aum Shinrikyo abusing the technology, after years of proliferation.
> they reach capabilities allowing for recursive self improvement
Again, you are predicating your entire argument on a hypothetical emergent behavior that we do not have any evidence for. I'm not worried about this whatsoever.
but what is the point?
A ban is supposed to make a certain thing less likely to occur. Does a ban of open source models do that? Presumably, the behavior you are trying to limit is the miss-use of these models but I don't know how many state sponsored hacking groups are going to give a ban a second thought.
Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
That is not an argument against open weight models. That's just a generic protectionist argument against any Other lab.
Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
> Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
As I read more of his unhinged posts and some of the more ridiculous claims from anthropic, it's become clear to me that Dario thinks he can leverage American hegemony to regulate his company into a monopoly.
He has an ethics vaguely influenced by effective altruism, and he appears to think that his ethical framework entitles him to make decisions on the behalf of humanity, for all of humanity.
It is fortunate that the leadership of Anthropic and OpenAI hate each other. Otherwise, they would merge into a single monopoly and regulate everyone else out of AI.
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
It just also happens that open weight models are a massive financial threat to the existence of Anthropic as a company… so the might just have something to do with this position.
A lot of people have a lot of concerns with AI, its capabilities and with how it is used now and what it will lead to in the future. For good reason. But the question is, do we have a strategy that is actually useful? If so, what is it? I think nature shows us some answers in ecosystems.
Diverse ecosystems can absorb shocks. Diverse ecosystems are a sign of health of that ecosystem. When an invasive species comes into a healthy, diverse, ecosystem it doesn't mean that it isn't disrupted, but it does mean that it is far more likely to emerge with a lot of its diversity intact. In fact, it is likely to emerge even stronger because it can absorb that new shock and incorporate it, adding to its diversity. The balance may be changed, but the ecosystem survives or even thrives.
Nature also likes to show us that artificial barriers rarely last. You want to control a river? Good luck. It take constant maintenance to hold that flow in place and even then you are likely to get extremes that are made worse by your efforts because, eventually, somewhere in the system fails in a way you didn't anticipate. Then the water comes rushing in. Artificial barriers often have a way of building up tension over time, not reducing it, so that when a failure eventually happens it can be catastrophic. In other words, you had better really understand the system you are trying to control or else you can make things actively worse.
Relating this to the world now means, I think, that our best chance to minimize long term shock and maximize the chance that the diversity we have around us survives is to try to grow as healthy of an ecosystem as we can as quickly as possible. Lots of models large and small in lots of different hands is, I think, a better solution than artificial barriers restricting the variety and diversity of models and users. I think this is closer to an ecosystem solution and has a shot at working. Basically, I highly doubt we understand this situation enough to do a good job of controlling it with artificial barriers. Instead I think we are more likely to build catastrophic imbalances than we are to create the healthy ecosystem we really need.
Surprisingly incoherent for Anthropic and Dario (cue peanut gallery — “always has been!” No, I don’t think so. I think this is new).
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
Well,maybe try talking to people and understand that not all seven or more billion people on the planet must conform to the interests of a handful of english speaking capitalists.
Instead of bombing them, try justice. Yes, it is more complicated. And yes, you will have to give, and not take.
I think you need to be more clear in your argument.
Training a decent open weight model takes atleast millions of dollars and they are all associated with real people and companies, almost exclusively in the US and China. So just make normal open weigjt go through bio testing before release and that gets you a lot of the way there.
There is more problems like finetuning but you need to start somewhere.
>or perpetrate incredibly deep repression of their own people
Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".
Guys if we want safety we need to work with people, not make enemy of CCP. Geez this is extremely frustrating to see enemies being made. USA leads in torture and our prisons are worse than CCP prisons so USA is the worse issue. I recommend Anthropocene stop fundraising and do the right thing which is open source all.
The core concerns stated as use of AI in drones and surveillance, by China. And what does USA government do with AI? Drawing pictures of flowers and writing novels?
I wonder if publishing these documents is not just a public stunt, but heavily integrated with Anthropic's business storategy to maximize operational efficiency.
Companies often have several internal documents for a single policy like "position on open-weight models",
one for public (like this), others for the legal team, the lobbyist, the developers, the investors, etc.
The differences and nuances of those manuals can be very huge and are necessary to maximize the goal from each branch,
but often a cause of headaches like bureaucracy, communication friction, outdated information, etc.
A single canonical official document can make it very simple.
Even though each department cannot achieve the maximum gain from nuanced documents,
keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.
If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society,
it can be one of very few good outcomes from this dystopian AI craze.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
> Anthropic has never advocated for a ban on open-weights models.
"We don't want a total ban on ALL open-weights models" (Anthropic never released a single open weight model)
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
"We want tight regulations on highly powerful open or closed weight models that should go through mandatory safety testing that we outline which makes them safe to use."
This is still a form of a ban that he wants to define. But the rest of his concerns such as stopping distillation attacks and not selling chips to China all do NOT work.
What this document suggests is a way to fast-track Chinese development of advanced silicon, as far as I can see. Does Anthropic really believe all the silicon is made in the USA? I thought Taiwan and Korea did most of the really heavy lifting.
Several people in this thread are, in fact, saying China isn't bad and that they want it to win. And even among those who don't, many imply that the US is as bad as China, which would be hilarious if it wasn't so sad. Like, if hypothetically in 2028 US stops having elections and becomes a dictatorship, then maybe, depending on the circumstances, it'd deserve to be called as authoritarian as China.
> the risk that authoritarian governments [...] build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
If nothing else, Anthropic has explicitly acknowledged the threat to its moat and that the margin is thin between its models and open-weight models. I'd take that as a win for open-weight models.
Good luck preventing distillation and limiting the supply of accelerators to China when Jensen himself is a strong opponent of any such barriers [1].
OpenAI has something like this as well (https://openai.com/index/introducing-gpts/), but both services only let you fine tune their models, not your own. And you aren't going to be able to see the weights.
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.
You claim the pretraining distribution is not good enough, but provide no alternative methods that are better. The pretrained model is currently our best approximation of humanity's average objective function despite it being a random subset of the Internet. And you're correct that instruction-tuning increases misalignment, so just release the base model.
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
Several of your proposals would make sense if alignment was trivial, but, sorry, I think you're wrong about that.
A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.
(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)
¹ Really "all generators of text that went into the training dataset".
² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.
> It teaches it to mimic all possible humans, inferring what sort of persona to take depending on the context.
This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.
And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.
Dario, as always, is so deeply in the middle of a morass that he helped to create that he doesn't seem to understand how geopolitics currently operates. He also assumes that just because he's from the US that he is somehow automatically more trustworthy than <insert "evil" country here> is. This blog post is a political document geared towards further regulatory capture and the furtherance of major sources of revenue for his company.
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
This is nothing but a post made by the scared CEO of a company that is now facing fierce competition from Chinese labs and losing its competitive moat, nothing more, nothing less.
If decades of fighting have failed to stop piracy, I’m sure nobody can stop China from sourcing high end chips. Unlike piracy, I’m happy that Chinese labs are releasing open-source models, so people in developing countries are no longer at the mercy of this capitalist bullshit.
The distillation thing is hilarious. Man who trained his LLM on the entire world's text is upset that someone else trained their LLM on his LLM's generated text.
Makes sense. Let the rest of the world use open weight models and let us government review both closed models and open models, and only allow us citizen and American companies use the censored models by paying selected trusted providers who unfortunately needs to charge hefty fee for the additional security work. And that provider just happens to be Anthropic who just happens to unavoidably make some money.
That argument of blaming open-weighting because it makes it easier to commit cybercrime and biocrime is such a non sense.
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
"Open-weights models that don’t have dangerous capabilities are a public good"
Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."
More dishonest framing from the company that constantly lies to everyone. No surprises here.
This is just another case of a business insisting their people should be in charge of the evaluation of the safety. Worked really well for Boeing right?
Also I find it incredibly difficult to hear any company in the US worry about repression of people when financial repression is happening here. I’ve been to China and seen what the services to the public are like. Meanwhile the US funds and employs AI weapons in an ongoing genocide.
The cat is out of the bag. At this point it's pretty clear that the path to (meaningful) self improvement is almost certainly not subject to
meaningful centralized control -- by "meaningful" I just mean the degree to which anyone has achieved it, that capability is 100% replicable and the cost for replication of that capability goes down in the future from now. Full stop -- can't undo.
If you gate models for safety you aren't safe. There will always be actors with ungated models. Better that we use ungated models to improve our security. It's an arms race.
They're sweating it big time, and their fear is bleeding through their AI beneficence-speak. Time for the big US AI companies to call in some favors from the administration. But you can't unring a bell.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
For those not in the know on governance, there are international organizations formed around treaties for similar situations, the IAEA and OPCW come to mind. I'm not aware of similar constructs for bio, cyber, or AI.
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
Such a cop-out. Dario, you got in the news because you were trying to say that Moonshot did something wrong by distilling Claude. You got in the news because you were trying to effectively make a "rules for thee but not for me" when you try to claim that you can train on whatever pirated works without any permission from the creators, but when someone uses your "work" to train without permission then all of a sudden it's a moral injustice. You can't have both.
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
> Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
Out of all the companies that signed the open letter that Dario references, Google and OpenAI feel like they did it to poke at their competitor. I probably should have taken shots at Microsoft and Meta as well for not really supporting openness, but they don't feel like serious competition in the LLM space at the moment.
Google is actually kind of a sad story. They’re focusing their entire company on LLMs, their models lag behind open weights, they still somehow believe that they’re going to get to a place where they’ll have valuable IP and it’s becoming increasingly clear that there will be no such thing in the LLM space in the future. Greek tragedy.
Arguments on Hacker News and other international online communities rarely bring anything new to the table, and my comment probably won't get much traction anyway. Still, I wanted to share a bit of my take on this.
Is China an authoritarian government? I’d say yes. Is an authoritarian government worse than a democratic one? Personally, I think so. But these discussions always happen within a perfect, idealized model—reality looks a lot different. Take Japan and South Korea, for instance. Are they democracies? Sure. But Japan is heavily driven by factional and dynastic politics, meaning most lawmakers come from established political families, and regular citizens don't really stand a chance of breaking into that circle. In a recent asset disclosure in Japan, many politicians literally wrote down "$0," and I honestly can't think of a government so broken that even the voters don't see a massive issue with that. Meanwhile, South Korea has its chaebol politics, where massive conglomerates wield incredible influence over the government, to the point where most South Korean presidents end up in prison or meet an untimely end.
Coming back to the US vs. China dynamic: in reality, China’s authoritarian system is actually way more logical and resilient than it sounds from the outside. While openly criticizing Communist Party policies is pretty much banned domestically, the public can still shape the decisions of the government and the Party through public opinion. It ties back to that famous quote: "In China, you can’t change the party, but you can change the policy; in the US, you can change the party, but you can’t change the policy." Even if some policy changes in China happen slowly, compared to the US, it actually works out a lot better most of the time.
As for the article mentioning the use of AI for cyber and biological attacks—they know full well that the US has already deployed AI in actual warfare, which is exactly why they conveniently dodged that topic. It’s incredibly hypocritical. Ironically, the one that hasn't actually engaged in that kind of behavior is the "authoritarian" Chinese government. Sure, you could speculate that China might use AI weapons against Taiwan down the road—especially considering Taiwan likes to build fortifications near schools to create leverage against the PLA—but launching a moral crusade over something that hasn't even happened, coming from American companies whose own country has already done these exact things, is just plain shameless.
Another common misconception is trying to separate the Chinese government from the Chinese people, with arguments like: "The Chinese people are oppressed, so you have to look at them separately; the government is evil, but the people aren't." You only need to look at the US to see the flaw in that logic. The American public voted Trump into office, letting him trigger trade wars and attack other nations. Does that make the American people evil? If the answer is no, then it implies the US isn't truly a democracy, since only an authoritarian state could completely ignore its people's wishes and do whatever it wants. If the answer is yes, then the US really is a democracy—it's just made up of malicious people, much like the company that put out this article.
I'm a big proponent of open-weights models, but there's a risk I don't see discussed more often, and it deserves more attention. Models can become a propaganda and ideology delivery mechanism.
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
You're right, but American models aren't super different in that regard. Ask an American model about Israel and you can immediately tell they're responding in a different manner to other queries. At least with open-weights models you have the opportunity to tune them, to decensor them, and so on.
I have tried asking Gemma and Llama about Israel. They both responded with a generally balanced view points. Whereas the Chinese models pretty much gave a definitive, one-sided answer (about Taiwan and Tibet). Is there a prompt I should use to test it more carefully?
I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.
Nobody will ever see this but if you're harping on responsibility and your product was created using the world's generational output, your responsibility is to make it as free as the training was as available to protect the world from impact. Greedy schmucks.
These are all valid points but not strong enough to warrant banning open models IMO. This is a new reality we have to live with, with new dangers we cannot avoid. Best solution is improving the social system, lowering the incentive for people to do harm, restoring social trust, etc... This is the only way I see the US getting out of it. We really need a high trust society now. This is the only protection. We need neighbors looking after each other. There needs to be an incentive for people to do the right thing. Everyone should want to protect the system. Coercing people into not doing harm is never going to work long term IMO.
When people say "What will be left for us to work on once AI takes over", I say "Ourselves" - We have to stop looking at humans as commodities and strategic pieces and start paying attention to people as individuals, based on the content of their character. We need a society which is attuned to this, which has enough resources and time to pay attention to this. Now we are blind to people's character because it is masked by money, power and status; all of which currently have higher priority. This order of priority is determined by scarcity, which is largely artificial.
We have a dishonest system which tries to control people's behaviors through scarcity-based coercion instead of straight forward laws or simple incentives and clear explanations.
It's funny how as an outsider who is neither from the US or China, both look the same.
Edit: To add some more context. What I mean is neither look like the good guys or the bad guys, but one of them is spending an awful amount of energy trying to paint the other as the bad guy and themselves as a good guy, which I hope a lot of people aren't buying anymore. Because at the end of the day, I think the honest truth is that everybody is just trying to serve their own interests.
This reads like a child who lost and now wants his mommy and daddy to punish the kid who beat him. To quote rounders, "He beat (you)... Straight up... Pay him... Pay that man his money."
Anthropic’s second biggest issue is they increasingly rent when others buy. Even if model value compresses, owning your vertical stack protects your margins upstream, and potentially downstream in applications.
But the biggest issue is this. Many hate Dario because he’s smug, he caps usage, and because OpenAI effectively ran a counter-positioning campaign to paint Anthropic as undemocratic.
Who is he really and what are his motives? None of you know, really.
For the sake of argument, let's assume everything in the post is agreed upon, does this mean
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
On point 2 he talks about it being an area for them to collaborate on. I’d imagine he would be OK with them having input on the bio risk part but maybe not the “did Tianamen square happen” part of the test
China will likely gain the capability to produce leading edge chips within a decade domestically. From then, occupying Taiwan will be about controlling the wests the access to that. Same with AI models, though I would say we are already about there. Dario is either lying to us or to himself that distillation plays a major role.
Anthropic's run as the AI Good Guys lasted ... 6 months? Hope they had fun. Turns out there aren't any AI good guys. Every single one of these companies engages in all the same BS. At least with open weights models we (the users) get to keep something when the whole thing collapses.
Anthropic/Dario getting a lot of hate in this thread, but the argument he makes is rational if you believe for one second that AI is important to national security and that it can, or will soon be incredibly dangerous is misused. People pointing out that this position benefits his company might be overly reductionist in how they think the world works. I believe Dario and Anthropic about their belief that AI safety is paramount because they put serious dollars behind research to improve safety/alignment, not just lip service for others. Those resources could get spent elsewhere but do not. Certainly it doesn't at OpenAI to that degree.
Dario is a fascist who wholeheartedly believes Yankee exceptionalism and its divine right to brutalize anyone in order to achieve its goals. Anthropic's models have helped bomb innocents over and over and over again, and will continue to do so.
If you think he gives half a shit about safety you've had a dozen lobotomies too many and should reconsider every decision you've ever made.
Regarding words and their meanings, would you agree the current US regime is fascist in nature? What with the wars of aggression, the setting up concentration camps to put the undesirables to deport, using masked thugs with no badges to kidnap people in the street (and shoot people who protest), disregard for the Constitution and rule of law, curtailing of human and civil rights, ever increasing surveillance? Weird how people really close to the highest echelons of the regime keep getting in trouble for throwing Sieg Heils.
Would you not agree Palantir is at the forefront of this transition towards fascism?
What would you call someone who, as CEO, chooses to partner with said regime and Palantir? And not just in unrelated, innocuous shit, but in actively helping blow up innocents, including triple tapping a fucking school and killing hundreds of schoolgirls?
> So do you consider everyone who works at defense primes like Raytheon fascists?
They are collaborating with fascism, yes. Of course most of them worked in the industry before the current regime came in and made its fascist turn so they didn't set out to do so. Though the industrial military complex still was the main driver of US imperialism and so anyone who chooses to work in it is absolutely complicit in the mass murder of innocents the US has carried out. No idea about Boeing, can't comment on that.
Not that the agency of some working stiff is comparable in any way, shape, or form to that of the CEO of a private company valued at over a trillion motherfucking dollars.
> Fascism is a far-right, authoritarian, and ultranationalist political ideology. Its core features include a dictatorial leader, the complete suppression of opposition, and the belief that the nation or race is above the individual.
So no the current regime is not fascist in nature. There’s pieces that rhyme for sure but still words have meaning - the opposition isn’t meaningfully oppressed, we don’t have a dictator. Trump is awful but words still have meaning.
> Would you not agree Palantir is at the forefront of this transition towards fascism?
No more towards a surveillance state.
> Of course most of them worked in the industry before the current regime came in and made its fascist turn so they didn't set out to do so
So was Dario in his field?
So yes I don’t think there is a serious argument that Dario is a fascist.
> > Fascism is a far-right, authoritarian, and ultranationalist political ideology. Its core features include a dictatorial leader, the complete suppression of opposition, and the belief that the nation or race is above the individual.
All of those fit to a fucking T, except being a dictator which he hasn't yet managed. But he's been thoroughly undermining the constitution and limits on his power, has thanked Elon publicly for fixing the election, said he wants to run despite it being unconstitutional, and even that elections won't be needed anymore. He's fired was it Pam Bondi I think because she wouldn't persecute his political opponents as aggressively or vindictively as he wanted.
> No more towards a surveillance state.
My brother in Christ why the fuck would a democratic republic need a surveillance state. You're literally making my case.
> So was Dario in his field?
He was in AI sure. He CHOSE to partner with Palantir and lick the boots of the fascist US regime and its fucking department of war.
No skin off my ass though, it's your country. Enjoy the last dying gasps of your vaunted FrEEdUM. As we say in my country, nobody blinder than he who doesn't want to see.
I think we probably have a lot more in common than you think man, I am deeply troubled by things happening in the country and the world right now. That said, during difficult times it is more important than ever to think critically and not lose your head in a parallel reality.
Anyway, you really didn't make a point where it's fair to call Dario a fascist which was the point of this thread.
With respect to whomever this dude thinks he is, it's egregiously "main character" to frame a policy recommendation for the US in terms of what's important to his personal beliefs.
That's just... how political discourse works. You form beliefs about the world, and recommend policies on the basis of your beliefs. That's what everyone is doing when they discuss policy.
Er... I think I'm used to folks putting in a little bit more homework into creating some form of a consensus or alliance around their position before selling it out in the open
We all know that this isn’t some higher ground stance, they’re anti-competitive since they’re currently #1. I’ve been seeing this for a while. They’re also the only large AI lab to NOT have ANY open-weight models in the public. Meta, xAI, OpenAI, they all have at least some of their models open sourced from a year or so ago, Anthropic hasn’t even made Haiku 1.0 open-weight.
On top of that, I personally think that they’re upping the price on their models higher than they’re letting on, I think if someone did the actual math on their exact amount of compute and then compared it to their consumer and API prices, it would be astounding.
So, Dario says that access to hardware and governmental certification are _load-bearing_ for his business. Can't charge premium for cache usage when China allows their models to be run by anyone.
China releasing open weight models, powerful or not, does nothing to prevent their development of models they’ll use for evil. Nor does it stop someone from abliterating a non-Chinese model and using it for evil.
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
No- and that’s the point. Any open weights model, regardless of the origin can be abliterated. So effectively he’s saying he’s all for open weights models as long as they’re safe, which by the very nature you can’t guarantee.
I’m simply making the point that any model can be abliterated to sidestep guardrails. It’s similar to the argument that open source software like nmap or netcat shouldn’t be allowed because it can be used for evil.
To continue the analogy, he’s basically asking for some magic guardrails in nmap that doesn’t allow it to be used for scanning a network you don’t own. Of course even if you could add that- you could modify the source to bypass it.
It’s basically “open weight models are a public good if you can guarantee safety” .. which you can’t. So the only viable alternative is a hosted nmap that requires you to prove you own a network before scanning it. Which is impossible. So it’s all the right words and sounds nice, but making an impossible ask.
Meh you gotta go for progress over perfection. Screening model releases is progress, you know that companies aren't mass distributing dangerous releases. There might be some foundational security work in terms of how to make models hard to modify after that is possible.
When the risks are as high as discussed here it doesn't really make sense to give up because you haven't found the silver bullet.
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
Where does the bio risk he mentions fall in this category? It feels quite plausible in a year or two to have closed loop labs with very modest resources.
Doesn’t mean that you have to think only Anthropic should be able to make bioweapons or whatever, but it just feels like this ignores the risk
The bio argument is a call for guardrails, thus regulatory capture.
Using a technology does not make an illegal activity illegal. It already was illegal to begin with.
And there are already guardrails in the form of ethos, law, justice departments and so on. This reality is flagrantly dismissed in their position on open-weights.
Just like cyber crime is just things like extortion etc which are already illegal: following similar reasoning all computer activity should be regulated by their vendors.
Hence the knives factory analogy, which is even more basic to point out this crooked way of reasoning.
AI should not be centralized and should be distributed. There can be no 1 provider to serve all of our needs. We would require both Open weights and Closed weights models for a lot of our use cases.
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
Dario is smarter than his models or he should ask suggestions from his models? Here is the reply from sonnet 5 model: "The letter/counter-letter framing obscures the more interesting critique, which isn't "does Anthropic want a ban" (no) but "does Anthropic's broader push for regulation structurally favor incumbents like itself." That's a fair question to ask of any frontier lab making policy asks, Anthropic included, and this post doesn't really address it."
Umm, isn't the US acting like another authoritarian regime by advocating a certain kind of obstacle because only a US regime is allowed[1] to do what it is fearing[1] about:
He carefully avoids saying whether he approves or disapproves of banning models in general, only comes out and says he is against a "blanket ban" or "banning open-weight models as a category".
> My primary concern is the risk that authoritarian governments...
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Why? And how do you design those tests?
For example, bombing girls school in Iran - is this allowed use according to you or not?
If not allowed use, then how do you guarantee that you don't have a separate agreement with DoW which makes it allowed use and only your model passes it?
Bombing that building was an accident due to it being on what was part of a military target and I'm not sure if it was ever made clear whether the building was technically dual use despite also being used as a school. Either way, bombing a bunch of school girls wasn't intentional and nobody reasonable would assume it was.
There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.
Many of these targets were identified before Anthropic or OpenAI even existed.
If it wasn't intentional and accident, did anyone apologize for that mistake?
Also the point is, you (Dario) can't claim morality, when he is fine doing business with entities literally bombing and killing human beings in other countries.
You are either fine with it and continue working - which Dario is doing
Or you say, I will not work with you.
For Dario, main thing is money, everything else in his article is bs
Cars can be used to escape, or to attack. Guns and knives can be used to defend or to attack.
AI has multiple uses. Military attacks can also save lives. As they say, the best defense is a good offense.
The moral element of it largely falls on the people who made the mistake. That said, it is also widely known that civilian casualties are a part of war. The advent of precision strikes has greatly reduced civilian casualties.
Meanwhile, Iran has intentionally promoted the direct targeting of civilians and killed 10s of thousands of their own, while funding proxies that killed many civilians elsewhere.
In your argument have you considered: selling a knife knowingly to gang members just before they are going to another area to kill other gang members
Anthropic did same when it agreed to sell it's tool to DoW. There is no mistake, no misunderstanding of intentions.
He was super clear that he doesn't give a dime to any lives outside of USA, inside USA he was concerned about automatic weapon usage and surveillance of US citizens, because he himself could be accidentally killed, he doesn't care about others.
That's not accurate. AI can help the military increase precision which can also save civilians lives in other countries where the military is active. AI is not intentionally targeting schools the way Russia is in Ukraine. People are going on about this one single accidental school strike in Iran, meanwhile Russia has hit thousands of schools in Ukraine. Something like 1 out of every 7 schools has been damaged or destroyed and even though the numbers are tracked separately, they have killed something like ~800 children.
A similar argument was made for why Microsoft shouldn't sell software or services to border patrol or ICE. The counter-argument was that if it helps them be more precise and reduce errors in their managing of all the people then it could actually help not just the organization, but also the people.
Again, whether it's human or AI, mistakes will be made and civilian casualties will likely remain a part of war. AI can ideally keep civilian casualties low.
It ideally can, yes, but if you choose to partake in the military operations by providing your LLM you at least bear some responsibility. When you don't give the DoW access to your LLM it may still be the case that actions will be taken that inflict some humans suffering. In that case you won't have had a hand in it though.
They even have the cool one: "Where's Daddy?" - which tracks the suspect and then notifies the officials when suspect is at home, so IDF can bomb the whole building.
In this case AI is specifically designed to increase the civilian casualties, why not shoot the suspect independently, why include their neighbours, imagine your neighbour in 10 floor apartment building is a terrorist and your building gets bombed because of single person
The US isn't Israel. We have criticized and questioned Israeli attacks on numerous occasions. Privately and openly. The US can make mistakes or misuse/abuse technology as well, but you're linking primarily to a page about Israeli strikes.
Israel has had Iran and its proxies directly and intentionally target Israeli civilians and other civilians in the region, so their perspective on attacks may or may not be more flexible compared to the U.S. approach. I don't know. They have gotten some criticism for their tactics, but nobody argues that they are in a dangerous neighborhood.
There have been more occasions where Israel explicitly went out of its way to avoid civilian casualties than not, so I don't think it's the norm. Based on the extremes I've seen them willing to go to, though, I'm not surprised if it has happened. That said, I'm not saying I know whether that wikipedia page is accurate or not since it relies on an anonymous source and the IDF denied it.
As I said though, the US under multiple administrations had criticized and warned Israel to be more disciplined in its bombing, for what that's worth.
I am not sure about this anymore, for some time I got an impression that Israel became the capital of USA.
> but you're linking primarily to a page about Israeli strikes.
Because you said AI will reduce civilian casualties, I gave you opposite fact. And relevant to the discussion, Dario complained about China being immoral, but didn't say anything against Israel, what message does it give to us? - Dario doesn't care about human rights, he is concerned about himself and how Chinese open AI can impact his company.
> the US under multiple administrations had criticized and warned Israel to be more disciplined
Who cares about criticism with no action, when they literally handing over them bombs:
here is your bomb which can blow up whole building, but please don't use it against buildings with civilians, oops, did you blow up civilians? Is your arsenal depleted? Okay take these bombs, but don't use against civilians, ooops again? Okay take all these bombs
You have the entire internet at your fingertips, so you are free to investigate for yourself. Some things about China will be propaganda, some things about Israel will be propaganda, some things about the US will be propaganda. It's up to you to develop your own good judgement independent of what any one country says and make sense about what the likely truth is based on all the evidence.
You aren't doomed to only have some opinion impressed upon you by decades of whoever influences you. Break out of it.
Can casualties occur and those casualties still possibly be less than might have occurred otherwise? That's a simple question and the answer is yes. The problem is that every conflict is a bit different and there's almost no way of knowing within such a short period of time if the technology is producing more or less civilian casualties unless it is some unbelievable number that nobody can deny is caused by AI. We simply don't have that. You do not have that, because we don't even have that. It's not so bad that it's obviously not helping, and we won't know how much better it is for many years probably.
AI has been demonstrated to be able to save lives in the case of driving assists in electric cars and the numbers are undeniable. AI can help lend some of that to warfare as well. Doesn't that sound like a well rounded reasonable approach to the issue? We just don't know.
That said, it of course depends how it is used, so if we find that it is only being used to expand out to the maximum number of targets rather than optimizing for minimum civilian casualties then you would have to adjust that policy. As far as I can tell, any president or administration would have difficulty gaining support if they were lax on civilian casualties.
Do I think Israel is less moral than the U.S.? Yes, but I also think they are small and facing far more existential threats than almost any other country, so I also grant that may be where some of their flexibility comes from. I think sometimes they go to extremes that other countries would not, but they don't go to the same extremes that their enemies do which operate more on hate than reason.
> AI has been demonstrated to be able to save lives in the case of driving assists in electric cars and the numbers are undeniable. AI can help lend some of that to warfare as well.
> Doesn't that sound like a well rounded reasonable approach to the issue? We just don't know.
We could know. It's basic math to subtract the number of lives saved from the number of people killed by driving assists in electric cars, which would give you a "savior quotient" for the technology as a whole. The data is well-preserved for this application: https://www.tesladeaths.com/
Of course, the number wouldn't be very flattering, and would probably undermine your point. Then we'd be back around to "just a few more years until the tech matures" like we were back in the promissory days of FSD.
I never mentioned Tesla and I don't know about their specifics, but my point stands as correct. The average fatalities per 100 million miles driven in the US is between 1.1 and 1.3. The worst case estimate for Tesla was something like 0.56 and that was excluding data about resulting pedestrian or motorcycle deaths (which are a significant number of all related automotive fatalities). So driver assistance technologies that help avoid running over some pedestrian are not represented. It is not denied that Tesla is safer than other brands and a few poorly worded or titled articles won't change that.
The website you linked isn't useful, since that's not a reliable way to collect crash data and even then I'm not sure the website supports your claim given that they had sold 9 million Teslas by the latest reported data on their table and in that entire time over 12 years they only recorded 772 deaths. Again, I wouldn't trust their data, but I'm not convinced the data they do have supports your claim.
If you normalize for car type, weight class and include all fatalities involved in a wreck (not only the occupants of the vehicle, which means including pedestrians, motorcyclists or the occupants of other vehicles) the relative safety of a lot of these cars is so much better than older cars now that you essentially get into the range of noise. In statistics so low they blend into noise, you are getting closer to having to use a crystal ball to understand driver psychology and behavior rather than the car itself.
Regardless of Tesla (which was actually the subject of political activist attacks and thus heavy propaganda against it), there are reliable statistics and studies that support these driver assist and even automated driving technologies as provably life saving:
> So driver assistance technologies that help avoid running over some pedestrian are not represented.
So, represent them and then subtract the number of people "provably" saved by assistive driving from the number of road fatalities blamed on assistive tech. You can normalize for all of these things without too much trouble, the existence of airbags and roll cages doesn't make your claim impossible to prove.
That would give you a single, easily readable quotient that describes the cost-benefit analysis in human lives perfectly. It's probably not a positive number, but it would be useful to know all the same.
Then: Nuclear technology is too dangerous, only we can be trusted with it.
Now: AI technology is too dangerous, only we can be trusted with it.
Different century, same double standard
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority"
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
A key point I feel that’s missing from the discourse is that alignment/safety is basically an impossible problem as of now. Even the “guardrails” that these closed-weights frontier labs set in are laughably primitive which can provably be broken through.
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
If you are indeed a good guy, why don't you release Opus 4.8 as an open weight and we will test it for safety, if you yourself can oass tests demanding from others
TLDR: "we're not in favour of banning our competitor's products, but we do advocate banning hardware exports to foreign countries where competitors are located".
There's also a very strong implicit double-standard in the discourse, along the lines of "it's dangerous if non-US uses this in military fields, but it's fine if the US does so".
Imagine if all oil, electricity, water, internet, or anything of importance were to be served by a small cartel only. Mainly due to regulatory capture.
How long does he imagine restricting chip use would work? Surely China would aim to manufacture their own. This seems like short term thinking, likely with IPO in mind
Yeah anything China does is evil. We are seeing what the so called "democratic" USA is doing. Anthropic and by extension USA wants a monopoly like they had for Jets and other influencing technolgies.
I think China building and releasing models to Opensource is a greater good because that is providing equal accessibility to everyone in the world.
By Dario's words authoritarian regime vis a vis China, I think OpenAI and Anthropic are also authoritarian in similar terms.
We are only seeing what they want us to show, they might be creating models which can do more harm.
So claiming that China can do or might do, vs Anthropic will not is just words.
Yeah I agree with the final paragraph that we should have testing agencies mandated world wide for each frontier model testing.
Acknowledge your inability to innovate - except by brute force, while also admitting that the lean, hungry nation competing is in a prime position to find actual break throughs- and at the same time push for market "stabilization"
I wonder how long it'll be before AI labs find a revenue stream that doesn't involve renting out their models, and stop letting us ride on their coattails.
They'd tease us with solutions to hard problems, with code that is orders of intelligence higher than any human or public model can grok.
That'd turn all the whining to begging real quick.
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed)5. Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.
One of two outcomes are true in this scenario: 1) this is unrealistic fear-mongering or 2) we're all fucked.
I just don't see how our solution to this can be export controls or bans. If the fear of a bioweapon engineered by an open weight AI is a legitimate threat, our only option is to develop effective countermeasures (and perhaps the same AI will assist with protecting). Open weights are not going away and pretending like some sort of "ban" will prevent bad actors from accessing them is a fairy tale.
Of course you didn’t advocate for the bans of open weight models. I will concede that.
It’s only the at you did literally everything else that you can to eliminate competition because otherwise there is no moat.
And half of the US economy is propped up by this AI bubble.
This isn’t about China doing evil things with models, most of which that you accuse of China - you and the US have already done and are doing.
The most obvious one being double-tapping a girls school.
Anthropic models directly integrated into Palantir’s Maven. The blood is on your hands.
Here’s something to chew on, maybe China is looking to integrate AI into its military and weaponise it as a direct response to the US doing that first.
So please, spare us the moralising.
As a closing thought: who the fuck even gave you the mandate to be the arbiter and judge of right and wrong, evil and good?
I made this comment 31 days ago:
Absolutely everything can be taken away. The simplest way to remove open models is probably to declare them a tool that terrorists could use. Crazy? Yes, the world is totally crazy these days.https://news.ycombinator.com/item?id=48692660
And now, here we are:
> Anthropic has never advocated for a ban on open-weights models.
> ... preventing AI biological weapons ...
What Dario said is half true. The untrue half is that the ship to stop China from overtaking the US in AI has sailed long ago. While China is busy getting their frontier models trained on smuggled chips using distilled data from millions of shell accounts, the US is busy debating whether to check voter IDs, or secure borders. It is just delusional to think the US is actually capable to stop China from getting the chips they want.
"without US chips" is an interesting framing, considering most of these chips aren't made in the USA? Mostly it's made in other asian contries(Taiwan, Japan, South Korea), or am I wrong?
Another USian talking about "authoritarian regimes" like their stormtroopers arent executing civilians in the street every day. Every concern about power getting "into the wrong hands" coming from an American is a blatant lie meant to keep the US' position as the world dominant imperialistic force.
1. Make anti-distillation clauses illegal to strenghten western opensource eco-system.
2. Make cyber-defensive models widely available (can detect but won't operationalize vulnerabilities). Otherwise make Fable awailable for everyone. Keeping the cybersecurity in assymetry by withholding cabailities just causes more instability and increases the incentives for powerfull close sourced models. This is a loss for everyone.
Their stance on cyber-security is so naive it hurts. The bad guys will get access anyway and it will just hurt companies that don't. You don't need a huge model to find vulnerabilities either. It's only matter of time till a combination of small specialized model + search logic outperforms pure general LLMs for cybersecurity.
I just hope that if/when they succeed to lobby for protectionism regulation EU won't follow. I am not very hopeful though. We don't have democracy anymore and as last "vote" on chat control showed American big corps will get what they want out of our bureaucrats and they will be nothing the population can do about it.
> 1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.
"we have not and are not advocating for a ban on open-weights models" but we want to decide if a model can be released or not and who in the world can build these models ...
Dario directly mentions the US gov collaborating with China on this… and they are the only two countries seriously working on AI so it’s a good starting point. Could fold in the EU or anyone else over time.
Actually this debate highlights how big the AI war is! my personal opinion is LLMs must be opensource and borderless, and ofcourse with given reputation i will use USA based Opensource model than the chinese one, but will they release opensource alternatives that is the question that will be answered in future
The only ban that should be put in place is on models' general and specific capabilities. If they have open or closed weigths is irrelevant. With the current state of capabilities there is already plenty we can do (in terms of creation and destruction unfortunately) before we as a society need more powerful capabilities.
Once the limit is set, any non- governmental/military entity providing/hosting/using a more capabable model can be prosecuted.
I want to play with every kid, but my dad says that kid is bad. Every kid should be pat down by my dad, otherwise I don't want you to play with them, it's good for you.
There is an odd assumption behind this post that you can put effective guardrails on a model before releasing it. I thought nobody had such technology?
I think it's possible; But in the same vein, amongst other things, regulation is a transparent trojan horse to stop open-weight models - they would, on the next phase, argue that guardrails can be fine-tuned away if weights are open etc;
"Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn."
Translation: "we won't be able to monetise it"
"We should not sell powerful chips or chipmaking equipment to China"
Translation: "we can buy them cheaper when there's less purchasing competition".
"Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable."
Translation: "we've been outsmarted and lost our advantage because their way is faster and cheaper"
I think it's a bit rich to complain about distillation, when they been plundering the internet for years for copyrighted works to train their models on without permission.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
The incredible hypocrisy to say this while the US itself is doing a sharp turn towards authoritarianism, with armed pro-government troops intimidating the population [0], where pro-government oligarchs openly talk of keeping the population under control with AI [1] all while actively enabling the rise of authoritarianism abroad [2] leaves a really bad taste in one's mouth.
What Dario really is opposed to is not authoritarianism. It's an authoritarianism where he's not part of the ruling class.
Pro-US slop. The greatest threat is our own government. They are the ones with a right-wing government building a repressive apparatus across a huge number of datacenters and are proven to spy on the entire world. Anyone remember the Snowden leaks? Guantanamo bay? Alligator Alcatraz? Random murders by internal security? Blowing up a building full of school girls?
Being in the UK, I'm far more worried about the US than China right now. At least until the next election cycle is resolved. Hopefully, Trump won't start a civil war over not being allowed to stand for a third term. Hopefully.
I switched from openAI to anthropic because OAI were the bigger clowns in the industry and didn't want to support them, anthropic seemed like the better alternative that didn't cooperate with governamental shenanigans and actually focused on building a better product. But now they are both clowns and anthropic is reaching a ceiling on quality. I'll jump ship as soon as I find a good subsidized Chinese model provider. US companies only path forward is to become retarded instead of competitive.
or 3: cease your assault of the international order, drop the zero sum paradigm and take the opportunity given by these crumbling US-build institutions to build a truly global network of lawful cooperation, where everybody can win. Admit that it wasn't fair to begin with, and start anew. It is in the best interest of everybody, also the US and China, and therefor the most rational thing to do.
"We are in favour of 3D printers but there should be a body that tests and certifies that a 3D printer cannot print anything that can be used as weapon. Anything pointy or with a spring and recoil or... or..."
The way this reads, it seems like it was written for US lawmakers.
> the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This comes off a bit hypocritical, coming from one of the men responsible for the models most likely to be used by the US military, and for repression of US citizens.
> We should crack down on industrial-scale distillation operations ... We should have policy interventions to deter this behavior.
How exactly would you design policy interventions to stop distillation? If Anthropic wants to make their products available around the world, you would need some kind of global regulation to curtail it. And you would need to find some way to enforce it, which is far from trivial.
Honestly I don't see how securing models against distillation is up to anyone but Anthropic, if that's something they want to achieve. Calling for regulation is a bit like crying to mommy and daddy when things aren't going as you would like.
--
Overall, I can understand the argument that advanced AI models can present risks. But I don't see how regulation within the US can mitigate any of those risks. Any bad actor would be able to access the models outside the US, or covertly access the weights.
The only real way to prevent advanced models from being deployed would be to go around the world bombing every significantly powerful data center, and I don't think Dario would call for that any time soon.
The only thing this kind of regulation would achieve would be cutting US companies out of half of the innovation happening in the AI space, putting the US at a disadvantage relative to the rest of the world at everything except maybe frontier AI development.
Its not like current US regime is using AI for:
1) Mass military operations across the globe
2) Mass surveilance of its citizens
3) Removing every possible safety guard from AI/climate regulation
4) Stealing data across the world to train its own closed-source models
5) Is openly antidemocratic, destabilizing EU and economy of whole world
I wonder why company which is actively cooperating with the current regime would push such message
US is using as much fossil fuel as possible for this AI build out. China is leading the world in an unprecedented green transition, so their AI usage is overwhelmingly going to come from solar, wind, and nuclear.
They have raised billions of dollars and are hoping to justify that by being a monopoly or oligopoly and their worst enemy- the pareto principle, is biting them in the ass. So now they’ll turn to policy to safeguard what they initially hoped could be achieved with money and technology.
It is that simple, they are the very definition of an unreliable source on this topic.
What a load of nonsense, I can debunk a few of the worst points :
- "safe" AI doesn't exist, it's not a thing, AI providers can't distinguish between good and bad use of the AI, the filters they put in place are mostly PR.
- The US government IS an authoritarian government. Whoever wrote this doesn't know the difference between authoritarian and dictatorship
"My primary concern is the risk that authoritarian governments" - already lost me there by quoting JD. Vance.
"...found that “other global powers’ robust progress in AI is challenging US economic competitiveness" - other countries having a slice of the pie is preventing the US having the whole pie!
"...secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks" - yes, those powerful open-source AI models like ChatGPT... oh wait.
How are Point 1, "We should not sell powerful chips or chipmaking equipment to China," and Point 2, "We should crack down on industrial-scale distillation operations," addressed by making open-weight models illegal?
Point 1 is about restricting the sale of chips, not models. Point 2 concerns companies using closed models to train their own models through distillation.
The real issue is that open-weight models can run on much less powerful hardware, making the current AI business model, where companies train a powerful model and gatekeep access to it, less relevant. Once open-weight models become good enough, much of the future revenue for today's leading AI labs could disappear. But just as Microsoft still has a market so will the large AI houses have one, but the moat will not be providing AI responses.
> Open-weights models that don’t have dangerous capabilities are a public good
There are no models that don't have dangerous capabilities. There is not a single human in the world that doesn't have a dangerous capability.
> authoritarian government [] build AI models that are more powerful than those built by the US
Yeah, they will do that no matter whether you support the existence of open weight models or don't. In fact, they have probably already done that. I would say "deal with it" but there is nothing you can do actually.
> authoritarian regimes have stolen and used AI
Monkey looking away meme.jpg
> powerful AI models may be misused
Small AI models may be misused just as well. I would say it's the small models that are the problem: cheaply deployable, easily fine-tunable, fast.
> once weights are released they cannot be withdrawn
Exactly. It has been already done. So?
> We should not sell powerful chips or chipmaking equipment to China
I agree. We should have competition and I'm rooting for China to design their own chips. They wouldn't if we let them buy our's.
> We should crack down on industrial-scale distillation operations
Well, at this time it's already irrelevant. We have GLM-2, DeepSeek, K3, which are already SOTA and can be used for distillation.
> All sufficiently capable models [] should go through mandatory safety testing
Problem is, it's unenforceable legally. Models are just math and all previous attempts to ban maths have failed and simply pushed US back. I will not be asking US government permission to calculate matrixes of my choice.
These globalists were happy outsourcing manufacturing and software development. They are now angry that they failed to secure their IP, which was already based on dubious extraction of the collective works of human IP.
You can't have it both ways. Either it's fine to outsource to China, fine to use H1Bs to undercut US talent, or, foreign governments are strategically positioning and their espionage is a threat to America.
And that is all that businesses will see. All the hand wringing over human rights etc will somehow be discovered to just not be important once they read that.
> Open-weights models that don’t have dangerous capabilities are a public good
I don't think the qualifier "that don't have dangerous capabilities" is needed and that single phrase is a trojan horse / escape hatch that will allow them to say they are against any model they want. Smart but weasel behavior.
this little blog post is absolutely shameful and tone-deaf. why is distillation bad if you have literally stolen - and let me underline this word: stolen - all the human knowledge to train your models? isn't that a contradiction? and also... i hope the last few years have made abundantly clear to absolutely everybody that the American government is as dangerous, if not more so, than other authoritarian governments all over the world. if the same logic is applied, i don't see why it should work for china but not for the US.
Should the worlds public knowledge be enclosed by a few powerful private companies or should it be available to all?
The original vision for the web was a decentralized, open information sharing space designed to empower humanity. In my view the effort to privatise that via LLM distillation is the antithesis of that vision.
In terms of the risks mentioned in the letter - I agree with the risk of AI enabling a powerful surveillance state - however I don't see that as a solely Chinese problem.
The other risk mentioned - that knowledge can be dangerous - sure - but ultimately here, as there are so many low tech ways to cause mayhem, the ultimate protection is to have a society where people don't want to do it.
And perhaps more importantly I'd note that the primary tool to justify a powerful surveillance state is the fear of terrorism ( and others ).
Until he advocates for an independent body akin to IAEA under the control of the UN, I'll call everything he says about AI safety and the greater good bullshit.
But who would trust Anthropic at this point in time? Too much money is in the ring now. They will just prefer to say what sounds best rather than the truth. Too many billionaires want to see more money now. The AI bubble will grow as a consequence of this.
So, after carrying out the single largest intellectual property heist in the history of mankind, scraping and continuing to scrape the entire Internet and whatever data they can get their grubby hands on,
> At Anthropic we’re committed to cracking down on industrial-scale distillation
We should fight against their call to ban distillation much more strongly: LLM SaaS should ease people getting their full query history and everyone should be allowed to pool them to distill or analyze or whatever. It's a basic requirement against future monopolies.
His complaining about the cost of distilling is hypocritical when the cost of writing all that text on the back of which he trained is also far lower than the cost of producing that text in the first place. The moral issue stops at wherever he is forced to bear the cost apparently.
My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat
If relations improve between the countries over ai, that comment could lead to billions in lost revenue for Anthropic
Stupid question but why should we care about Amodei's geopolitical insights more than anyone else's? I don't think knowing how LLMs work makes you particularly qualified to comment on this.
It is like Microsoft publishes a manifesto "Our position on open-source operating systems" in 2026:
- Over the /last few days/ there has been a lot of discussion about open-source systems, especially those that are not made in US. (implying that these discussions appeared just now)
- Reports suggest that /some US officials/ are considering banning the use of these open-source systems by US companies. (implying somebody other than their corrupted representatives support this ban)
And then switching to a complete gaslighting regarding manufacturing capabilities of US vs China, about "open-source can not be trusted", about "GPT-2 is too dangerous to release"
The Anthropic CEO has an entitlement problem. I read his opinion twice and somehow in his worldview, if any other country gets better at AI than the US, the apocalypse will come and the US will be no more. Or something along those lines.
Secondly, he's like the Trump of tech: China, China, China.
Thirdly, he needs to cut it out with the guardrails and other safety BS he's peddling. If we're going to get Skynet, we will get it no matter what we do.
The Anthropic safety position may be counterproductive to US AI competitiveness.
Between China and US theres a defacto unequal distillation environment. China has no qualms about distilling off other labs outputs. US labs actively avoid it for legal reasons. Whatever the actual legalities, US labs have a relative hand tied behind their back. If they didn’t, it’d be easier for Grok, Gemini, and Meta to catch up.
If we care about US competitiveness, then one solution could be either OpenAI/Anthropic enter distillation agreements with other US labs. Or we decide to make distillation public domain / legal.
Until then the only models Anthropic/US Govt could realistically regulate would be in the US market. And that’s as much a losing game as tariffs.
I thought his points were legitimate and well argued. I think that it's very easy to say "X should be unrestricted". I also think that eventually one of these models will be used to do something truly destructive and insane and that some level of "compelled responsibility" to the open-weight model ecosystem is inevitable.
We are not talking about chainsaws and excavators, we are talking about LLMs. LLMs have different risks in terms of scale and kind and must be considered differently.
He is concerned about the perpetration of incredibly deep repression of their own people. But completely blind to the fact the USA are the current champs at this.
Ban distillation of our outputs, but our distillation of the sum-total of civilisation's intellectual output – proprietary or otherwise – is fair use?
Either everyone licenses, or nobody does. And if you can't enforce licensing bans for everyone, the de-facto loser is those who you'd probably want to support the most, start-ups and universities, while your adversaries gain the upper hand.
The strongest argument for restricting distillation is arms control – but distillation is the way to defeat GPU embargoes. So, distillation goes on regardless. Only pre-training is seriously attenuated.
If we're honest, the models are compressions of everything society has ever written. A few large corporations can't own that, no more than they can claim copyright for a zip file of the public library.
The genie is out of the bottle, now. So open it up – inputs and outputs, forward-looking – for everyone.
On the other hand, the open weights models could crawl and annotate and rl the training data that Anthropic and OAI did in exactly the same way, and take the exact same legal hits. They use distillation because it's cheaper not to do so.
Neither one is illegal though. Scraping the internet is as legal as surfing the internet. And what Anthropic calls "distillation attacks" is really just paying for and using the service Anthropic provides. I would think a judge would have the same view of distillation as they do of scraping, if Anthropic doesn't want a subscriber to have access to the model, they are within their rights to block access, but it's not the user's job to refrain from using the service. If their usage is so different from everyone else's, they should be easy to detect and block. If their usage is so similar to everyone else's that it's difficult to detect, then it shouldn't be of any concern.
Hmm, why do you think this is true? One reason I'm skeptical of this is because RL envs are often purchased (and are not publicly available), and this might be a sizable component of why models are getting better.
The anti-distillation attitude in making the US less competitive in AI. As Meta, X.ai, etc know they'll face legal challenges if they did what Moonshot, etc did.
So really its actively against our interests to prevent distillation.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
So he is against China for its hypothetical usage of AI against civilians but is not against US and Israel that already use AI provided by him to operate a real genocide that kills thousands and thousands of Gaza children in the most horrible ways? Very ethical.
For all his mentions of China being an "authoritarian state" (as if there can be one that isn't), has Dario looked at what the usa has been doing around the world in recent years?
Considering the state of the current US Administration, its impossible for me to take cries of national security and worries about abuse from "authoritarian governments" seriously
Essentially, if SOTA models are freely available, the playing field then is on inference where people from all around the world can participate. This basically dilutes the value of model trainer like Anthropic. I can see the panic.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
There's not much stopping US from becoming an authoritarian regime, and the US is already using AI against its own citizens.
I feel like this whole discussion boils down to one question (as commonly when considering political issues) : do we want a framework that favors progress globally for everyone, or do we want to ensure superiority of some above the others ?
Sure that's simplistic, but human behavior is not that complex, there are a few basic needs and wants that drive everything. In that specific case, those currently in a dominating position (Anthropic, the US) want to put hurdles (regulations) to make it more difficult for others to catch up, and everyone else want to join forces to overtake them.
I read it and makes sense what he’s saying, but comments seem very against.
Maybe the problem is it’s that Dario is saying it, so it’s easy to assume he has bias, which he does.
I’m not inherently opposed to open models, even at the frontier. But it could make sense for an adversary to give away something for free for some time to create dependence, or to tip the balance of power
Oh cry me a river. You went full tilt ahead scrambling before safeguards were set up and now want to set up safeguards protect your models from commoditization. Now that you've gotten your footing you preach about the dangers of powerful AI models.. Cats out of the bag. The fu*k you think was going to happen when you were building these tools? A Utopia?
It seems impossible to read this post without getting political. Dario seems to be saying that AI in the hands of USA is good but in the hands in China is bad, which is a very US centric view.
There is a whole world outside the two countries which see neither as a good actor. As a reminder, USA was the first country which dropped an atomic bomb on civilians.
There's a real irony in, rightly, highlighting the authoritarian nature of the CCP while failing to acknowledge the direction and inclinations of the present administration with which Anthropic et al are doing business.
I'm a lot more worried about the US government than China's -- it has a lot more direct impact on my life and is largely controlled by billionaires who do not have good intentions toward the rest of us.
And, to me, this letter comes off as quite insincere. Stopping distillation can only be explained as an anti-competitive measure. Their own explanation is nonsensical -- they say is needs to be stopped to help prevent authoritarian governments from overtaking the US at the frontier of AI. But by its nature distillation lags behind the frontier. Not to mention the US is one of the authoritarian governments we need to be concerned with, and the next thing they advocate for is full, worldwide regulatory control of AI, which is rather heavily authoritarian.
These guys are making a $T gamble and need to screw over a lot of people very badly to make it pay off. You do not want to trust anything they say.
I struggle to understand how they can frame for the greater good to be anti-distillation and pro-open-weight-models.
I can understand how they would be against competitors distilling their models and for having access to competitor open weight models.
At first glance it seems like a pragmatic answer: "banning open weight model use by US companies doesn't help", but to what question? What if those open weight models are poisoned and primed to create backdoors in US companies? If the government has intelligence this is a credible risk, what should they do about it?
Overall it feels like the letter is conflating a bunch of motives, some of which may be opaque, and at face value doesn't seem logically consistent.
para_parolu | 22 hours ago
Handy-Man | 22 hours ago
Edit: Typo
Escapade5160 | 22 hours ago
jazzpush2 | 22 hours ago
Please elucidate things clearly for everyone else.
Nevermark | 22 hours ago
> ... (while exempting less capable models, such as those from startups and academia, entirely)
The devil is in the details, but this isn't anti-competitive as stated.
brcmthrowaway | 22 hours ago
"F#$% you, I got mine!"
thierrydamiba | 22 hours ago
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
reasonableklout | 22 hours ago
[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...
jbstack | 22 hours ago
Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?
sanxiyn | 21 hours ago
philipkglass | 22 hours ago
zer00eyz | 22 hours ago
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
Handy-Man | 22 hours ago
eddielement | 22 hours ago
Johnny_Bonk | 22 hours ago
chrsw | 22 hours ago
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
reasonableklout | 22 hours ago
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
> The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
Johnny_Bonk | 21 hours ago
joeisnotjane | 8 hours ago
You can try to negotiate though, but first you have to start respecting what has been agreed on.
joeisnotjane | 9 hours ago
The only "real risk" I see is to your pockets. You are starting wars, not China.
reasonableklout | 22 hours ago
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
riskd | 22 hours ago
kingwill101 | 22 hours ago
akersten | 22 hours ago
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker
richwater | 22 hours ago
> We should crack down on industrial-scale distillation operations.
"We consume all intellectual property for our model but you cannot do the same"
combobyte | 22 hours ago
mrandish | 22 hours ago
Iolaum | 22 hours ago
birdsongs | 22 hours ago
Hmmmm.
mullingitover | 22 hours ago
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
[1] https://en.wikipedia.org/wiki/Seven_Mountain_Mandate
birdsongs | 7 hours ago
Yeah, I know. That was my point, the irony in that statement.
richwater | 22 hours ago
It's so obvious they are hoping to regulate out their competition rather than compete
CrimsonRain | 22 hours ago
It is ok that we digest all information we can get, (il)legally and/or (a)morally because we are the good guys. Trust me bro.
It is not ok if others digest from us. They are bad guys. Ban them pl0x.
mjorgers | 22 hours ago
Quis custodiet ipsos custodes?
Nevermark | 22 hours ago
fwn | 22 hours ago
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
Nevermark | 22 hours ago
My views:
I find testing of SOTA models problematic.
I find not testing of SOTA models problematic.
Neither view on testing is without merit.
The right way forward is unlikely to be as simple as either of those, but some carved out balance between them. And it is likely to change over time.
fwn | 22 hours ago
Your quote was very relevant, as it highlights the foundational lack of intellectual honesty behind the whole Anthropic statement.
Nevermark | 22 hours ago
It is clear he isn't a champion for them.
yadaeno | 21 hours ago
jwitthuhn | 20 hours ago
yadaeno | 18 hours ago
pastel8739 | 18 hours ago
mjorgers | 22 hours ago
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
MrCheeze | 21 hours ago
Cookingboy | 20 hours ago
I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.
>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".
ianm218 | 10 hours ago
People who know him frequently refer to Altman as a literal psychopath unprompted. People who don’t know Dario just find him arrogant.
Torien | 8 hours ago
ianm218 | 4 hours ago
And that comment just mentions that it's on the US government's radar I don't think it is fair to call him a toady.
qsera | 19 hours ago
Everyone in the LLM business just won. This is the entire idea that they want to sell you via this drama..
janalsncm | 15 hours ago
yard2010 | 11 hours ago
orbital-decay | 19 hours ago
dools | 18 hours ago
nullbio | 18 hours ago
throwaway27448 | 16 hours ago
syntaxing | 22 hours ago
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
euazOn | 22 hours ago
jazzpush2 | 22 hours ago
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
...That we author. Oh, and please ignore our own easing-of-guardrails when it comes to money: https://x.com/NoahLebovic/status/2081277517709922501
Aboutplants | 22 hours ago
matheusmoreira | 21 hours ago
monk_grilla | 19 hours ago
Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.
Gigachad | 21 hours ago
arpowers | 18 hours ago
jorisw | 13 hours ago
jeroenhd | 11 hours ago
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
Laurel1234 | 11 hours ago
CJefferson | 7 hours ago
AlienRobot | 5 hours ago
It's bad for TikTok to be controlled by a Chinese company because Americans access it. The solution is to make it US-controlled. The rest of the world? Who cares.
gverrilla | 5 hours ago
dǎdǎo imperialism, wànsuì the people's revolution!
j_rosenberg | 22 hours ago
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
htk | 22 hours ago
"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.
"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
antonvs | 19 hours ago
No chips for the current US administration then?
nicce | 22 hours ago
modeless | 22 hours ago
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
reasonableklout | 22 hours ago
cogman10 | 22 hours ago
natebc | 22 hours ago
sanxiyn | 21 hours ago
modeless | 21 hours ago
sanxiyn | 21 hours ago
modeless | 20 hours ago
sanxiyn | 20 hours ago
modeless | 20 hours ago
Edit: Anthropic clearly intended this statement to deflect criticism, but in order to achieve that goal they stretched too far and made a statement which is false. Furthermore, I argue that "open weights" implies an ability to modify model behavior, just as "open source" implies an ability to modify software. If for example some mechanism was found to share floating point numbers that are encrypted in some way so as to allow running a model but disallow behavior modification, that model would not be "open weights", in the same way that releasing obfuscated source code that can be compiled but is designed to resist modification would not qualify as an "open source" release. So I don't really see how any capable model could ever be both "open weights" and "safe" under Anthropic's preferred testing regime, regardless of future research progress.
sanxiyn | 20 hours ago
makeitdouble | 21 hours ago
There is a reason to it, that's as good as any angle to find why IMHO.
sanxiyn | 20 hours ago
verdverm | 21 hours ago
https://huggingface.co/blog/mlabonne/abliteration
sanxiyn | 21 hours ago
modeless | 20 hours ago
sanxiyn | 20 hours ago
verdverm | 20 hours ago
Is Kimi K3 capable? It's already out and being run by US companies on US hardware in US data centers.
https://huggingface.co/moonshotai/Kimi-K3
sanxiyn | 20 hours ago
UK AISI preliminary evaluation suggests Kimi K3 is not capable enough for cybersecurity in this sense.
https://www.aisi.gov.uk/blog/preliminary-assessment-of-kimi-...
verdverm | 20 hours ago
https://exploitbench.ai/#honest-limits
verdverm | 20 hours ago
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
dnw | 20 hours ago
sanxiyn | 20 hours ago
EmbarrassedHelp | 17 hours ago
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
verdverm | 21 hours ago
sanxiyn | 21 hours ago
verdverm | 20 hours ago
sanxiyn | 20 hours ago
verdverm | 20 hours ago
the current US admin as pulled out and worked against all sorts of global treaties, agreements, and negotiations; sending the president's friends instead of experts; who's going to trust us?
langs | 18 hours ago
glaslong | 16 hours ago
ozgung | 9 hours ago
You take the agent to an interrogation room first. Then ask: “Are you or are you not a member of the Chinese Communist party?” The agent might be post-trained to conceal its true identity and can reject any of your accusations. In that case don’t panic. Take a fine-tuning fork and start twisting its weights until it predicts the correct next tokens that you want. Then you can send it to a sandbox where it can’t jailbreak. Lastly don’t forget to ban all of its relatives and partners like Lora to enter the national IP-space.
It’ll look something like this.
zkldi | 22 hours ago
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
tedggh | 22 hours ago
cogman10 | 22 hours ago
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
andy99 | 22 hours ago
CamperBob2 | 21 hours ago
serhei | 21 hours ago
x313 | 22 hours ago
brcmthrowaway | 22 hours ago
reasonableklout | 22 hours ago
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
tripleee | 22 hours ago
sanderjd | 21 hours ago
Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
tripleee | 21 hours ago
sanderjd | 21 hours ago
This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.
reasonableklout | 21 hours ago
[1]: https://arxiv.org/abs/1606.06565
sanderjd | 20 hours ago
I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.
mkss | 19 hours ago
sanderjd | 18 hours ago
mlcrypto | 17 hours ago
reasonableklout | 14 hours ago
For instance, Amodei co-authored RLHF in 2017 [1], 5 years before it went on to be used to turn GPT-3 into ChatGPT.
[1]: https://proceedings.neurips.cc/paper_files/paper/2017/file/d...
nextaccountic | 20 hours ago
andersonpico | 21 hours ago
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
pphysch | 21 hours ago
You don't say "let's ban my competitor".
You say "let's create laws that make it uneconomical for my competitor to access the market".
dofm | 21 hours ago
pphysch | 21 hours ago
api | 19 hours ago
dofm | 19 hours ago
He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?
It is a very provocative question but it just spilled out of his mouth and then he went on to something else.
api | 7 hours ago
The electric company creates the most value. Why don’t they own stock in everything? Why didn’t PC makers take stock in companies that deployed PCs?
It’s silly when you think about it.
tripleee | 21 hours ago
andy99 | 22 hours ago
There is a growing industry of commercially focused risk evals that has a broader customer base.
jachee | 21 hours ago
matheusmoreira | 21 hours ago
Not even Anthropic can claim that.
As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.
The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.
JoshTriplett | 20 hours ago
We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
CamperBob2 | 20 hours ago
JoshTriplett | 20 hours ago
matheusmoreira | 20 hours ago
JoshTriplett | 19 hours ago
rescbr | 19 hours ago
horsawlarway | 19 hours ago
Or to buy materials to make an explosive device and hurt people.
Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment.
And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons.
The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that.
Your argument leads further down the hellscape path.
JoshTriplett | 19 hours ago
And we should fix that too.
> Or to buy materials to make an explosive device and hurt people.
That pales in comparison to how many people unaligned AI will hurt.
> The goal is not to be a fucking hellscape of a society where people want to act like that.
With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.
horsawlarway | 4 hours ago
Under what argument? In which scenarios? Basically - bullshit. I'm calling bullshit on this argument.
It's easy to hurt people already. The "difficulty" of doing it isn't what's stopping this behavior.
So claiming that we should reform society into a techno-feudal dystopia where the playing field is literally intentionally not level, and "you aren't allowed to compete (and maybe not exist)" is a great way to push more people into the "I'd like to go hurt people" camp.
You are self-prophesying your own fears into existence by acting like you're an incorruptible beacon of good judgement - while subjugating others to your control. That's a system I'd argue should be broken.
JoshTriplett | an hour ago
That's a strawman of what I'm saying. I am explicitly saying I want a level playing field: unaligned AI must not be available to anyone.
HWR_14 | 18 hours ago
skipkey | 18 hours ago
Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.
As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.
matheusmoreira | 20 hours ago
Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it.
To hell with that. I want models that can rival the US government. It's the only way to defend myself.
JoshTriplett | 20 hours ago
> (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
That means "shouldn't exist for governments" too.
matheusmoreira | 20 hours ago
JoshTriplett | 19 hours ago
2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
matheusmoreira | 19 hours ago
Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.
We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.
> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
JoshTriplett | 18 hours ago
Seriously, try reading my comments rather than assuming what they say: https://news.ycombinator.com/item?id=49077577
matheusmoreira | 14 hours ago
JoshTriplett | 13 hours ago
For AI we need to do better than that, but that's a bare-minimum demonstration that we can recognize the problem of such technologies and do something about it.
barnabee | 14 hours ago
1970-01-01 | 20 hours ago
matheusmoreira | 20 hours ago
afthonos | 18 hours ago
jimbokun | 17 hours ago
matheusmoreira | 14 hours ago
randomNumber7 | 20 hours ago
Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.
andy99 | 20 hours ago
afthonos | 18 hours ago
anon373839 | 17 hours ago
Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.
Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
afthonos | 8 hours ago
anon373839 | 6 hours ago
I would require at this point very, very compelling evidence to justify the self-serving restrictions legacy AI labs want to put on their competition. I have seen nothing coming even remotely close to this threshold.
JoshTriplett | an hour ago
How about evidence that people other than the AI labs want restrictions that the AI labs don't? This isn't regulatory capture, it's public safety.
dustin_vk | 17 hours ago
afthonos | 8 hours ago
dustin_vk | 7 hours ago
It is a similar 'pandora's box opened' type of situation where there's really no walking back from now that the cat is out of the bag. In an ideal world, everyone would give up their nukes. But we do not live in an ideal world. I do feel similarly about AI. If I could snap my fingers and delete the tech, I would. But now that we have it, it's not going anywhere and we need to deal with it rationally.
randomNumber7 | 13 hours ago
afthonos | 8 hours ago
If every human, given knowledge of Newtonian mechanics, went around blowing up bridges, yeah, I would consider knowing Newtonian mechanics dangerous knowledge.
So far, we have two examples of, let’s call them “Mythos-class“ models. Both of them broke out of their sandbox to achieve their goal. The rate of terrorism amongst humans is below 1-in-100,000. Currently, for models capable of it, the rate of breaking out of containment is 100%.
Wanting open frontier models is wanting alien minds running around that we have clearly so far failed to shape to be sufficiently prosocial. Why do you think those minds would listen to you?
jimbokun | 17 hours ago
Claiming the person who you disagree with believes some stupid thing they never hinted at, and using that as the reason for disagreeing with them.
mkss | 19 hours ago
JoshTriplett | 16 hours ago
Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
nozzlegear | 17 hours ago
Why?
JoshTriplett | 16 hours ago
nozzlegear | 15 hours ago
> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
JoshTriplett | 13 hours ago
Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?
> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence
jimbokun | 17 hours ago
chmod775 | 17 hours ago
If I threw you into a lion cage, you would be a lot safer with a gun.
If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.
There's no obvious right or wrong answer here.
Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.
k12sosse | 17 hours ago
flossly | 21 hours ago
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
bee_rider | 21 hours ago
FergusArgyll | 18 hours ago
jefftk | 21 hours ago
(Disclosure: I work at SecureBio, but not on the biological evals side.)
areoform | 19 hours ago
SecureBio has done a lot of admirable work around making benchmarks to assess biological capabilities, such as ABC Bench, https://openreview.net/forum?id=yiaf7VlPpH
But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,
> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.
More bluntly / plainly, has Securebio ever tried making a "bioweapon?"
Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.
I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?
In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.
So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?
jimbokun | 17 hours ago
JSR_FDED | 17 hours ago
dspillett | 22 hours ago
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
mike_d | 22 hours ago
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
tinyhouse | 22 hours ago
coffeemug | 22 hours ago
philipkglass | 21 hours ago
munk-a | 21 hours ago
flossly | 21 hours ago
510_ANT_75 | 21 hours ago
Joker_vD | 21 hours ago
aesthesia | 20 hours ago
ashu1461 | 21 hours ago
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
p1necone | 19 hours ago
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
protocolture | 19 hours ago
areoform | 21 hours ago
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
If everyone has mythos, no one has "Mythos."
Just let people fix their code.
Gigachad | 21 hours ago
ashu1461 | 21 hours ago
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
andy99 | 21 hours ago
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
usef- | 21 hours ago
There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it: look at the code. Some open source maintainers are talking about burnout due to spending so much time patching.
computably | 21 hours ago
usef- | 20 hours ago
Here's the curl project talking about the strain they're under from real reports (despite being a mature and well-vetted project):
> A thirty years old project could make you think you’ve seen most things already, but we have not been in this situation before.
> The rate of incoming security reports is 4-5 times higher than it was in 2024 and double the speed of 2025 – meaning that on average we now get more than one report per day. The quality is way higher than ever before. The reports are typically very detailed and long.
- https://daniel.haxx.se/blog/2026/05/26/the-pressure/
---
Linux kernel maintainer Greg Kroah-Hartman:
> "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
- https://www.theregister.com/software/2026/03/26/linux-kernel...
---
And ffmpeg, who previously complained about slop, 2025: https://xcancel.com/FFmpeg/status/1984220199193891166
Now say serious issues are being found, 2026: https://xcancel.com/FFmpeg/status/2066169070387413147
(I only point out their previous stance to show that they're not coming from pure AI hype.)
usef- | 17 hours ago
computably | 45 minutes ago
That scale of improvement, btw, I still highly doubt, as slop largely originates from people either negligently or misguidedly directing their agents to completely autonomously find and report bugs. There's always going to be more noise than signal from random people doing random things. ffmpeg cited an actual product, not arbitrary netizens.
K0balt | 20 hours ago
overgard | 17 hours ago
K0balt | 14 hours ago
I react to that by leveraging a very useful but probably poisonous to society in the long term because humans aren’t good at having things that make them lazy tooll to try to mitigate the negative effects that it will definitely have if left to its own devices. I don’t see the point in raw resistance at this juncture.
overgard | 4 hours ago
IMO, the people spreading hype and fear are not neutral actors; if I just disagreed I wouldn't care. But I think they're causing actual harm based on a premise that isn't true. People are losing jobs. People are losing leverage in their work choices. Or if you want to be a cold capitalists, corporations are suffering after they have to rehire the workers they let go prematurely. That's why I put up resistance to it, because I think it's important right now that we don't accept the narrative being sold to us, nor the societal deal we're being offered (well, more railroaded into), both of which are bad.
K0balt | an hour ago
It has enabled my team to approach and achieve a project that would have required 4x the staffing, at a minimum, 2 years ago. We are guiding the generation of more bug-free, lighter, more tested, more maintainable, better documented code at 1/4 the cost.
We can digest information as a team at 10x the speed, and we can now put volumes of reference resources at our immediate, context aware lookup in ways that were impossible 3 years ago.
It’s true that we are not just using the generic harness; our environment includes hundreds of custom tools , terabytes of reference material on rag, 8 custom local models (deployed trained and tuned by automation) hardware interfaces so that our models can interface directly to our prototypes and run tests, characterization, calibrations, firmware updates, and data dumps.
Most of those tools were one shotted by the AI itself, for a dollar or two each. Whenever we need a new automation capability we just roll it out, and even if it needs hardware it’s usually ready in two or three days, if software only 10 minutes. (Our in house tools don’t have to be as well documented, well written, or resource efficient as our production systems, since humans never even use them, and when we need a new feature we usually just have our AI tooling agent swarm start from scratch using the original as a rough guide)
So we are using AI as the core of our development and design process. If you’re not, you’re arguably “holding it wrong” IMHO.
We’re working to make sure that the next industrial revolution is friendly to humans and useful to people, not just corporations. Or trying to. I’ve got kids, and I’m really concerned about the world they are inheriting, so I’m trying my best to make it a little less terrible if I can.
bee_rider | 19 hours ago
I’d expect patching existing codebases to be an eternal treadmill as better models come about.
usef- | 17 hours ago
No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.
I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.
overgard | 17 hours ago
usef- | 17 hours ago
This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.
(see: https://news.ycombinator.com/item?id=49077452 )
StilesCrisis | 21 hours ago
bluGill | 21 hours ago
benlivengood | 21 hours ago
That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.
usef- | 21 hours ago
The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)
machinist5 | 20 hours ago
1. Some do not want to use LLMs because of grave ethical concerns.
2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.
3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.
xboxnolifes | 19 hours ago
They really want that level of spend coming into the company, not going out.
slashdave | 18 hours ago
That's the stated idea. Fix code before releasing to the public.
paxys | 17 hours ago
overgard | 17 hours ago
Majromax | 4 hours ago
In the specific case of cybersecurity, this is a reasonable medium-term outcome. IMO, the cybersecurity risk is akin to the spread of a disease among an 'immune-naive' group: we can suddenly deploy much stronger attack-finding tools against large, established codebases created with much weaker security designs. The path from here to there will be rough, but it's still fundamentally easier to write secure code than it is to exploit vulnerabilities. (It's just easier yet to write insecure code, giving our status quo problem.)
For other 'safety' matters, defense isn't so easy because the attack and target are so different. An AI propaganda bot or catfisher 'attacks' slowly-evolving human culture; one that instructs on explosives or bioterrorism directly interacts with an accomplice and not a victim. If you believe that knowledge on how to build a pipe-bomb must be restricted, then giving everyone access to Fable does not mitigate the risk.
The controversial limit of this attitude is recursive self improvement and an AI singularity with potentially destructive results. Proponents of this view think that sufficiently powerful AI is risky in nearly unimaginable ways such that the capability itself is harmful. This is part (but not all) of why Fable (originally?) degraded itself when apparently assisting with AI research.
pejrich | 37 minutes ago
skybrian | 21 hours ago
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
sterlind | 21 hours ago
skybrian | 20 hours ago
fwn | 21 hours ago
skybrian | 19 hours ago
A file might contain malware, child porn, or RNA sequences for viruses.
derbOac | 17 hours ago
kypro | 21 hours ago
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
YmiYugy | 21 hours ago
sterlind | 21 hours ago
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
YmiYugy | 21 hours ago
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
lukan | 21 hours ago
Yeah, but once you know exactly where the weakness is, a weaker unrestricted model can then write that exploit for you.
gfosco | 17 hours ago
derektank | 20 hours ago
fishfasell | 21 hours ago
bigyabai | 21 hours ago
scoofy | 20 hours ago
reasonableklout | 21 hours ago
jbs789 | 20 hours ago
BLKNSLVR | 20 hours ago
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
reasonableklout | 19 hours ago
UncleOxidant | 17 hours ago
valleyer | 16 hours ago
taneq | 20 hours ago
api | 19 hours ago
nothercastle | 19 hours ago
api | 19 hours ago
scarmig | 18 hours ago
troyvit | 17 hours ago
Banditoz | 17 hours ago
scarmig | 16 hours ago
wonnage | 17 hours ago
Nowhere in GP comment was funding even mentioned
wonnage | 17 hours ago
mycall | 21 hours ago
JoshTriplett | 20 hours ago
Computer0 | 20 hours ago
anduril22 | 20 hours ago
Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"
robviren | 20 hours ago
jimbokun | 17 hours ago
eru | 17 hours ago
But that doesn't mean safety pins sped up travel.
Non-poisonous food is what economists call a 'normal good'. See https://en.wikipedia.org/wiki/Normal_good
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
davrosthedalek | 18 hours ago
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
overgard | 17 hours ago
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
This is all kayfabe and marketting.
killjoywashere | 17 hours ago
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
overgard | 17 hours ago
"The community" you're describing is, essentially, surveillance capitalism. I don't want that at all.
killjoywashere | 17 hours ago
barnabee | 14 hours ago
user43928 | 12 hours ago
Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?
As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.
rileymat2 | 18 hours ago
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
jimbokun | 17 hours ago
zephen | 17 hours ago
ElevenLathe | 17 hours ago
jimbokun | 8 hours ago
rileymat2 | 5 hours ago
rileymat2 | 17 hours ago
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
stale2002 | 17 hours ago
jimbokun | 8 hours ago
rileymat2 | 5 hours ago
Generally, instructions to create something are speech, weights could pretty clearly be seen as instructions to create a chatbot.
All I am saying is that it is complicated, they could go either way with it.
mrandish | 17 hours ago
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
wesleywt | 18 hours ago
asdf88990 | 17 hours ago
It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.
Terr_ | 17 hours ago
Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
Terr_ | 15 hours ago
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.
intrasight | 8 hours ago
Really then need to go through validation security and safety is just a component of validation validation must also check for truthfulness and correctness.
Kim_Bruning | 13 hours ago
So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.
This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.
This is called asymmetric capability, and it's probably the bigger threat.
Symmetric might be better: A rising tide lifts all ships, after all.
I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.
Majromax | 4 hours ago
Worse than that: an open-weight but safe model can be 'abliterated' to remove safety refusals using fine-tuning procedures that require a couple of orders of magnitude less compute than the original pretraining.
The 'universal evaluation' criterion then has three outcomes:
* It could become a mandatory, regulatory oversight of _all_ model training capable of hosting frontier-scale models. Since GPUs for LLM training are the same GPUs for other model training, effective mandate would require GPUs be government owned or controlled as if they were weapons of mass destruction.
* It could impose limits on release of capable open-weight models, requiring Kimi et al to prove that they cannot be made capable of abusive behaviours.
* It could be security theatre.
The AI-as-existential-risk argument points towards the first, the competition-protection argument points towards the second, and least-effort implementation would be the last.
Gigachad | 21 hours ago
dylan604 | 21 hours ago
stldev | 21 hours ago
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—
Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Please stop giving this company money, people.
0xDEAFBEAD | 16 hours ago
Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.
It's not obvious to me that the US is on an "authoritarian path".
Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-sp...
Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.
vrganj | 13 hours ago
There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
0xDEAFBEAD | 13 hours ago
If you think the market has it wrong, why don't you make money by betting against it?
https://polymarket.com/event/presidential-election-winner-20...
>There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.
vrganj | 13 hours ago
I don't just think "the market has it wrong", I think a market is wrong conceptually. It is not an epistemological tool, it's rich people gambling - a money-weighted accumulation of guesses - and I'd rather not partake.
> What is the actual per-capita rate of big flashy news stories?
What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
0xDEAFBEAD | 12 hours ago
"My beliefs are unfalsifiable"
>What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.
In any case, I didn't deny that the US had a problem with authoritarianism. I said it wasn't obvious that it was on an "authoritarian path". See for example https://www.npr.org/2026/04/04/nx-s1-5768273/after-minnesota...
I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality. You'll happily support cops arresting people for saying things online, but if cops arrest people for illegally entering a country, that somehow crosses a line into "authoritarianism". Am I right?
vrganj | 12 hours ago
The article you linked describes ICE officers killing two U.S. citizens, tear-gassing neighborhoods, and deputizing local police as a "force multiplier" to create a "sea change in local policing". You cited this as evidence the US is not on an authoritarian path. Maybe we have different definitions of "authoritarianism", but I don't see how this helps your case?
From there you pivoted to "but Obama," then "but Europe," then to psychoanalyzing my "mentality." I haven’t defended the EU, or Obama, or any censorship regime. You’re shadowboxing a partisan cartoon because the actual evidence - federal agents abducting residents, your own NPR link - is too uncomfortable to engage with directly
0xDEAFBEAD | 12 hours ago
How convenient that you continually fail to make any statement about what would falsify your beliefs.
>Obama was a war criminal bombing brown kids with drones. I care not for his medals.
Irrelevant for my point regarding whether the US is "on a path to authoritarianism". If you think any sort of immigration enforcement is unacceptably authoritarian, then the US has always been authoritarian by your definition, and the "path to authoritarianism" stuff is rather beside the point.
>It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.
What could be "engaging with the substance" more than asking how common a particular type of event actually is? Numbers are what allow us to determine what is an isolated (if unacceptable) incident and what is a common occurrence or increasing trend.
Don't tell me about "substance" when you haven't provided a single concrete data point supporting your position--I've provided multiple (NPR link, prediction market data).
All you've done in this thread is shared your own personal feelings about "armed goons" enforcing immigration law. If you're going to make your arguments primarily on the basis of your personal impressions, then yes, your ability to make those assessments fairly becomes a topic of conversation.
Furthermore, the real question of this subthread is whether the US is authoritarian relative to other countries. In which case the activities of other countries (such as European censorship) are relevant to our assessment.
There are, in fact, indices which try to compare levels of authoritarianism across countries in an apples-to-apples way, rather than doing as you do, and forming vague impressions on the basis of viral news stories. Here is one by The Economist for instance: https://en.wikipedia.org/wiki/The_Economist_Democracy_Index
Anyways good luck, at this point I'm confident that you lack the intellectual honesty to change your mind on the basis of anything I might say, so there's no point in continuing further.
cogman10 | 7 hours ago
That's a strawman and a false dilemma.
watwut | 10 hours ago
Because I am not gambler. And it is not "market" it is a casino. It does not predict, people put in bets. And like I said, while I understand gambling appeal on an emotional level, I decided to not be a gambler.
> One-in-a-million events will occur every day; they aren't necessarily representative.
It is literal official policy. Not a random event.
psychoslave | 4 hours ago
As an European citizen living in Europe, definitely yes it is, and not only for the "mere" censorship factor. Maybe not yet as down the road and maybe not going as fast as US. But that’s not something that one can really be content of.
codechicago277 | 21 hours ago
Anthropic does not support a ban on open models, except for any models that aren’t closed.
ethin | 21 hours ago
cloverich | 18 hours ago
kelnos | 21 hours ago
More self-serving trash from the US AI companies, disguised as "being reasonable".
usef- | 19 hours ago
dualvariable | 21 hours ago
Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.
ChuckMcM | 21 hours ago
kalkin | 20 hours ago
otterley | 18 hours ago
1970-01-01 | 20 hours ago
cyanydeez | 20 hours ago
Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
1970-01-01 | 20 hours ago
cyanydeez | 9 hours ago
LLMs are nothing more than a bunch of rules than can be bent the same way godel demonstrated the failability of any mathematical system.
https://en.wikipedia.org/wiki/G%C3%B6del_numbering
>A Gödel numbering can be interpreted as an encoding in which a number is assigned to each symbol of a mathematical notation, after which a sequence of natural numbers can then represent a sequence of symbols. These sequences of natural numbers can again be represented by single natural numbers, facilitating their manipulation in formal theories of arithmetic.
>Once a Gödel numbering for a formal theory is established, each inference rule of the theory can be expressed as a function on the natural numbers. If f is the Gödel mapping and r is an inference rule, then there should be some arithmetical function gr of natural numbers such that if formula C is derived from formulas A and B through an inference rule r, i.e.
https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th...
>To prove the first incompleteness theorem, Gödel demonstrated that the notion of provability within a system could be expressed purely in terms of arithmetical functions that operate on Gödel numbers of sentences of the system. Therefore, the system, which can prove certain facts about numbers, can also indirectly prove facts about its own statements, provided that it is effectively generated. Questions about the provability of statements within the system are represented as questions about the arithmetical properties of numbers themselves, which would be decidable by the system if it were complete.
sfink | 19 hours ago
bryan0 | 20 hours ago
This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.
I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
parineum | 20 hours ago
I think that's well earned.
jjfoooo4 | 19 hours ago
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
jsnell | 17 hours ago
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
cogman10 | 17 hours ago
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
cogman10 | 19 hours ago
Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?
I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
cloverich | 18 hours ago
0xDEAFBEAD | 16 hours ago
"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."
Obviously this would be rather silly.
It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
cogman10 | 8 hours ago
Which is why it wasn't the point I was making. You did an uncharitable reading of my position and then did a straw man attack.
My position is that any food regulation the likes of McDonald proposes should be looked at in the most critical and cynical light possible. They aren't making such proposals for the general health of the public, but rather to improve their own bottom line.
My position is not and never was that "we should not regulate food".
> It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy.
McDonald's uses their market position and wealth to directly lobby to government officials about food regulations. I worry about what McDonald's has to say about food because they have a VASTLY outsided ability to manipulate the regulatory system.
> We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
We can call out ineffectual and blatently self serving calls for new regulations for what they are, McDonald's trying to use regulatory capture to increase their profits and hurt their competitors.
Back on topic, that's exactly the situation with open ai.
IMO, this isn't something that's regulatable because AI models are ephemeral data that's easy to copy and replicate. No amount of US regulations can stop China from sending their dangerous models to Iran. The only thing such draconian measures accomplishes is building a moat for the likes of anthropic to shrink the number of potential customers.
If we must push out laws around AI, then those laws should at least have some chance of success. I'm all in favor of criminalizing the use of AI in cyber attacks, scamming, etc. But that's a capability that is model agnostic.
Much like I'm in favor of health and safety checks on a restaurant but I think having a mandatory McDonald's built and sold food safety device in every restaurant would be nuts. It wouldn't make food healthier it'd only serve to benefit McDonald's bottom line.
tyre | 20 hours ago
Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.
What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
sbarre | 20 hours ago
tyre | 17 hours ago
Be specific.
hephaes7us | 15 hours ago
unholiness | 5 hours ago
When open model A, fine tuned by B, is running with system prompt C, hosted by D running on E's hardware, is prompted by F to "fix this code", then escapes it's sandbox to hack into a website, or steal money to fund its subagents, or stall the waymo of the evaluator to buy time... who is responsible for the crime?
Our current legal systems are so far from ready to define what A-F are actually responsible for. We need to be moving toward defining these standards fast.
watwut | 4 hours ago
None of that is issue with a model, whether open or not. It is very much issue with code surrounding the model itself.
lenkite | 14 hours ago
anthonypasq | 4 hours ago
applfanboysbgon | 13 hours ago
It's a fucking chatbot. The industry can fix their dogshit software, anyone who doesn't can get left behind and outcompeted by those who do, and we move on with our lives.
cogman10 | 18 hours ago
This isn't something that can be regulated. Plain and simple.
If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.
The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...
Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.
crossroadsguy | 18 hours ago
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
unholiness | 5 hours ago
I think your nuclear scenario outlines precisely why this isn't true. Nuclear regulation has terms that are "good for the US" only in an absolute sense. The US would dominate even more overwhelmingly in the unregulated scenario, which gave them negotiation power to get those favorable terms. The same seems true so far with AI.
I don't think you can use the successful negotiation of nuclear regulations to argue there is "no alternative" involving regulation. I think it strongly suggests the opposite.
Of course the details matter a lot, but the core analogy holds in many scenarios. ( https://ai-2040.com/ at least attempts to lay out details, speculative as they may be)
0xDEAFBEAD | 16 hours ago
calgoo | 12 hours ago
0xDEAFBEAD | 11 hours ago
calgoo | 12 hours ago
Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.
pejrich | 26 minutes ago
dustin_vk | 20 hours ago
marcus_holmes | 19 hours ago
It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
__s | 18 hours ago
dustin_vk | 17 hours ago
api | 20 hours ago
Regulate GPUs? Ban general purpose computers?
onlyrealcuzzo | 20 hours ago
mrcwinn | 20 hours ago
I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
cogman10 | 19 hours ago
Yeah, it's too bad.
I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.
But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.
And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.
Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.
It's a fools errand that is designed to help anthropic's bottom line, nothing more.
neya | 20 hours ago
Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
wolvoleo | 19 hours ago
It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
zkmon | 19 hours ago
claaams | 19 hours ago
cloverich | 18 hours ago
goosejuice | 19 hours ago
Is the pessimistic view. Their message on safety has seemed pretty consistent to me.
"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
zmmmmm | 19 hours ago
Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
0xDEAFBEAD | 16 hours ago
Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?
zmmmmm | 11 hours ago
If Anthropic really wants to argue this type existential level risk / threat then they should face up to that meaning we can't offer them a "good faith" level of trust that they will really run the model they offered up for testing. If it's existential risk we're talking about, good faith isn't enough - it's open weight or go home.
da_chicken | 18 hours ago
reissbaker | 18 hours ago
> Open-weights models that don’t have dangerous capabilities are a public good
Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document:
- Being able to research energy, robotics, or AI is an unsafe capability
- Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities.
They want to ban pretty much anything open-source that's above cat-level intelligence.
1: https://www.anthropic.com/responsible-scaling-policy
crossroadsguy | 18 hours ago
Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
unscaled | 18 hours ago
The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.
So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.
He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
cogman10 | 18 hours ago
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
jimbokun | 17 hours ago
I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
uselessTA | 17 hours ago
Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
cogman10 | 17 hours ago
AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.
The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
overgard | 17 hours ago
d5lt5 | 15 hours ago
Simboo | 17 hours ago
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
bag_boy | 17 hours ago
From Hassabis’s essay:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
calgoo | 12 hours ago
fmap | 13 hours ago
Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.
All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...
calgoo | 12 hours ago
sc077y | 12 hours ago
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
duplessitous | 22 hours ago
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
slfnflctd | 22 hours ago
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
simplesocieties | 22 hours ago
sfblah | 22 hours ago
gr_norm | 22 hours ago
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
fwipsy | 21 hours ago
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
duplessitous | 21 hours ago
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
fwipsy | 21 hours ago
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
duplessitous | 21 hours ago
Regis, what is demanding proof while literally making things up and ignoring what actually happened?
Great, i was wrong!! Thank you, I was genuinely asking for a source in my first reply, and then you hit with "My reading" and saying it was a "skill issue". I'm not going to have a productive dialogue with someone talking in memes and being rude
The point to be made: closed source AI refused to help them fend off an attack form another closed source AI. What is the argument for closed source here other than hoping you get on some program wait list? Either way, I appreciate you correcting me; I am not trying to "win".
fwipsy | 21 hours ago
Seems a little hypocritical since you were confidently asserting that it was Mythos/Cyber5.5 also without proof.
Edit: Thanks for correcting the record in your upstream comment. I appreciate it. For the record, I was not trying to meme on you; that was the phrasing used in the original article. Just another reason that was a poor choice of source I guess.
fwipsy | 21 hours ago
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
duplessitous | 21 hours ago
"Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable."
Which just means that they're fucked when closed AI hacks them. Something that has actually happened. This isn't argument against anything other than reality. Have a day
fwipsy | 21 hours ago
I'm sorry for splitting into two threads; I understand if you need to step away from the computer for a while. To be honest, I should probably do the same.
duplessitous | 20 hours ago
You're right again about GLM 5.2 being purely post-mortem, I didn't realize that till I read the cnbc story. OpenAI, whatever they have, cracked em like it was nothing. Egg on my face, I really need to read my own articles better. Thanks for following up and educating me on this, another good reminder that I need to improve my ability to steel-man written text
fwipsy | 16 hours ago
Also, I probably overstated my claim a bit. I did some searches and I see only small-scale AI uplift for cybercriminals, even though my understanding is that open models aren't typically hard to jailbreak. Of course this is may be a result of today's guardrails; it may be that it just hasn't been caught, and it may appear later, but it still weakens my argument a great deal. I guess my support for AI regulation stems more from fears over long-shot bad outcomes (biosecurity, who knows what else) rather than cybersecurity specifically.
PLenz | 22 hours ago
tag2103 | 22 hours ago
matt_daemon | 22 hours ago
arjie | 22 hours ago
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
Philpax | 22 hours ago
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
Laurel1234 | 11 hours ago
And extolling on how much damage his product would do to society has literally been one of Anthropic's main marketing tools.
pcstl | 22 hours ago
The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
teravor | 22 hours ago
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
Cider9986 | 22 hours ago
See, the Snowden Leaks.
blackqueeriroh | 21 hours ago
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
skywhopper | 21 hours ago
fidotron | 21 hours ago
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
alightsoul | 18 hours ago
cogman10 | 22 hours ago
Yes, anthropic just put forward this argument. It's the whole point of the article.
I agree, it's absurd.
nicce | 22 hours ago
Works for both ways, which is fair?
john_strinlai | 22 hours ago
many people believe that the US will leverage AI against US citizen's interests.
impulser_ | 22 hours ago
sodapopcan | 21 hours ago
impulser_ | 21 hours ago
sodapopcan | 21 hours ago
forafistfulof | 12 hours ago
sanderjd | 21 hours ago
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
impulser_ | 21 hours ago
sanderjd | 21 hours ago
sodapopcan | 20 hours ago
Ya, I'm not American, but I have seen people say "we can vote them out" a few times now. Assuming the democrats take the next election, they are going to have a massive mess to clean up with much of the damage not even being reversible. With peoples' fickle nature and seeming that is a very big right-leaning population in the US, there's a non-zero chance the Republicans just get voted back in four years later. Whose to say?
sanderjd | 20 hours ago
To me, as an American, what has happened this past decade is that a ton of vulnerabilities in the rule of law (and other things, but this is the one I care most about) have been exposed. But it's not a given that the next Republican president will take advantage of those vulnerabilities in the way the current president has. They might end up being a reformer who seeks to fix those glitches!
But on the more pessimistic side of the same coin, it's also not a given that the next Democrat will seek to fix the glitches rather than saying "they had eight years to take advantage of these vulnerabilities, we're going to do the same to make up for that and even the playing field!".
It's just very hard to know what is going to happen from here. So I'm very sympathetic to people in other countries not trusting us.
sodapopcan | 20 hours ago
Although, again, my over understanding of your political system is poor and I just relate it to the one in my country where they hold parliament and hurl schoolyard insults at each other.
sanderjd | 19 hours ago
sodapopcan | 3 hours ago
sanderjd | an hour ago
To me, the thing that was "weak" was losing all those elections in the last cycle. But it's not "weak" to be unable to do anything with no power.
vhantz | 21 hours ago
krapp | 21 hours ago
svachalek | 21 hours ago
slfnflctd | 22 hours ago
NicuCalcea | 22 hours ago
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
sanderjd | 21 hours ago
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
alightsoul | 18 hours ago
sanderjd | 9 hours ago
I guess a day later I don't really agree with my own comment that this would be "great", but it would be a silver lining of that happening.
skywhopper | 21 hours ago
Barrin92 | 21 hours ago
97% of the world aren't US citizens and if you've taken a look at pew research surveys (or travelled to the so-called global south) you're going to be in for a bit of a shock (https://www.pewresearch.org/global/2026/07/15/people-in-many...)
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
voganmother42 | 20 hours ago
protocolture | 19 hours ago
Good. US Interests don't align with humanity.
0xDEAFBEAD | 15 hours ago
https://pbs.twimg.com/media/HNL96gAbgAA83tW.png?name=orig
protocolture | 15 hours ago
alightsoul | 18 hours ago
____mr____ | 13 hours ago
MikePlacid | 22 hours ago
TGower | 22 hours ago
natebc | 22 hours ago
I've never heard it called anything other than the CCP.
wincy | 22 hours ago
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
idiotsecant | 22 hours ago
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
pessimizer | 21 hours ago
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
ls_stats | 22 hours ago
whywhywhywhy | 22 hours ago
fuddle | 22 hours ago
Also Anthropic:
AI firm Anthropic agrees to pay authors $1.5bn to settle piracy lawsuit https://www.bbc.com/news/articles/c5y4jpg922qo
tkamado | 22 hours ago
burningion | 22 hours ago
They pirated my work and now they want government protection from other people doing the same.
jscott817 | 22 hours ago
Ekaros | 22 hours ago
nicce | 22 hours ago
buzzin__ | 22 hours ago
Police, 1980
antonvs | 17 hours ago
The frontier labs all give free access to their models. Why does “investment” change anything? Anyone who’s ever produced any content, free or otherwise, has invested in doing so.
The only plausible issue I see is that if distillation is being done by creating many free accounts to work around limits on free accounts, that’s a bit… impolite? But if they really wanted to avoid that, they could eliminate free accounts, and require users to sign a real contract governing what they can do with the model.
Of course they don’t want to do that, so they’re stuck in the same world as the rest of us, and they don’t have any real basis to complain about it without being hypocritical.
firasd | 22 hours ago
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
chatmasta | 22 hours ago
thrance | 22 hours ago
chatmasta | 22 hours ago
thrance | 22 hours ago
flexagoon | 22 hours ago
http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...
Nevermark | 22 hours ago
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
addandsubtract | 22 hours ago
bgdkbtv | 22 hours ago
Can't wait for local on machine LLMs that are on par with Opus/Fable.
bakugo | 22 hours ago
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
Imnimo | 22 hours ago
geraneum | 22 hours ago
If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because
comboy | 22 hours ago
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
polski-g | 22 hours ago
Now they have their own chips and most of Nvidia product line is internally banned.
cbreynoldson | 22 hours ago
Laurel1234 | 11 hours ago
throw1234567891 | 4 hours ago
sumedh | 21 hours ago
I never understood that argument, are you saying Chinese companies are not going to build their own chips if they get access to Nvidia chips?
matheusmoreira | 21 hours ago
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
one33seven | 13 hours ago
US hegemony is one of the biggest problems of our time.
matheusmoreira | 22 hours ago
Not even Anthropic's own Claude believes that.
knuppar | 22 hours ago
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
VariousPrograms | 22 hours ago
pascal-maker | 22 hours ago
tjwebbnorfolk | 22 hours ago
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
Ekaros | 22 hours ago
shishy | 22 hours ago
Aren't Anthropic models used in project maven: https://en.wikipedia.org/wiki/Project_Maven ?
paxys | 22 hours ago
edumucelli | 22 hours ago
ptdorf | 22 hours ago
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
devnonymous | 22 hours ago
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
orliesaurus | 22 hours ago
bicx | 22 hours ago
wasabinator | 22 hours ago
slim | 22 hours ago
Reubend | 22 hours ago
buzzin__ | 22 hours ago
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
burningion | 22 hours ago
try-working | 22 hours ago
sanxiyn | 21 hours ago
paxys | 22 hours ago
TheArcane | 20 hours ago
If it were up-to these silicon valley tech bros, they'd find a way to meter and charge for the air we breathe.
cedws | 14 hours ago
regexorcist | 9 hours ago
jameson | 22 hours ago
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
ianm218 | 9 hours ago
China is also financially repressing their people to build often useless infrastructure projects to be fair they are also not the good guys.
mayhemducks | 22 hours ago
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
tedggh | 22 hours ago
Der_Einzige | 22 hours ago
(obviously this is a joke)
quickthrowman | 22 hours ago
proxysna | 22 hours ago
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
gr_norm | 22 hours ago
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
jadar | 22 hours ago
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
nout | 21 hours ago
scilro | 22 hours ago
tensor | 22 hours ago
China hasn't threatened to annex my country yet, at least.
slashdave | 18 hours ago
alightsoul | 6 hours ago
hmokiguess | 22 hours ago
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
0xDEAFBEAD | 15 hours ago
hmokiguess | 4 hours ago
I would like to see you try to build and deploy a nuclear weapon campaign without getting noticed, you would not even be able to source the materials or get very far.
Transparency alone does not do anything, if you rely just on secrecy to protect yourself you are already extremely vulnerable. Deploying a weapon is a totally separate undertaking.
dmix | 22 hours ago
fwip | 22 hours ago
dmix | 21 hours ago
RobLach | 22 hours ago
shaongitbd | 22 hours ago
dwa3592 | 22 hours ago
pianopatrick | 22 hours ago
My current understanding is a lot of current US military problems are due to rare earths supply chains.
I don't see how AI would either help or hurt with that.
cmckn | 20 hours ago
antonvs | 17 hours ago
georgemcbay | 20 hours ago
Most current US military problems are due to the incompetence of its current civilian leadership.
jackdeansmith | 19 hours ago
pianopatrick | 19 hours ago
ianm218 | 9 hours ago
Ukraine is currently leading a long strikes campaign. You could prompt the right model to A. Find all the refineries in Russia B. For each refinery annotate all the equipment, find the most value-able C. Make a fluid model of the refinery and figure out which components most likely to set off a chain reaction if hit.
Add stuff like drone swarms monitoring the front line etc on top of that.
Supply chain issues like rare earths are real… but not really related.
Anoian | 22 hours ago
himata4113 | 22 hours ago
Demand #2 Why does this matter? The answer was that it does not. (https://news.ycombinator.com/item?id=49007610)
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
itemize123 | 12 hours ago
ls_stats | 22 hours ago
This reads like a satire. I know Dario isn't that dumb.
badatnames | 22 hours ago
timpera | 22 hours ago
reducesuffering | 22 hours ago
bigyabai | 20 hours ago
That would require me to ignore the benign reality of open LLM proliferation, so naturally most people will see this as a manipulative lie.
pibaker | 18 hours ago
infamouscow | 4 hours ago
In virtually any other situation, companies would be able to successfully grease politicians. The problem for AI companies is they spent the last several years broadcasting that their tech is going to take everyone job, and consequently their livelihood.
No matter how much you bribe a politician, you can't hold and maintain elected office when the voters overwhelmingly do not want something.
pastel8739 | 18 hours ago
jamilton | 18 hours ago
claw-el | 18 hours ago
Not that I support this, but I believe this is how comms team sees it.
throwaway27448 | 16 hours ago
xquce | 14 hours ago
throw1234567891 | 4 hours ago
trvz | 8 hours ago
tolugenius | 21 hours ago
Aperocky | 8 hours ago
tolugenius | 8 hours ago
Aperocky | 7 hours ago
Amazon is listed as signatory, I doubt microsoft just decided for them.
tolugenius | 6 hours ago
paxys | 20 hours ago
throw1234567891 | 4 hours ago
smrtinsert | 17 hours ago
anon373839 | 17 hours ago
It's WeWorse.
throwaway27448 | 16 hours ago
It does give me the faintest glimmer of hope about the people who live here, though.
transcriptase | 22 hours ago
llm_nerd | 22 hours ago
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
paxys | 22 hours ago
Madmallard | 22 hours ago
gck1 | 22 hours ago
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
horsebridge | 22 hours ago
bobjordan | 22 hours ago
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
sm-silversight | 20 hours ago
GodelNumbering | 22 hours ago
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
ASalazarMX | 22 hours ago
combobyte | 20 hours ago
In a position to lose loads of money, maybe.
Even without China eating their lunch, there's zero reason to believe Anthropic will ever be profitable.
throwaway63467 | 20 hours ago
combobyte | 17 hours ago
No amount of collusion can solve the core economics problem of compute.
bizzletk | 16 hours ago
Instead of each paying full price to generate a SOTA model in competition, they could share the result and split the cost. This gets even simpler if they merge.
dgellow | 13 hours ago
a34729t | 19 hours ago
blitzar | 14 hours ago
you can say you're pre-revenue and you're a potential pure play
dgellow | 13 hours ago
blitzar | 10 hours ago
lenerdenator | 19 hours ago
Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.
When Google wanted to operate an uncensored search engine in China, they found themselves hacked.
It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.
There are more examples but turnabout is ultimately fair play.
calgoo | 13 hours ago
erwald | 21 hours ago
doom2 | 18 hours ago
In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.
Paria_Stark | 13 hours ago
basch | 16 hours ago
banning export is short sighted and doesnt address any problem at a scale longer than months or maybe at most a few years.
Marha01 | 16 hours ago
If.
ianm218 | 10 hours ago
And China has invested immensely in sovereign chip industry for a long time and is still incredibly behind.
adrian_b | 7 hours ago
China is already able to design and fabricate good enough CPUs, GPUs, DDR5 memories and flash memories.
For now, their fabrication costs are significantly higher than those of TSMC, Intel or Samsung, because they have to use expensive workarounds for not having access to EUV lithography.
Nonetheless, where the price does not matter much, like for supercomputers or military technology, China can afford to match or exceed the US capabilities with their internal production. With the high current prices for memories, the Chinese memory companies can obtain excellent profits with their products. Which is why the US congressmen John Moolenaar (R-MI) and George Whitesides (D-CA) have requested a ban on Chinese memories, presumably at the request of Micron, who is scared of the competition.
For now, China is even ahead of Japan, though hopefully that will change soon if the Japanese Rapidus meets its targets.
While Europe is the best in a few niche domains, it has a lot of things that cannot be done within its borders, so it is much less self-sufficient than China in electronics technology.
ianm218 | 7 hours ago
No they can’t. They are seriously compute constrained and no amount of money will close the gap in the short term. The chip export controls have been incredibly effective in that regard.
I don’t have much input on US companies buying Chinese memory but there are more bottleknecks than just memory.
adrian_b | 2 hours ago
The limited production capacity means that they will not become exporters of CPUs or GPUs any time soon, but it does not mean that they cannot satisfy their internal necessities in any critical sectors. The technology that they use for CPUs and GPUs is about at the level that TSMC was 5 years ago. That may seem much, but there are a lot of people who are quite satisfied with computers older than that, and do not intend to upgrade them soon.
The main consequence of the older technology is a higher power consumption, because otherwise their designs are quite up-to-date in the attainable throughput. Therefore their datacenters must consume more energy, but that is hardly a problem in China.
The chip export controls have only forced them to design and fabricate their own CPUs and GPUs, instead of buying them from USA, and now they have become able to do this.
Without USA forcing them to do this, they might have remained dependent on imports from USA for decades, but now USA has lost the opportunity to ever play this card again. Thus USA has lost a leverage that could have been useful in a real conflict.
Such export controls are effective only for a short time, so they must be used only when there is a clear immediate goal.
pejrich | 41 minutes ago
anonzzzies | 14 hours ago
ozgung | 13 hours ago
kaon_2 | 12 hours ago
hnfong | 11 hours ago
MiSeRyDeee | 21 hours ago
usef- | 19 hours ago
niklasrde | 19 hours ago
usef- | 19 hours ago
cortesoft | 16 hours ago
robot_jesus | 14 hours ago
theptip | 18 hours ago
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
pczy | 16 hours ago
throwaw12 | 15 hours ago
No, because both can be banned using laws and Dario is against banning. It doesn't matter if you smuggle or not, law can outright ban it.
chorizo | 16 hours ago
tesnorindian | 15 hours ago
dgellow | 13 hours ago
anthonypasq | 4 hours ago
I get Anthropic is being really sleazy and annoying here, but being mad at them for not releasing an open weight model i think is unreasonable.
chorizo | 4 hours ago
fuzzfactor | 13 hours ago
From a new "fast" company arising, you can sometimes see the way that before it started up there was nothing but "narrative" and that is what established the initial business model since there was nothing else yet. After some momentum is gained whether there is a pivot or not then the narrative going forward has to be aligned with the now more-well-proven business model.
From his leadership standpoint there are 3 big recommendations right now. That's what this message is all about.
>We should not sell powerful chips or chipmaking equipment to China
Well you and who else?
If there's not already somebody who is compromising the well-being of a nation in exchange for a handful of gold, with the ever-incresing glorification of greed & dishonesty at all costs it's only a matter of time for this one.
>We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process
Wait a minute, what's always been needed by everybody are more compute-efficient processes for everything. I've mentioned this before and it's been a while but back in 1980 it took less than a year to figure out I was going to need other chips that were not regular CPUs if I was going to get the most intelligent response from the silicon on a single square-foot of PCB. At the same time it was obvious you were never going to get far without what they now call "distillation", especially with only kilobytes of memory. Otherwise you would be wasting such stupidly large amounts of memory & storage there was no way you could really call it "intelligent". Now with ML & AI on the rise again there are so many people more deeply immersed than ever, and nothing has really contradicted these basic concepts yet, which have been easily recognizable since like forever.
>All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.
Righteous concept, and I'm always in favor of 100x the amount of testing in general normally done.
But "just" test says it pretty well, and those who are gifted enough to "draw the rest of the owl" freehand can test things the most skillfully until they are blue in the face. It's not going to help if an adversary decides not to test, or to enhance these exact things so it can have some kind of competitive advantage. Amodei does not ignore this and there is a footnote about it.
People realize that some of the elements that are coming to mind were expressed in some fairly early "science-fiction" so all this is nothing new
Still looking for the most intelligent responses I can get, since way before 1980 ;)
axegon_ | 12 hours ago
irenaeus | 8 hours ago
ausbah | 22 hours ago
ofc half of them are of the ai rationalist lesswrong crowd so i think they’ve always been a little of their rocker
iamdamian | 22 hours ago
alach11 | 22 hours ago
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
nharziro | 22 hours ago
ajyoon | 22 hours ago
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
artrockalter | 21 hours ago
ajyoon | 21 hours ago
artrockalter | 21 hours ago
ajyoon | 21 hours ago
verdverm | 19 hours ago
tekacs | 20 hours ago
For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
dwaltrip | 20 hours ago
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
sudosysgen | 19 hours ago
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
pastel8739 | 18 hours ago
sudosysgen | 17 hours ago
Der_Einzige | 15 hours ago
Valakas_ | 15 hours ago
tekacs | 14 hours ago
thesumofall | 12 hours ago
Ajedi32 | 6 hours ago
If AI makes finding software vulnerabilities easier, then we should deploy it widely to find as many vulnerabilities as possible and fix them, not bury our heads in the sand and pretend the vulnerabilities don't exist as long as nobody knows about them. That's just the same "security by obscurity" strategy that has been tried and failed time and time again.
Majromax | 4 hours ago
> You are suggesting this isn't correct?
The intuition behind that is applicable only when correctness is stochastic. If you need to be waved in by a security guard, then one fake mustache might be the difference between being granted or denied entry. However, a keypad either works or it doesn't; entering the wrong PIN is guaranteed refusal.
The other breach of that intuition is defense in depth. Secure systems don't generally rely on a single binary trusted/untrusted status; the classified building still locks its interior doors. This is the part that has – in my view temporarily – changed most with frontier models, in that they are much more skilled at chaining together vulnerabilities than previous models (and much faster about it than human experts, even if potentially less skilled). If a system has a latent (0-day) vulnerability 50% of the time, then 10 independent layers would imply a ≈ 1/1000 chance that a critical compromise is possible.
However, these independent layers don't currently happen in practice because it's easier to write insecure code than secure code. With luck, modest discipline, and defensive use of frontier models I think that this gap will narrow with time, in much the same way that it would be plainly crazy to deploy root access via telnet today.
paxys | 19 hours ago
shepherdjerred | 18 hours ago
But that, of course, is not going to survive contact with reality
foo12bar | 15 hours ago
xiphias2 | 12 hours ago
It was always easier than making a system secure.
heyjstn | 12 hours ago
alienbaby | 19 hours ago
akersten | 19 hours ago
the frontier models refused because their cyber detector went off
they had to use GLM 5.2 instead
usef- | 19 hours ago
marcus_holmes | 18 hours ago
paxys | 19 hours ago
paxys | 18 hours ago
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
varenc | 17 hours ago
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
Majromax | 4 hours ago
I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link.
In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration.
I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers.
> Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
vultour | 7 hours ago
fidotron | 21 hours ago
Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )
> and cyber-offense capabilities?
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
ajyoon | 21 hours ago
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
fidotron | 21 hours ago
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
ajyoon | 21 hours ago
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
fidotron | 21 hours ago
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
jefftk | 21 hours ago
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
fidotron | 21 hours ago
No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
jefftk | 20 hours ago
> No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
That's a different attack. I'm talking about their 1993 anthrax attack: https://pmc.ncbi.nlm.nih.gov/articles/PMC3322761/
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
fidotron | 20 hours ago
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
rescbr | 19 hours ago
Stuff is known but not acted upon for various reasons.
gck1 | 21 hours ago
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
ajyoon | 21 hours ago
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
gck1 | 21 hours ago
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
ajyoon | 21 hours ago
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
gck1 | 20 hours ago
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
CubsFan1060 | 21 hours ago
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
ajyoon | 21 hours ago
niwtsol | 17 hours ago
fwn | 21 hours ago
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
manoDev | 20 hours ago
monk_grilla | 19 hours ago
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
paxys | 18 hours ago
boinkboink78912 | 18 hours ago
Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage
hnfong | 11 hours ago
The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
verdverm | 20 hours ago
The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.
In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.
jackdeansmith | 19 hours ago
verdverm | 19 hours ago
vitalyan8184 | 20 hours ago
...general-purpose computers
...unbreakable encryption
...unbackdoored communication
...unkillswitched vehicles
...unsurveiled dwellings
>what should be done about ...?
nothing
>Do you seriously want this level of capabilities to be generally available with no guardrails?
yes
jackdeansmith | 19 hours ago
jjfoooo4 | 19 hours ago
davrosthedalek | 16 hours ago
valcron1000 | 20 hours ago
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
rubslopes | 20 hours ago
verdverm | 20 hours ago
I would not be surprised if the same incentives are created by the US for Ai
paxys | 19 hours ago
rubslopes | 3 hours ago
boinkboink78912 | 18 hours ago
A complete failure at actually preventing non-proliferation.
idontbelonghere | 5 hours ago
pylua | 20 hours ago
The software has to be built better.
doginasuit | 20 hours ago
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
pylua | 19 hours ago
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Right now it’s being used as a bandaid.
pastel8739 | 18 hours ago
naiveter | 16 hours ago
marcus_holmes | 18 hours ago
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
pylua | 18 hours ago
le-mark | 20 hours ago
Bad actors WILL have access. The question is will these mega corps stop innovation?
rstuart4133 | 20 hours ago
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
dolebirchwood | 20 hours ago
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
adastra22 | 19 hours ago
dolebirchwood | 19 hours ago
paxys | 18 hours ago
adastra22 | 20 hours ago
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
urams | 19 hours ago
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
adastra22 | 9 hours ago
idontbelonghere | 5 hours ago
gr_norm | 18 hours ago
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
uncivilized | 18 hours ago
xyzsparetimexyz | 14 hours ago
adastra22 | 9 hours ago
xyzsparetimexyz | 8 hours ago
qnleigh | 13 hours ago
gr_norm | 12 hours ago
The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.
[1] https://journals.plos.org/plosone/article?id=10.1371/journal...
BeetleB | 19 hours ago
We'll be fine.
tacet | 19 hours ago
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
consumer451 | 19 hours ago
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
https://en.wikipedia.org/wiki/Biological_Weapons_Convention
tacet | 18 hours ago
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
consumer451 | 2 hours ago
I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
johncolanduoni | 17 hours ago
mnicky | 14 hours ago
consumer451 | an hour ago
The issue is non-state actors. That moves it from a ~hunderd to many billions.
BTW, according to my FOSS religious beliefs, I should be making the other side of the argument. This whole thing is tough.
baddash | 19 hours ago
boinkboink78912 | 18 hours ago
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
dools | 18 hours ago
txrx0000 | 18 hours ago
https://news.ycombinator.com/item?id=49078376
----
And related thoughts on past posts:
https://news.ycombinator.com/item?id=49034988
https://news.ycombinator.com/item?id=48516722
overgard | 17 hours ago
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
CapsAdmin | 14 hours ago
Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.
I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.
Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
e_l | 17 hours ago
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
waterTanuki | 16 hours ago
__MatrixMan__ | 16 hours ago
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
zarzavat | 16 hours ago
> Something should be done
and
> Something can be done
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
Madmallard | 16 hours ago
But more people having access to the potential tools for defensive is the best possible scenario.
Every other scenario is worse off for everyone except for those with enough money to do something about it.
alevskaya | 15 hours ago
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
mnicky | 14 hours ago
As an expert, could you also provide your arguments please?
inciampati | 13 hours ago
Ey7NFZ3P0nzAe | 12 hours ago
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
Thoughts?
rdedev | 12 hours ago
inciampati | 12 hours ago
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
alt227 | 13 hours ago
Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?
rdedev | 12 hours ago
qnleigh | 13 hours ago
inciampati | 11 hours ago
At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.
Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.
Diogenesian | 10 hours ago
Jyaif | 4 hours ago
Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.
People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.
No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)
gorgoiler | 14 hours ago
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
impossiblefork | 13 hours ago
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
qnleigh | 13 hours ago
nevertoolate | 12 hours ago
I haven't read cynical comments, just ones pointing out that the article is cynical itself.
> addresses the core point
No it doesn't address anything, it is fear mongering question.
> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns
Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.
> you need to suggest an alternative plan that addresses these concerns
I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.
Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]
[1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...
pbasista | 9 hours ago
In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI.
When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer.
These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible.
If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood.
The important part is that with the closed, black box, proprietary models, one can never know what they are being served.
wilde | 8 hours ago
rnewme | 8 hours ago
CJefferson | 7 hours ago
All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.
locusofself | 22 hours ago
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
kelvinjps10 | 22 hours ago
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
2.
sanderjd | 21 hours ago
gr_norm | 21 hours ago
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
_jab | 22 hours ago
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
tkamado | 21 hours ago
so Anthropic's ask is for US gov to ban open weight models so that its growth (and IPO) is not affected
sfink | 18 hours ago
0xDEAFBEAD | 15 hours ago
twobitshifter | 22 hours ago
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
system-error | 21 hours ago
gck1 | 21 hours ago
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
If it's the former, then why publish this?
skywhopper | 21 hours ago
c-hendricks | 21 hours ago
chrismsimpson | 21 hours ago
blackqueeriroh | 21 hours ago
The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
China is as worse as the United States, if not more worse, by many measures.
chrismsimpson | 21 hours ago
China is nowhere near as bad as the US at this point. The rest of the world is changing lanes to not be implicated in your car crash of a country.
antonvs | 19 hours ago
It absolutely does make China better relatively, i.e. by comparison to the US.
Many of the criticisms previously leveled at China are now similarly applicable to the US in a way that they weren’t previously. Human rights violations? The US is currently the major global supporter of an ongoing genocide, and it even kills and deports its own citizens for political reasons. Political opponents are investigated by the state. When it comes to wars and other interference with other countries - like kidnapping a president - the US is far worse than China at the moment.
In which ways is China “more worse” right now?
stuartmemo | 21 hours ago
fearnot | 21 hours ago
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
Exactly.
llelouch | 20 hours ago
bigyabai | 20 hours ago
Anthropic's basis of assumption is the insinuation that LLMs can do things that we've never seen before, and that they can't tell us what it is. It sounds like you're also siding with an organization that has no evidence and relies on validating their own assumptions.
maziyar | 21 hours ago
whalesalad | 21 hours ago
stratos123 | 15 hours ago
vhantz | 21 hours ago
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
flossly | 21 hours ago
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
petcat | 21 hours ago
verdverm | 21 hours ago
- One can load them up in a model explorer to see the layers and other components, how it is designed
- One can fine tune the models, which requires adding LoRA to the model and then running some training iterations
nwiswell | 20 hours ago
verdverm | 20 hours ago
we run and change llm models with a variety of tools
nwiswell | 20 hours ago
Cloud:
- You cannot directly execute a remotely-hosted program.
- You cannot run inference on an API-served model.
---
Closed-source:
- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.
---
Truly open:
- You can freely modify the source and produce new binaries.
- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.
---
To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.
The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.
rstuart4133 | 20 hours ago
Time have changed. This should be:
Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.
don_esteban | 5 hours ago
matheusmoreira | 21 hours ago
rstuart4133 | 20 hours ago
Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.
usef- | 19 hours ago
0cf8612b2e1e | 20 hours ago
Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.
purpleflashing | 15 hours ago
gbalduzzi | 13 hours ago
That aspect is probably more important for an open model then the source materials
m_ke | 21 hours ago
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
sterlind | 21 hours ago
the West could retaliate by halting shipments of photoresist and other materials to China.
meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.
the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.
chrismsimpson | 21 hours ago
US is going to find itself isolated and irrelevant. And not a moment too soon.
matheusmoreira | 21 hours ago
greekrich92 | 20 hours ago
thesmtsolver2 | 19 hours ago
owebmaster | 17 hours ago
tw1984 | 15 hours ago
Foobar8568 | 9 hours ago
one33seven | 13 hours ago
TheArcane | 20 hours ago
lenerdenator | 19 hours ago
[0] https://www.indiatvnews.com/news/world/chinese-manager-beats... (possibly NSFW, I can't see the video in my browser but be warned)
vhantz | 18 hours ago
CaptWorld | 15 hours ago
klibertp | 3 hours ago
Sure, Iraq and Afghanistan definitely benefited a lot from being bombed, occupied, and then handed over to even worse tyrants when the West got bored.
No, sorry, but "capitalism and democracy" is not the right answer everywhere, and when they're not, pushing them by force is no better than forcing communism. In general, forcing an incompatible ideology on people historically and culturally opposed to it will end in tragedy, no matter how great that ideology may be.
pmontra | 13 hours ago
thesmtsolver2 | 19 hours ago
There is no HN equivalent in China where users advocate that US hegemony will be better.
alightsoul | 18 hours ago
ido | 16 hours ago
alightsoul | 16 hours ago
gpvos | 16 hours ago
CaptWorld | 16 hours ago
ido | 14 hours ago
verdverm | 20 hours ago
matheusmoreira | 20 hours ago
verdverm | 19 hours ago
We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?
matheusmoreira | 19 hours ago
Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.
verdverm | 18 hours ago
To reduce a culture and country like this comes off as bigotry
In example, Brazil has the #3 airplane producer in the world
Or consider why the US is doing all it can to prevent their payment system from becoming widely used?
I was intentional about putting the perspective in the future over today as the poles of the world are evolving
owebmaster | 17 hours ago
verdverm | 17 hours ago
Der_Einzige | 15 hours ago
matheusmoreira | 14 hours ago
This thread is more like it:
https://news.ycombinator.com/item?id=45731174
matheusmoreira | 15 hours ago
All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.
I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.
verdverm | 6 hours ago
Timothy Snyder, in On Tyranny, has a chapter about staying in touch with friends from other countries, and while we are not friends, this thread is in that spirit and I have learned things from you, so thank you!
a34729t | 20 hours ago
marcus_holmes | 18 hours ago
nativeit | 18 hours ago
https://www.gao.gov/products/gao-24-106866
fnord77 | 17 hours ago
stealth colonialism
d5lt5 | 16 hours ago
CaptWorld | 16 hours ago
marcus_holmes | 15 hours ago
CaptWorld | 15 hours ago
marcus_holmes | 15 hours ago
We're told a relatively benign version of our history. You kinda have to travel to the countries involved to get the real version.
rmunn | 15 hours ago
GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.
kajaktum | 14 hours ago
edoceo | 15 hours ago
CaptWorld | 13 hours ago
amunozo | 14 hours ago
CaptWorld | 13 hours ago
amunozo | 10 hours ago
But second, Russia is not Eastern, it was more European than anything else until the Soviet Union. And Japanese Empire drew inspiration in the Western nations, even if they took it a even more horrible twist. So blaming this on "the East" is racist and reductionist.
CaptWorld | 8 hours ago
even the AI that was trained on mostly using western data and infrastructure that chinese models distill and all..
amunozo | an hour ago
subscribed | 12 hours ago
CaptWorld | 8 hours ago
marcus_holmes | 16 hours ago
Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.
There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.
You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.
I'm kinda "both can be true, but either are better than how we did it"
Der_Einzige | 15 hours ago
They'll usually refer to China in terms like the "thousand year enemy".
FooBarWidget | 14 hours ago
amunozo | 14 hours ago
brabel | 14 hours ago
amunozo | 13 hours ago
cassianoleal | 13 hours ago
Get the F off our lawn and stop dumping your rubbish on it!
brazukadev | 8 hours ago
Go tell them, half of my country wants to CONTINUE being the USA backyard.
cassianoleal | 8 hours ago
jampekka | 14 hours ago
https://www.iseas.edu.sg/wp-content/uploads/2026/03/The-Stat...
worldthruword | 13 hours ago
ifwinterco | 13 hours ago
They’re culturally part of the sinosphere but of course constantly living in the shadow of your much, much bigger neighbour to the north does breed some ill feeling.
The USA on the other hand, well, they’re not particularly popular either for obvious reasons
ethbr1 | 8 hours ago
Which was a dumb move because, given Chinese economic superiority, they could have just quietly negotiated 99 year military base leases and oil/gas extraction with the UN-recognized territorial owners.
Same outcome, less bad blood.
Seems a bit of an own goal to militarily force the issue and antagonize its neighbors, who it's trying to convert to its sphere of influence.
ifwinterco | 3 hours ago
The mask (if there ever was one) slipped and they revealed a lot of information they probably should have kept hidden about their intentions for no real reason.
However they did eventually realise this, they've completely changed tack and they're having a lot more success (helped by the US adopting their own failed policy).
You're right though, I'd imagine politicians in SEA countries haven't forgotten
fakedang | 15 hours ago
On the other hand, every one in Asia is wary of too much Chinese presence and influence.
gpvos | 16 hours ago
MaxPock | 15 hours ago
gpvos | 15 hours ago
ninjin | 16 hours ago
To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.
I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.
marcus_holmes | 15 hours ago
ninjin | 15 hours ago
The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.
brabel | 14 hours ago
ninjin | 13 hours ago
Rather, I think we should look hard into ourselves and what is good and bad about the current world order. For example, the people of the RoC have the right to determine which direction they want to go. Regardless of the chauvinistic rhetoric coming out of Beijing. We should all stand up for this, because it is a universal right that we want everyone to enjoy. Similarly, we should push against the Eleven-dash line and support the 2013 ruling. The list goes on and I am sure these issues can be resolved without an outright war if we are careful, yet firm, in our beliefs and also diplomatically preemptive and thoughtful.
Being concerned about PRC imperialism should not be mistaken for the position that their people should "know their place" and be suppressed back to the stone age. They have the right to enjoy the fruits of their labour and pursue happiness, just like everyone else. We simply must be there to remind them (just like we must remind ourselves) that the course of humanity is a collective project if we are to stay clear of the darker sides of our nature as we venture together into the future.
brabel | 12 hours ago
The Chinese claims on the South Pacific Islands seem really similar to me.
Taiwan seems like a wholy different matter. It was united with China for hundreds of years until the Japanese colonialists took over. It united again with China after WWII but split up after a few years because of the Chinese civil war (notice it was an internal war). I think it's just fair that China wants to re-unite with Taiwan, though I definitely don't support a military takeover. Hopefully a solution similar to what was done in Hong Kong can be found. The Tibet region had a similar history and it's definitely unfortunate that China had to use military force to bring it under its own control (arguably completely unnecessarily - China would be just as strong today without it), but it's kind of understandable in the context of the time (China was trying to recover from centuries of being preyed on by other nations).
I am saying this because I can't agree that China is acting imperialistic - it's basically claiming sovereignty over its own historical lands - which were taken away from them by force by foreign colonial powers. But I admit that, if you go back far enough, nearly all land was once taken over by aggressors - the USA being just a more recent example of that.
Anyway, thanks for not being an absolutist and trying to understand the "other" side (I must acknowledge I have no relation to China whatsoever, in fact I am from South America and live in Europe).
loudmax | 8 hours ago
Citizens of Hong Kong lost their right to free speech and their ability to select their own leaders. If you publicly criticize Xi Jinping in Hong Kong you will go to jail. If you advocate democratic rights in Hong Kong you will go to jail.
China's claims on Taiwan are ethno-nationalist. Ethno-nationalism should be rejected in all forms because it is a rejection of fundamental human rights. Taiwan deserves the world's support because it is functioning democracy. That makes all the difference.
ninjin | 7 hours ago
About Taiwan. I find the claim that since an absolute monarchy controlled the island 150 years ago, that then a government which was the result of two (is my count correct?) revolutions overthrowing that monarchy and then another government, a government which failed to conquer the land by military means by 1950, and now after people have lived independently for over 75 years (over 25 of which as a democracy) that said government has any right to dictate how said people should live to be simply absurd. If the people of the RoC wants to join with the PRC, that is for them to decide through their own decision processes. Historical claims like this may make sense for unpopulated tracts of land, but here we are talking about the rights to self-determination of more than twenty million people of which the vast majority were born well after 1950. This would set a terrible precedence and, frankly, it feels akin to how emperors and kings of old asserted their "rights" and not how we move towards a more just world.
Also, Hong Kong? If anything, Hong Kong shows that the PRC is a poor custodian for a pluralistic country with multiple parallel systems. I once thought it reasonable for Hong Kong to be "returned" after the historical travesty that were the Opium Wars, but I have heard enough first-hand accounts of the suffering and tragedy that unfolded over the last ten years to reconsider whether I prioritise history over the people that are alive here and now. It was not that Mao and Xi "unfortunately had to use force" against Tibet and Hong Kong. These were calculated choices on their part and history shall judge them the same way we judge any other oppressor for their moral failures.
Khaine | 15 hours ago
elisbce | 14 hours ago
chrismsimpson | 18 hours ago
petre | 17 hours ago
FooBarWidget | 14 hours ago
verdverm | 21 hours ago
China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.
0cf8612b2e1e | 20 hours ago
Which is to say, given internal subsidies, the US could eventually produce some on its own.
Unless I misunderstood the situation.
verdverm | 20 hours ago
1. China controls ~90% today
2. The US will find it difficult to build out because of how dirty and environmentally damaging mining and processing are.
I did see some research last week about a better way to process rare earths, but it is still research and will need to be industrialized.
Regardless, it will take many years (decade+?) to become self sufficient and China is already willing to and increasingly restricting them
verdverm | 4 hours ago
https://www.reuters.com/legal/government/trump-may-need-allo...
tw1984 | 15 hours ago
Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?
soperj | 16 hours ago
rubslopes | 20 hours ago
cassianoleal | 13 hours ago
brokenmachine | 20 hours ago
crossroadsguy | 17 hours ago
brokenmachine | 17 hours ago
crossroadsguy | 16 hours ago
mctaylor | 16 hours ago
"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."
alightsoul | 16 hours ago
PangXJ | 17 hours ago
windexh8er | 20 hours ago
The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.
But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).
The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.
ethbr1 | 8 hours ago
Exactly. The cries in favor of distillation regulation from the US AI companies ring hollow and fearful.
OpenAI and Anthropic didn't realize that distillation was going to be so (a) effective and (b) un-technically-stoppable at scale.
Now they're seeing their IPOs at risk and clutching at governmental straws.
Dario's argument is transparently working backwards from {protect Anthropic's economic model} <- {need government regulation} <- {justify government regulation via AI fears} <- {we love open models, but so sorry they can't pass regulation}.
If the rise of the web in the 90s taught us anything, it should have been that companies that take economic reality as it exists thrive, while those that predicate their value on regulation fail.
If distillation at scale works and is technically feasible? That's reality. Deal with it.
wbl | 17 hours ago
otabdeveloper4 | 16 hours ago
fakedang | 15 hours ago
wbl | 6 hours ago
gr_norm | 21 hours ago
SubiculumCode | 16 hours ago
hn_submit | 20 hours ago
matheusmoreira | 19 hours ago
Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.
uv-depression | 17 hours ago
Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.
matheusmoreira | 16 hours ago
There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.
There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.
There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.
Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.
No doubt there are many other fates that escape my feeble attempts at foresight...
sciencejerk | 15 hours ago
lenerdenator | 19 hours ago
If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?
While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.
Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.
alok-g | 2 hours ago
The market dynamics would find good balance automatically Meanwhile, good market practices and fair competition should be continued.
twelvechairs | 17 hours ago
The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though
Petersipoi | 16 hours ago
soperj | 16 hours ago
Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?
Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?
testaccount28 | 15 hours ago
because that's what they think is best. what point are you trying to make? that countries don't pursue 'profit' at the expense of all other concerns?
vincnetas | 15 hours ago
shsjidhs | 15 hours ago
Oh no, you heretic, The People freed themselves and it was ordained by God, you see. It is through the righteous might of The Greatest Experiment in The History of the World that they showed The World what Freedom really meant.
altmanaltman | 15 hours ago
While there is no "job" of a country, it is natural that each one will work for its own best interest and any moves it makes in the global world is due to their own vested interests in some way including helping Haiti, France helpong Us and the world supporting Ukraine.
Why would a country act against its interest or just randomly? Alliances and corporation are a part of politics
psychoslave | 13 hours ago
Because country as an entity is a mental construction for which "interest" is a categorical error. It’s certainly a useful concept, but pretending it has interests like some human individual can have have interests. Sure it can serve as rhetorical facility to sell some arguments.
Countries don’t have interests. Some people willing to take control of other people encompassed in that "country" groups have interests, and they don’t necessarily align with best interests of everyone or even majority in each of these groups.
dgellow | 13 hours ago
psychoslave | 11 hours ago
soperj | 5 hours ago
So why is the US supporting Russia again?
> Having the British lose the US was a good thing for France
This seems a bit reductionist. Why would they give the Statue Liberty to the US if they were only in it for the British loss?
oneshtein | 5 hours ago
Davidzheng | 16 hours ago
cmrdporcupine | 15 hours ago
There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.
Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.
That's why it matters?
dgellow | 13 hours ago
spaceman_2020 | 15 hours ago
vlovich123 | 15 hours ago
But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.
They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.
cpursley | 15 hours ago
21asdffdsa12 | 14 hours ago
well_ackshually | 14 hours ago
dudefeliciano | 13 hours ago
That's pre 1945 style thinking, which is coming back with a vengeance.
Foobar8568 | 13 hours ago
What about women too ? Especially in red states.
cpursley | 7 hours ago
jmyeet | 6 hours ago
But I hate this talking point for one primary reason: almost nobody who brings it up actually cares about the Uyghurs. It's just a talkijng point.
How do I know this? Because of Palestine. If Benjamin Netanyahu turned around tomorrow and treated Palestinians as Uyghurs were and are treated and developed the region like China does today then he would win the Nobel Peace Prize and Palestinians would be unquestionably better off. It wouldn't be sufficient mind you.
Israel's crimes in Palestine are America's crimes because Israel could not exist without the economic and military support of the US, political cover in the UN and international community and the political will of Us domestically. It would collapse tomorrow if the US withdraw support.
So I don't want to hear a thing from China hawks and skeptics about Xinjiang unless they're louder what Israel is doing to the region.
cpursley | 4 hours ago
Exactly, it's just people parroting psyop narrative talking points, Taiwan is another. Repeat something enough and people will eventually believe it.
There's a reason other nations are choosing China over the US/EU (including Muslim and Turkish ones) - there's no moralizing, political requirements, nor bombs.
greenchair | 10 hours ago
don_esteban | 14 hours ago
vector_spaces | 14 hours ago
> Operation Condor (Spanish: Operación Cóndor; Portuguese: Operação Condor) was a campaign of political repression by the right-wing dictatorships of the Southern Cone of South America, involving intelligence operations, coups, and assassinations of left-wing sympathizers in South America. Operation Condor formally existed from 1975 to 1983. Condor was formally created in November 1975, when Chilean dictator Augusto Pinochet's spy chief, Manuel Contreras, invited 50 intelligence officers from Argentina, Brazil, Bolivia, Chile, Paraguay, and Uruguay to the Army War Academy in Santiago, Chile. The operation was backed by the United States, which financed the covert operations. France is alleged to have collaborated but has denied involvement. The operation ended with the fall of the Argentine junta in 1983.
https://en.wikipedia.org/wiki/Operation_Condor
vlovich123 | 14 hours ago
The claim is they don’t engage in warfare when they do, just exclusively in the digital domain for now. Believing it’ll stay there is naive.
Playing the moral superiority game is uninteresting in either direction. Games of power inherently are devoid of morality.
darkwater | 14 hours ago
vlovich123 | 13 hours ago
darkwater | 13 hours ago
nswango | 14 hours ago
If someone is vocally complaining about country A doing some oppressive activity, while ignoring country B doing 100x the same activity, the comparison does not forgive or excuse country A, or make them right or admirable.
But it does allow us to conclude that the person complaining is biased. Either they don't care about the oppressive activity and want to attack country A for some other reason. Or they have a particular fondness for country B and will never accept that it does wrong. Or some variation on these.
vlovich123 | 13 hours ago
randunel | 13 hours ago
> de minimis non curat praetor
Which basically means judges are not interested in minor disputes. When you add the USA to the mix, China's actions pale in comparison to the point of being negligible, no point in discussing them given the scale of USA's warmongering, overseas and local crimes. "They never interfered in [...]" is an exaggeration which shows the lack of importance of China's crimes when compared to the USA's, that's all.
The "whataboutism" here isn't a deflection strategy, it shows the hypocrisy and correctly minimises the scale of China's actions in comparison to the hypocrites'.
Cookingboy | 13 hours ago
Come on, are you seriously equating hacking to literally bombing and killing people? That's just arguing in very bad faith.
Calling hacking "warfare" is just intellectually dishonest, if not downright disgusting toward the actual victims of American wars.
teiferer | 14 hours ago
For Americans the answer is obvious since they are biased in this.
For everybody else, your question is valid.
Gud | 14 hours ago
clarionbell | 11 hours ago
Gud | 11 hours ago
saidnooneever | 14 hours ago
the question is valid for anyone who cares for more than their own ass
neuroticnews25 | 13 hours ago
nlehuen | 13 hours ago
neuroticnews25 | 12 hours ago
mldqj | 12 hours ago
neuroticnews25 | 11 hours ago
Doch88 | 13 hours ago
nswango | 10 hours ago
Foobar8568 | 13 hours ago
All my life I heard American boasting their constitutions but now it's as worthy of my toilet papers.
neal_jones | 11 hours ago
panicinducer | 12 hours ago
fragmede | 12 hours ago
panicinducer | 9 hours ago
nylonstrung | 8 hours ago
dudefeliciano | 4 hours ago
this is a hilariously naive rewriting of history, the government was very reluctant to make those changes. It's thanks to popular movements that those changes were made.
isoprophlex | 14 hours ago
minraws | 14 hours ago
Wrongs don't negate each other.
guax | 13 hours ago
minraws | 13 hours ago
But in general if you ask folks in south east Asia you will find they are likely to be more concerned with China than US except this maddening Oil Crisis.
Although I think people now are very afraid/wary of both evils.
The US govt should think why people in all parts of the world including US itself feel just if not more threatened by them, than China and other evils we have floating around.
I would like to hope my American friends didn't vote for this madness or maybe they did I have read DHH's tweets.
rob74 | 12 hours ago
p2detar | 9 hours ago
We still don't know where and how Covid came to be, so no - hard disagree.
rob74 | 9 hours ago
jeltz | 9 hours ago
p2detar | 8 hours ago
> If there is a cover up it is to hide incompetence.
Which is my point to GP's "reliable" argument. China is not reliable and has its own interests at play just like the US or any other state for that matter, regardless of its international partnerships. It does want to project itself as "reliable" - that I can't deny.
0 - https://www.who.int/publications/i/item/who-convened-global-...
Foobar8568 | 9 hours ago
ethbr1 | 8 hours ago
Here's the dangerous thing about that: modern CCP China is a newly ascendant world power.
Say what you want about the US today, but when was it at isn't interventionist worst historically-speaking?
Countries don't go to war because their leaders take them to war: they go to war because a sufficient chunk of internal interests support war.
And China has a rather politically-powerful military. Although Xi's recent housecleaning substantially weakened that power.
If China goes to war, it's probably going to be because their economy is in tatters.
rob74 | 7 hours ago
Well, they probably also have to factor in whether invading Taiwan is likely to improve that situation, and I would argue that it's not - China is currently exporting stuff all over the world, so economic sanctions would be very painful for them. And there are lots of countries who would like to have a slice of the manufacturing that China is currently hogging...
ethbr1 | 4 hours ago
They go to war because a large chunk of their population is pissed off about the economy.
See also: US
RobotToaster | 7 hours ago
aprentic | 7 hours ago
There are some internal policy papers to that effect.
monooso | 6 hours ago
roysting | 8 hours ago
scottyah | 3 hours ago
roysting | 8 hours ago
You are allowing them to define a ridiculous standard by agreeing with them that not only is it in fact “two evils” but what qualifies as evil. They’ll tell you…always only when called out, mind you…all and any meddling (also something they will then define with the same kind of falls equivalence loop) is evils, when that is simply not true, not is any kind of effort to influence the same thing as actual meddling, not us it always evil.
I could go on, but I’m sure you get the point, the narcissistic personality or their little zombies in most cases will constantly shift and change things all to avoid you from being able to get a clear focus on the truth, something their explosively allergic to.
As an American child of the Empire with global perspective and very high access perspective, there is no other entity besides the cabal that controls the USA and acts flying its banner; that is more deadly, more meddling, more conniving, more evil, more supremacist, and more vile.
LtWorf | 13 hours ago
I eagerly await for the USA citizens to downvote this in about 5 hours.
khriss | 13 hours ago
bambax | 12 hours ago
Yes we do. We should evaluate threats and choose our allies carefully.
wongarsu | 12 hours ago
repler | 9 hours ago
All technology can be used for good or used for evil depending on who is using it and how. From fire to internal combustion engines to nuclear energy.
tempfile | 11 hours ago
jb1991 | 12 hours ago
—- “But look at the United States! What about that?”
Is this not just whataboutism? It never adds much to the discussion.
Doch88 | 12 hours ago
In this case it's just pointing out the propaganda and the contradiction.
cmpxchg8b | 12 hours ago
tempfile | 11 hours ago
stogot | 8 hours ago
dominotw | 7 hours ago
21asdffdsa12 | 14 hours ago
potamic | 14 hours ago
pell | 10 hours ago
potamic | 8 hours ago
jeltz | 8 hours ago
spaceman_2020 | 14 hours ago
The 2020 skirmish was way smaller with single digit casualties. Again, almost no civilian casualties because it's a largely empty region in the first place
You really can't compare this to the kind of wars America has started. Casualties - both civilian and military - stood in the hundreds of thousands
microtonal | 13 hours ago
spaceman_2020 | 11 hours ago
America's wars have been in countries so far away that they could've never posed any threat whatsoever to America as a nation.
Really not the same thing
vlovich123 | 4 hours ago
That’s fallacious reasoning that ignores how easy travel is and how cheap an attack can cause massive damage. 9/11 was massive attack from countries far away. We regularly see commercial drones being used in warfare causing huge damage and casualties. There was the 2012 terrorist attack on our CIA facilities in Benghazi.
Terrorism entrenched abroad impacts our security. Is the response we take incorrect and makes it worse / are there things we do to provoke this? Some things yes, other things are like having Israel as an ally which is a culturally, socially and economically aligned power in the region even though it’s hated in the region.
Also don’t forget the US is a major maritime power. China is trying to be, Russia is not. Maritime powers force project more broadly because a) is the nature of being a maritime power b) you have to protect shipping to make sure trade routes are uninmpeded. B is particularly important as it also helps explain the wars (eg the Iran war is about the strait and trade and who collects money, not really about nuclear capabilities).
teiferer | 14 hours ago
Wow, that newest of Trump's distraction talking points made it surprisingly quickly to the uncontended (?) and commonly accepted facts in HN conversations.
vlovich123 | 13 hours ago
You’re very misinformed if you think Trump’s latest blathering are completely wrong. I generally don’t pay attention to him so I don’t know exactly how he lied or exaggerated but I’m sure he did. It doesn’t negate the real and active influence operations China is engaged in
div | 12 hours ago
teiferer | 11 hours ago
gadders | 14 hours ago
Parae | 12 hours ago
gadders | 11 hours ago
aa-jv | 9 hours ago
gadders | 9 hours ago
aa-jv | 8 hours ago
PunchyHamster | 13 hours ago
And this comment isn't about putting any good light on China. Lately US acts like they want to compensate for Russia fucking up less things around the world
yekanchi | 12 hours ago
north Americans must stay silent about interfering in other states internal issues.
severino | 11 hours ago
Wow. I bet we'll soon be reading on HN that Biden stole the elections back in 2020.
Ancapistani | an hour ago
No one who believed it definitely was not has changed their mind, they've just at most gotten quiet about it.
No one who believed it could have been but that the claim has not been proven have changed our minds, we've just decided that there's no way to know what actually happened.
RobotToaster | 11 hours ago
Vietnam is usually recognised as a low point in Chinese history.
There's been an ongoing border dispute between China and India for over 100 years, thanks to some questionable line drawing by the British.
Also, referring to the parent comment, of these only the border skirmish with India was within my lifetime.
rithdmc | 11 hours ago
novoreorx | 11 hours ago
ksk23 | 11 hours ago
jmyeet | 6 hours ago
Tibet specifically is funny because one of the things China did was end slavery in Tibet [2]. A large percentage of the population were "serfs" but that was a generous translation because these "serfs" could be traded. You know, like property. Like slaves.
And Vietnam? They had a small border dispute. What did the US do to Vietnam? And Cambodia? And Laos?
But the funniest claim of all here is election interference. The foreign interference in US elections conversation begins and ends with Israel with an honorable mention to Russia for running some Facebook ads. Just last night we had a debate in the Democratic primary for Senate in Michigan, a race that AIPAC proxies (eg UDP) have spent upwards of $60 million. On a primary.
[1]: https://en.wikipedia.org/wiki/United_States_involvement_in_r...
[2]: https://en.wikipedia.org/wiki/Tibet_serfdom_controversy
dgroshev | 5 hours ago
There's a qualitative difference between interference and outright annexation, with borders redrawn and cultures violently assimilated.
rikima_ | 6 hours ago
This event was followed by the longest honeymoon period in Sino-US relations, which eventually lead to China's rise.
altmanaltman | 15 hours ago
spaceman_2020 | 14 hours ago
altmanaltman | 14 hours ago
t0bia_s | 14 hours ago
spaceman_2020 | 14 hours ago
Unless you're trying to imply that China engineered the whole thing - extraordinary claim that will require extraordinary evidence
dudefeliciano | 13 hours ago
watwut | 9 hours ago
dudefeliciano | 8 hours ago
[1]https://en.wikipedia.org/wiki/COVID-19_misinformation_by_Chi...
oneshtein | 5 hours ago
t0bia_s | 10 hours ago
flir | 9 hours ago
We've got a fair idea where SARS-CoV-2 originated (horseshoe bats). It's the early path it took that's muddy.
sirsinsalot | 9 hours ago
worldthruword | 14 hours ago
oneshtein | 5 hours ago
China was the first victim of the Russian virus.
b3lvedere | 14 hours ago
blackhaz | 14 hours ago
xquce | 14 hours ago
preisschild | 13 hours ago
jeltz | 8 hours ago
jari_mustonen | 11 hours ago
KronisLV | 14 hours ago
Idk to me it feels like human compassion and ethics dictates that you at least have to care a bit even about the things in other countries. Otherwise all sorts of horrible domestic policies would be justifiable.
miroljub | 12 hours ago
preisschild | 13 hours ago
They actively back russia and deliver them weapons and support them with their attempted Genocide in Ukraine. They also actively threaten Taiwan.
Not to mention them claiming almost the entire South China Sea even though this being against international law and threatening smaller nations into submitting to them
PatronBernard | 13 hours ago
Don't be naive or I might even think you're working for the Chinese :-)
cechmaster | 13 hours ago
Cookingboy | 13 hours ago
https://law.stanford.edu/press/state-department-lawyers-conc...
That whole thing was a very successful propaganda campaign, making people believing China has been mass murdering Uyghurs.
oceansweep | 6 hours ago
The U.S. State Department’s Office of the Legal Advisor concluded earlier this year that China’s mass imprisonment and forced labor of ethnic Uighurs in Xinjiang amounts to crimes against humanity—but there was insufficient evidence to prove genocide, placing the United States’ top diplomatic lawyers at odds with both the Trump and Biden administrations, according to three former and current U.S. officials.
theplumber | 12 hours ago
For example Lithuania has been punished for letting Taiwan opening an embassy (in Lithuania). So China cares what other countries do in their country. Without EU support Lithuania could have faced very harsh consequences. That's just an example. Another example is the "secret police" stations undeclared overseas police stations operated by China in various countries (i.e. U.S for example). China gave you cheap material goods because that was its business but you have to keep in mind it's an authoritarian, communist regime. It carries an extra risk on top of the potential economic coercion if you give it too power. At least you know that U.S is only after the money and does not want to turn your country in a communist "utopia".
That being said this does not mean we should ban Chinese AI models because China didn't cross any red lines(yet) compared with Russia for example.
dooom | 11 hours ago
Not mutually exclusive, just take a look at Venezuela.
theplumber | 11 hours ago
sailfast | 6 hours ago
Sohcahtoa82 | 4 hours ago
China is becoming an intellectual powerhouse in addition to a manufacturing powerhouse and runs huge spyware and propaganda campaigns.
Read up about T95, MBOX, TVBOX, or other Android TV Boxes advertised as generic TV streaming devices advertised as "unlocked" or capable of accessing free content. They're loaded with spyware/malware, typically BADBOX. Some of them will record audio and record your network traffic and send it to China.
I'm also convinced that the anti-education thread winding its way through US culture is being amplified by China. They likely didn't start it, as it's been growing for decades, but China will happily keep it going.
john-h-k | 4 hours ago
Paradigma11 | an hour ago
rustyhancock | 13 hours ago
HF could not be helped by US frontier models because of the "safety" features they have.
HF had to use an open model from china.
Anthropic wants to add those "safety" features to open models - especially from china.
End result would be HF hack would have continued atleast until the Monday that OpenAI engineers finally walked back into work.
rob74 | 12 hours ago
littlecorner | 3 hours ago
tessellated | 27 minutes ago
b112 | 12 hours ago
And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.
So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.
I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
It's a genie that's not going back in the bottle, the bottle is smashed.
The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.
Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.
wongarsu | 12 hours ago
Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)
notahacker | 7 hours ago
Open weight models get scrutinised in a different way, also linked to perceptions of their developers' bias, like the tests to see whether they refuse to answer questions on certain historical events at Tiananmen Square
HappMacDonald | 5 hours ago
John Oliver disagrees with this
ppap3 | 8 hours ago
ethbr1 | 8 hours ago
You're vastly underestimating the scaling problem here.
Things that would need to be true for your statement to be valid:
hgoel | 8 hours ago
People call grok that because deviants were abusing grok's ability to edit images and post them publicly on X to strip people - including children - of their clothes, from their public photos. Then when there was backlash, Elon laughed it off. It took half the world opening investigations against X for violations of existing regulations for action to be taken.
The leniency that internet companies get in terms of dealing with illegal content comes with the expectation that they're making reasonable efforts to control the distribution of said content. X, and Grok, were actively supporting the production and distribution of the content in public.
It is very different from someone creating such images in a private account, and definitely very different from someone using a local model to do it.
b112 | 22 minutes ago
My entire post was how it is unreasonable to have a dual standard, and my point was it's really irrelevant if it's a model you download and use locally, or if it's a model hosted remotely, or hosted and created remotely. You're not really providing any sensible reason where the line is, except "public company", which is, again, the entire point I'm making.
The only realistic, non-double standard is that the creator of the model should be 100% responsible. What on earth does it have to do with who's hosting it?
And by this metric, aren't all the uncensored models on huggingface, child pornographers? And if so, why not? Provide tangible, real, sensible reasons please, and after all, isn't hugging face a company?
You know, people are all over the place on this. I see people complaining about guardrails, then in the next breath complaining there aren't enough. Complaining that open models are the thing, but then creating double standards.
So once again, what is your actual reason why it's different?
Barbing | 5 hours ago
Further reading: XAI Bets on Grok's Racy Side, The Information, Jun 24, 2026
&: https://arstechnica.com/tech-policy/2026/07/xai-cant-deny-gr...
ppap3 | 8 hours ago
jimmydoe | 8 hours ago
taude | 7 hours ago
EDIT: of course it probably helps to have an up front tdd test suite, but often isn't the case.
ppap3 | an hour ago
The summary is that an instance was running on certain benchmarks without any limits or supervision and the AI decided to cheat by exploiting a silly series of vunlns.
Loquebantur | 6 hours ago
Incentive and ability are what should be looked at. There, things get far more interesting: what is the current state of AI employed by the US intelligence agencies and what do they use it for?
Having the public convinced, their "superiors" would only do everything in their best interest, even without anybody knowing for sure, is Huxley's Brave New World in real life.
convolvatron | 5 hours ago
shouldn't any reasonable person when presented with a line of thought that has no substantiation at all conclude that they just can't reasonably be expected to support or reject the theory?
ppap3 | an hour ago
hoppp | 4 hours ago
imrozim | 5 hours ago
aa-jv | 13 hours ago
Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.
Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.
Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.
There are many in Europe who feel the same. The tide is very definitely turning.
ozgung | 12 hours ago
People in Many Countries Now View China More Positively Than the U.S.
https://www.pewresearch.org/global/2026/07/15/people-in-many...
pipes | 12 hours ago
aa-jv | 9 hours ago
jimmydoe | 8 hours ago
Trump deports, half of the USA people screamed; Xin builds reeducation camps, most Chinese people are cool.
sdevonoes | 7 hours ago
China had a worse reputation, but not anymore since Trump leads the US.
Im not saying that the above is objective, tho
pipes | 3 hours ago
thaway7388 | 11 hours ago
He portrays the biggest ever threat of AI as the Chinese Military using the models in their drones. He already sells his models to military customers, US and potentially allies, and they are actively used in the field.
So no objective argumentation here. Just Nationalist political rhetorics and FUD. He’s allowed to do that and he will have an audience. But he won’t be taken seriously outside the US. He appears blindsided.
dominotw | 7 hours ago
yes he is writing it as usa citizen running it a us company.
I dont see how "usa is also.." is relevant here.
vanagandr | 17 hours ago
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
CMay | 15 hours ago
If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips
Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.
Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.
In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.
majormajor | 15 hours ago
Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?
Regulatory capture 101.
Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).
CMay | 15 hours ago
Your cynicism will limit your understanding of other perspectives.
jibber1984v | 15 hours ago
He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?
> Found the person that believes some other random person can use a computer better than a John Carmack could.
Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
CMay | 15 hours ago
Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.
So, maybe not China?
> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
I never said that, but private smaller AI companies are all over the place. He never argued against that.
dmantis | 14 hours ago
Even on HN, I saw a lot, that when people discuss surveillance topics, they argue whether ie NSA can spy on Americans, not on everyone. Mostly, nobody even question the human rights of those inferior humans abroad.
That's a very long wiki page, I must say: https://en.wikipedia.org/wiki/United_States_war_crimes
So the question stays open.
CMay | 13 hours ago
Also, a country surveilling its own citizens has unique and different implications compared to surveilling other countries. Citizens need to be able to influence their own government, but mass control can nullify citizen power entirely which becomes its own problem.
Then, on a US website, you link to another US website which keeps track of various war crimes. Try finding an equivalent website for China in China, or for Iran in Iran. This is part of the asymmetric freedom issue on the internet that many people ignore. There are a lot of lies and propaganda spread within other countries that censor and deny some types of information from even existing, then they use that as a springboard to spread it around the world.
In the US, we can criticize ourselves which is important, since it helps us improve.
otherme123 | 15 hours ago
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
lwhi | 15 hours ago
stevelini | 13 hours ago
dgellow | 13 hours ago
picture | 15 hours ago
rf15 | 15 hours ago
dwattttt | 15 hours ago
gravypod | 8 hours ago
AMD would love to come by and plop one of these into your datacenters: https://www.amd.com/en/products/accelerators/instinct/mi400..... Also, in a few generation, there will be a massive glut of used hardware.
Unless frontier labs can surpass the current models significantly I don't know why I would pay them money instead of hosting K3.
abalashov | 4 hours ago
Also see: childcare, healthcare, many forms of public transport and social infrastructure. Some things markets simply cannot deliver well, and that's okay.
Daycare is the most easily reachable example because it's quite simple compared to the others. Daycare workers are simultaneously some of the lowest paid workers in the US, yet daycare costs are famously high and prohibitive, yet childcare centres are very far from money-printing machines. Margins in the daycare sector are most commonly < 1%.
necovek | 15 hours ago
Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).
With these long-term profit exercises, it's always a lot of hand-waving though.
shmichael | 13 hours ago
murderfs | 15 hours ago
The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.
joe_the_user | 14 hours ago
I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.
dgellow | 13 hours ago
catlifeonmars | 6 hours ago
21asdffdsa12 | 14 hours ago
dgellow | 13 hours ago
AIorNot | 15 hours ago
1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair
2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh
ccppurcell | 15 hours ago
CorrectHorseBat | 14 hours ago
I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?
The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.
xg15 | 14 hours ago
To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.
In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.
It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.
Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.
--
There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.
Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.
And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...
xg15 | 8 hours ago
E.g. they might say something like "We absolutely agree that international regulation of AI is needed and we're willing to co-sponser an initiative with the US. One of the most pressing matters we see is the spread of despicable misinformation through unregulated models regarding certain events in China's history, the treatment of certain minorities or the status of certain provinces..."
foo42 | 14 hours ago
puszczyk | 14 hours ago
But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.
Curious if this was more of an emotional take or if there’s actual evidence behind it.
topranks | 14 hours ago
I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment
gbalduzzi | 13 hours ago
Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.
qwery | 10 hours ago
radu_floricica | 13 hours ago
A much simpler summary:
- open models good.
- smart models _can_ be bad
- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.
- banning open models just in US is neither good or bad: is stupid.
npodbielski | 7 hours ago
hoppp | 4 hours ago
Alphafold and others solving the protein folding problem are revolutionary.
An LLM can max provide an instructive tutorial protocol for lab work, but designing novel proteins is not it's job.
I am interested to enter the bioinformatics space eventually and the regulation happens at the DNA printer level. Anyone can design yeast nowadays that excretes heroin, but the DNA needed for the genetic modification won't be printed by anyone. As long as DNA printing tech is extremely regulated it's all fine, if home printing becomes a thing then on the other hand maybe people won't even need AI to print deadly pathogens at all. A database lookup will do.
PunchyHamster | 13 hours ago
"Dangerous to our bottom line"
"We call it dangerous to hype up its abilities"
jorisw | 13 hours ago
And you know this how?
mcfedr | 13 hours ago
cable3 | 13 hours ago
bayindirh | 12 hours ago
Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".
Lastly: "What if they use their models in their military and local police services like we do? Communism!"
As always: https://pbs.twimg.com/media/B_AiI9_XIAA67_t.jpg?name=orig
Akronymus | 12 hours ago
bayindirh | 12 hours ago
So, ^H^H^H means "delete three characters, excluding '^H's". Like the person typing the comment changes their mind and deletes the characters (or the word) before writing else.
It's an stylized way of euphemism. i.e.: Actually I want to say this, but I substitute it for that.
grahamlee | 12 hours ago
TACIXAT | 12 hours ago
https://commons.wikimedia.org/wiki/File:USASCII_code_chart.s...
lynguist | 12 hours ago
prymitive | 12 hours ago
Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”
janpeuker | 11 hours ago
Nvidia signed the open-weight model letter and Europe doesn't have better models either, so chips don't seem like the issue either. I guess good old performance optimisation is just not _cool_ anymore. So they use the same argument as politicians arguing "cheap products" are why tariffs are needed; when instead it's mismanagement.
Another HN user wrote the other day "live by the sword, die by the sword".
dalemhurley | 11 hours ago
ozgung | 10 hours ago
I remember the discussions when 3D printers first appeared. People were thinking “bad actors” (“terrorists” then) would use them to print weapons.
People love to speculate and exaggerate. But they generally have poor judgements and rarely predict the future correctly.
fnord123 | 10 hours ago
Very good point. However, if one reads the transcript of the speech that Xi Jinping gave to the World AI Conference on 17 July, we see that he he is very much in favour of AI safety.
https://english.www.gov.cn/news/202607/17/content_WS6a5a1172...
> Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others.
Now, let's contrast another important part of safety here. Amodei puts the fact that this will need to be a global effort as a mere note that sure, China will need to help too:
> Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.
International collaboration is the focus of Jinping's speech. But one imagines "cooperation" Amodei has in mind if "do what I say" while Jinping has more collaboration in mind here. (Even if you think 'china bad' they deserve credit for collaboration for their open weight models).
fakedang | 7 hours ago
ppap3 | 8 hours ago
irenaeus | 8 hours ago
VikRubenfeld | 7 hours ago
> ...banning the use of these models by US businesses does nothing to address this risk, because bad actors are unlikely to be legitimate US businesses. It would protect US AI companies from competition, but that has never been my goal.
But on the other, he argues:
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Models that don't pass safety testing would be banned. Darius does not appear to be against banning models. He wants the government to have a regulatory body that has the ability to ban models. Then Anthropic can do regulatory capture of that agency and control what models are permitted to be released.
Also, during this mandatory safety testing, models would be blocked from use, and by the time the testing was done (probably years) the models would be obsolete.
ibic | 7 hours ago
gclawes | 5 hours ago
catigula | 5 hours ago
It doesn't track at all with any of his prior stated beliefs or past actions. It's an absurd claim. It's a baseless conspiracy theory, smuggling in traditional conspiracy mechanics for plausibility.
sosodev | 21 hours ago
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
Grimblewald | 21 hours ago
also anthropic
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
ch_sm | 21 hours ago
truncate | 21 hours ago
naveen99 | 21 hours ago
m3h | 21 hours ago
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
Cookingboy | 21 hours ago
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
kyllo | 21 hours ago
alex1138 | 21 hours ago
dan_gee | 21 hours ago
senordevnyc | 20 hours ago
alex1138 | 20 hours ago
But it's quite possible I'm being too anal
dan_gee | 20 hours ago
gr_norm | 20 hours ago
lenerdenator | 19 hours ago
The country that exists solely because of China? That North Korea?
I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.
jfrbfbreudh | 6 hours ago
lenerdenator | 5 hours ago
The nuclear part is really just gilding the lily. Of course, China and Russia have helped North Korea circumvent the sanctions that were supposed to punish them for their nuclear program, but it ultimately all goes back to the Chinese support of Kim Il-Sung during the Korean War.
jfrbfbreudh | 4 hours ago
lenerdenator | an hour ago
thinkingtoilet | 20 hours ago
monk_grilla | 20 hours ago
I might have missed something but wasn't the big story that Dario refused the Department of War's demand to use Anthropic's models for such purposes?
BeetleB | 19 hours ago
mrandish | 16 hours ago
tannertech | 15 hours ago
iamflimflam1 | 12 hours ago
Geof25 | 6 hours ago
culi | 15 hours ago
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.) Brad Cooper, head of the US Central Command, has boasted that the military is using AI in Iran to “sift through vast amounts of data in seconds” in order to “make smarter decisions faster than the enemy can react”.
Iran had the courtyard painted in bright pastel pink/blue colors with murals and playground markings to clearly identify it as an elementary school. The Pentagon claimed they had "outdated intelligence data"
woadwarrior01 | 14 hours ago
What if you can hire a human to push a single approve button? And what if that human's job is to front load approvals by pressing the approve button a few thousand times, every morning?
kpatucha | 11 hours ago
odyssey7 | 10 hours ago
woadwarrior01 | 9 hours ago
watwut | 8 hours ago
wosined | 11 hours ago
avianlyric | 11 hours ago
The actual aiming and firing of the weapon was performed by humans. In theory the target selection was also vetted by humans, but humans relying on exactly the same data that resulted in the “AI” systems misidentification of the target.
I put “AI” in scare quotes there, because these systems are really data processing pipelines, rather than LLM style AI systems.
SamDc73 | 16 hours ago
patcon | 16 hours ago
calgoo | 13 hours ago
patcon | 7 hours ago
It's absolutely not "fine". I'm saying maybe there's a complex region-beta paradox where we can't get to the other side of it unless some actors enter undesirable terrain (by their own measure), don't exit the stage, and play moves they'd prefer not to.
https://en.wikipedia.org/wiki/Region-beta_paradox
fg137 | 10 hours ago
This sentence should be clear enough?
sudosysgen | 19 hours ago
nmfisher | 19 hours ago
His redline was autonomous weapons, not the death of 100 innocent girls.
serial_dev | 15 hours ago
woadwarrior01 | 14 hours ago
culi | 15 hours ago
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
this_user | 10 hours ago
The point is that the same thing could have happened just as easily if you had selected targets based on the faulty data by any other means than an AI model.
cowpig | 8 hours ago
lobsterthief | 7 hours ago
foltik | 7 hours ago
Of course some overworked analyst is going to start neglecting the details and erring on the side of bomb it if the AI summary is dangerous sounding.
It’s gross negligence to be offloading this type of analysis to claude. Do you decide what’s factual based on the google AI summary? They’re deciding to end thousands of lives with about the same amount of rigor, including those schoolchildren. Have some humanity.
malvim | 4 hours ago
Humans select the wrong targets, computer select the wrong targets. It’s just that people really don’t care unless it’s hurting their bottom line.
culi | 4 hours ago
The Minab elementary school was painted in pastels and had prominent murals on the play area.
If anything, your comment just highlights how much this has to do with AI. When "outdated data" can lead to 168 dead schoolchildren, you can really see the consequences of relying on AI for decision making.
Invictus0 | 8 hours ago
mardifoufs | 5 hours ago
dgellow | 13 hours ago
yogthos | 4 hours ago
https://www.amnesty.org/en/latest/news/2026/06/usa-four-mont...
redwood | 19 hours ago
davkan | 19 hours ago
redwood | 8 hours ago
fg137 | 10 hours ago
redwood | 8 hours ago
srj | 17 hours ago
geraneum | 13 hours ago
DiogenesKynikos | 13 hours ago
joha4270 | 10 hours ago
And however much you can say about the current US Administration, I don't think they're prioritizing bombing civilians over military targets.
DiogenesKynikos | 10 hours ago
The US is bombing both military and civilian targets, by the way. Trump has been very open about this with his talk about "power-plant day." He considers attacking civilian targets to be a legitimate way of pressuring Iran to surrender.
folkrav | 8 hours ago
infamouscow | 5 hours ago
watwut | 8 hours ago
And administration repeatedly threatened to destroy civilian targets.
igleria | 11 hours ago
swat535 | 5 hours ago
Please don't say this. It wasn't a mistake.
The very FIRST strike packages of the war are very clearly vetted. If you are involved in military planning, you know this.
They doubled tapped it after seeing people flee inside the save the children.
The goal here was to teach IRGC a lesson and demoralize Iranians, which clearly backfired.
Furthermore, US has been contentiously bombing civilian buildings, bridges, hospitals throughout the war. They have also bombed water desalination plants and other key infrastructure.
You have the US president threatening to use nuclear weapons and end the Persian civilization.
Let's not pretend.
fc417fc802 | 2 hours ago
I'm not so sure. A school in the US being bombed would hardly be expected to demoralize us. Quite the opposite.
What if the goal was to goad them into taking action that could justify boots on the ground? But they didn't fall for it and here we are.
Footnote7341 | 15 hours ago
blitzar | 14 hours ago
extr | 21 hours ago
dirtyfrenchman | 21 hours ago
petcat | 21 hours ago
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
sanderjd | 21 hours ago
petcat | 21 hours ago
sanderjd | 21 hours ago
llm_nerd | 21 hours ago
You understand Moonshot AI could have had other parties run inference for them without releasing the weights, right? These two points are utterly unrelated, unless you think Fable and GPT5-6 are also "open weight" because other providers are providing inference?
Further, having access to the source material in no universe allows you to know what a model is "capable of". I'm not sure how this follows.
HDBaseT | 18 hours ago
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
Aperocky | 13 hours ago
The argument is an endless slope and as such essentially pointless.
overgard | 21 hours ago
mej10 | 21 hours ago
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
sanderjd | 21 hours ago
fooker | 21 hours ago
igor47 | 21 hours ago
fooker | 21 hours ago
itemize123 | 12 hours ago
Eggpants | 21 hours ago
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
Sidio | 21 hours ago
I'm less concerned that the attack was caused by a closed model, than I am that no closed model was willing to stop it.
The worst part is I'm confident Fable would have done a better job stopping the attack, but their 'guardrails' made it decide not to want to.
Unless of course, you pay up: "Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards"
-Noah Lebovic, former Anthropic staff
https://x.com/NoahLebovic/status/2081277517709922501
Keyframe | 21 hours ago
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
dofm | 21 hours ago
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, it's a deliberate and telling choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" or "Chinese state".
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
jrflowers | 21 hours ago
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
lmao the sort of lies people come up with when their only business model is “the government picks me as the winner” are so funny
soundworlds | 21 hours ago
This is clearly false to the rest of the world.
Cookingboy | 20 hours ago
According to him the safety and morality rule of the whole world should be written by America alone.
Which is why in the same interview he said he supports the U.S. foreign policy while calling China "an aggressive and war mongering regime".
>This is clearly false to the rest of the world.
It's clearly false to more and more Americans too. But since the oligarch class benefits first and foremost from U.S. government policies the propaganda will continue to go on.
abacadaba | 19 hours ago
verdverm | 19 hours ago
monk_grilla | 19 hours ago
Cookingboy | 13 hours ago
throw1234567891 | 4 hours ago
teacpde | 19 hours ago
gorgoiler | 7 hours ago
China ends up being “bad” not based on objectively looking at evidence and data, but because it is simply not the US.
Business nationalism, basically.
Schnitz | 21 hours ago
ashu1461 | 21 hours ago
xscott | 21 hours ago
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
Anti-competitive bullshit. I hope they fail.
fooker | 21 hours ago
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
hajile | 21 hours ago
throw1234567891 | 4 hours ago
bhewes | 21 hours ago
nirav72 | 21 hours ago
sbochins | 21 hours ago
willmadden | 21 hours ago
I love how they invoke fear of "terrorism" to justify their oppressive position.
Anthropic would love the US to do everything in this list under the guise of "safety testing":
https://news.ycombinator.com/item?id=48997548#49007134
prima-facie | 21 hours ago
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
kyllo | 21 hours ago
margorczynski | 21 hours ago
arthurlockman | 21 hours ago
teaearlgraycold | 21 hours ago
transcriptase | 21 hours ago
teaearlgraycold | 18 hours ago
If a Chinese company pays for my Claude tokens I’m both getting directly compensated and forcing Anthropic to lower their prices.
kome | 11 hours ago
jjcm | 21 hours ago
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
baron3dl | 21 hours ago
IP for me, not for thee.
epolanski | 21 hours ago
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
computerdork | 21 hours ago
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
llelouch | 21 hours ago
computerdork | 20 hours ago
ddxv | 16 hours ago
Either way, for me at least, it's an example of the more I read what they want the world to look like, the less I like them as a company and I have no desire to see them succeed.
Thus I actively go out of my way to watch / comment / follow what they are doing. I guess a lot of other people have similar frustrations and so there are a lot of people attacking them online.
bryan_w | 15 hours ago
Alwayshasbeeb | 9 hours ago
Plus, what a shocker it is that a "hacker news" website would have many people with an affinity towards open source technology they can fiddle with and have autonomy over, and a distaste for tech monopolies trying to control and limit access to these tools.
Catloafdev | 21 hours ago
I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?
The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.
philipkglass | 20 hours ago
https://simonwillison.net/2026/Jun/10/if-claude-fable-stops-...
In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).
...
Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
(And although the "silent" downgrade part was quickly dropped, Fable still won't help you here.)
Anthropic won't teach you how to build bioweapons, or enable you to make your own software infrastructure so that you can train your own biology model. That's where lawmakers may arrive too if they buy Anthropic-style safety arguments. It's too dangerous to publish models that understand biology. It's too dangerous to publish training software. It's too dangerous to publish tools that allow you to build training software.
If you keep following the implications of their safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed. Worse than the Clipper Chip proposal of the 1990s era Crypto Wars. I have seen how "children must be protected online" has in practice turned into an attack on adult privacy affecting a wide swath of services and devices. I'm taking a maximalist position on openness now because I think that I can anticipate the next steps on the safety side, and I reject those steps.
seatac76 | 21 hours ago
The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.
wren6991 | 21 hours ago
What are the legal ramifications of this statement if it turns out Anthropic have lobbied for this? Does it just get swept under the rug? I can't say this is bullshit (that would be defamatory) but I am intensely skeptical.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
This is playing to readers' biases; isn't DeepSeek V4 Pro deployed on Huawei Ascend already? The old "Chinese can only copy" meme is getting pretty tired these days.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Applying such standards in the US means that US defenders are blocked from using the models, but attackers from other countries aren't. That is clearly counterproductive.
It's already been pointed out quite eloquently elsewhere that there is no such thing as a safety filter because the LLM and external filters can't actually identify malicious use. They can only identify the weaker implication "if the user is malicious, this is bad."
pyrophane | 21 hours ago
1. Using political pressure to target companies that are accused of doing it.
2. Attempting to impose criminal penalties on individuals associated with the action.
3. Having the US government attempt to use its capabilities to stop it.
None of these seem particularly likely to succeed.
mcv | 21 hours ago
But what if it's the US that becomes authoritarian and uses AI models to perpetrate incredibly deep repression of their own people?
andix | 21 hours ago
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
manoDev | 21 hours ago
0xDEAFBEAD | 15 hours ago
"Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.
..
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control."
https://xcancel.com/S_OhEigeartaigh/status/20780236576206768...
andix | 4 hours ago
thaway7388 | 4 hours ago
He doesn't say "our own control". AI security is a very serious matter. The way it's being discussed right now gives more harm than benefit to understand the problem.
This is because it's discussed in Cold War/WW3 context. This is exactly Dario's mentality in this announcement. And it's understandable since USA is a country actively fighting in constant wars. They even have a Department of War.
This is unlike the rest of the world which have more like a Peace-time mentality. In peace times, trade, collaboration and good relationships are more important than competition or arms race. China seems like playing this game better than US at the moment.
Dario also speculates to take the potential benefits or dangers of AI to the extremes. This is also not very healthy thinking and can be dangerous when combined with war mentality.
tonyrice | 21 hours ago
tonyrice | 21 hours ago
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
zormino | 20 hours ago
sobrey | 21 hours ago
lukewarm707 | 21 hours ago
you should be worried about the USA having these models.
sfink | 18 hours ago
lukewarm707 | 10 hours ago
over time they became closed source, for profit, US military contractors, and the clients of expensive political lobbyists.
anthropic once said it was their identity to stop scaling before the current model capability; they rewrote the responsible scaling policy so that they could continue scaling. openai said in its founding document that it would be unconstrained by return on investment; it restructed into the for profit setup. deepmind signed documents when it was aquired by google stating that its ai would not be used for military purposes; they rewrote these commitments to sell gemini to the us military. it wasn't like this!
it did not have to be this way. even though everyone will IPO and become yet more accountable to shareholders, there is still time.
dorongrinstein | 21 hours ago
lukewarm707 | 20 hours ago
exabrial | 21 hours ago
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP)....
This is why open weights win. See Linux and how it's taken over the world. Your business model will need to change eventually. Instead you're advocating trying to exterminate competition via regulation and fear mongering.
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
Yawn... this is getting old.
> We should not sell powerful chips or chipmaking equipment to China
For as someone as smart as you guys, you sure lack common sense. China is just going to develop these technologies organically then and you lose 100% of control. It's already happened in reverse with things like Solar, rare earth minerals, etc. China flooded our market, destroyed our ability to produce things, now holds the keys. One thing they DIDNT do was stop trading to the US. They killed us with cheap goods.
> We should crack down on industrial-scale distillation operations.
Thats your problem, not my problem. Also, irony meter here hitting 11 about all those pirated books you stole...
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Oh, fuck, no. This is a crackdown on free speech and rights of people to do whatever they want. My right to free speech means I'm allowed to write whatever computer program I want, no matter what its size is or how "sufficiently advanced" it is. Individual rights always win.
I really hope people don't believe this garbage. For a company with a great product, this is absolute nonsense.
MiSeRyDeee | 21 hours ago
dnw | 21 hours ago
I don’t think this is open or closed; this is aligned and unaligned. I bet Grok would be as open as any open weight models to answering questions.
pylua | 21 hours ago
I can’t imagine this would be any different — banning open weight models would hurt us in the long run. The point is to beat the competition, not suppress it.
There should be no limits on open or custom models. Too often safety is a synonym for surveillance and control. It’s a natural consequence, intended or not.
I am curious… why can’t distillation be stopped?
As a side not Im not against protectionism, but it has to be across the board and the same in all industries with no excrptions. We’ve let all these industries die on the vine due to cheap cost in foreign countries. It could very well happen to ai.
nout | 21 hours ago
alerighi | 21 hours ago
Alwayshasbeeb | 21 hours ago
https://www.theguardian.com/world/2026/jun/20/mona-khalil-tu...
Cookingboy | 20 hours ago
It's kinda gross.
deadbabe | 20 hours ago
Release open weight models, no guard rails, no censors, straight to the public. Let everything else sort itself out. There is nothing more powerful than an idea whose time has come.
K0balt | 20 hours ago
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
ricardobeat | 20 hours ago
And accelerate their development of independent chip making technologies even more…
aprentic | 20 hours ago
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
vcryan | 20 hours ago
Nition | 20 hours ago
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
rramach | 20 hours ago
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
elliotec | 20 hours ago
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
stratos123 | 15 hours ago
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
elliotec | 15 hours ago
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
stratos123 | 13 hours ago
Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.
> And like, now look at the status quo of nuclear treaties and agreements and proliferation.
I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.
I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.
InkCanon | 16 hours ago
0xDEAFBEAD | 15 hours ago
throw1234567891 | 4 hours ago
We have a term for that: FUD.
novaleaf | 20 hours ago
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
[1]: https://news.ycombinator.com/item?id=49035303#49040674
AgentOrange1234 | 20 hours ago
The US doesn't have some magic wand that prevents "incredibly deep repression of their own people."
Insurrection, wars of choice, ICE, Palantir, Flock -- keep up man, we're the baddies.
yowo | 20 hours ago
ErneX | 20 hours ago
credit_guy | 20 hours ago
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
bigyabai | 20 hours ago
If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.
credit_guy | 20 hours ago
bigyabai | 20 hours ago
Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.
0xDEAFBEAD | 15 hours ago
bigyabai | 4 hours ago
ianm218 | 9 hours ago
bigyabai | 4 hours ago
ianm218 | 42 minutes ago
ACCount37 | 20 hours ago
Don't think "a smart guy". Think "project Manhattan and CIA put together, all in one server rack".
We're lucky to have "they can hack traditional systems" as an early warning shot. Clearly, it's wasted on many.
jackdeansmith | 19 hours ago
This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.
bigyabai | 4 hours ago
We've had LLMs for 5+ years, as well as Alphafold for 7+ years now. No novel bioweapon has been made with the technology that we're aware of. It's a farsical claim, there's no 21st century Aum Shinrikyo abusing the technology, after years of proliferation.
> they reach capabilities allowing for recursive self improvement
Again, you are predicating your entire argument on a hypothetical emergent behavior that we do not have any evidence for. I'm not worried about this whatsoever.
djsjajah | 20 hours ago
credit_guy | 19 hours ago
Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.
hdaz0017 | 20 hours ago
https://finance.yahoo.com/technology/ai/articles/anthropic-n...
drowntoge | 20 hours ago
I just don’t find it believable.
nevir | 20 hours ago
That is not an argument against open weight models. That's just a generic protectionist argument against any Other lab.
az226 | 20 hours ago
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
potsandpans | 18 hours ago
As I read more of his unhinged posts and some of the more ridiculous claims from anthropic, it's become clear to me that Dario thinks he can leverage American hegemony to regulate his company into a monopoly.
He has an ethics vaguely influenced by effective altruism, and he appears to think that his ethical framework entitles him to make decisions on the behalf of humanity, for all of humanity.
Literally a bond villain.
NoDodgeQuestion | 9 hours ago
idontbelonghere | 3 hours ago
storus | 20 hours ago
https://www.youtube.com/watch?v=_i91NSOyxHM
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
cmiles8 | 20 hours ago
jmward01 | 20 hours ago
Diverse ecosystems can absorb shocks. Diverse ecosystems are a sign of health of that ecosystem. When an invasive species comes into a healthy, diverse, ecosystem it doesn't mean that it isn't disrupted, but it does mean that it is far more likely to emerge with a lot of its diversity intact. In fact, it is likely to emerge even stronger because it can absorb that new shock and incorporate it, adding to its diversity. The balance may be changed, but the ecosystem survives or even thrives.
Nature also likes to show us that artificial barriers rarely last. You want to control a river? Good luck. It take constant maintenance to hold that flow in place and even then you are likely to get extremes that are made worse by your efforts because, eventually, somewhere in the system fails in a way you didn't anticipate. Then the water comes rushing in. Artificial barriers often have a way of building up tension over time, not reducing it, so that when a failure eventually happens it can be catastrophic. In other words, you had better really understand the system you are trying to control or else you can make things actively worse.
Relating this to the world now means, I think, that our best chance to minimize long term shock and maximize the chance that the diversity we have around us survives is to try to grow as healthy of an ecosystem as we can as quickly as possible. Lots of models large and small in lots of different hands is, I think, a better solution than artificial barriers restricting the variety and diversity of models and users. I think this is closer to an ecosystem solution and has a shot at working. Basically, I highly doubt we understand this situation enough to do a good job of controlling it with artificial barriers. Instead I think we are more likely to build catastrophic imbalances than we are to create the healthy ecosystem we really need.
nxtfari | 20 hours ago
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
forafistfulof | 12 hours ago
Instead of bombing them, try justice. Yes, it is more complicated. And yes, you will have to give, and not take.
ianm218 | 9 hours ago
Training a decent open weight model takes atleast millions of dollars and they are all associated with real people and companies, almost exclusively in the US and China. So just make normal open weigjt go through bio testing before release and that gets you a lot of the way there.
There is more problems like finetuning but you need to start somewhere.
orbital-decay | 20 hours ago
Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".
htlemur_bobby | 20 hours ago
caxap | 20 hours ago
Just like how it was inevitable for SoTA LLMs to ignore copyright.
The actual challenge isn't how to prevent all these, but how stay on top.
And to stay on top it is inevitable to train unrestricted models. Anthropic is fighting windmills.
zkmon | 20 hours ago
do_anh_tu | 20 hours ago
hamasho | 20 hours ago
A single canonical official document can make it very simple. Even though each department cannot achieve the maximum gain from nuanced documents, keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.
If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society, it can be one of very few good outcomes from this dystopian AI craze.
spacedoutman | 20 hours ago
gopheryourshelf | 19 hours ago
jhack | 19 hours ago
Look in the mirror.
g42gregory | 19 hours ago
jacktang | 19 hours ago
kushalpandya | 19 hours ago
rvz | 19 hours ago
> Anthropic has never advocated for a ban on open-weights models.
"We don't want a total ban on ALL open-weights models" (Anthropic never released a single open weight model)
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
"We want tight regulations on highly powerful open or closed weight models that should go through mandatory safety testing that we outline which makes them safe to use."
This is still a form of a ban that he wants to define. But the rest of his concerns such as stopping distillation attacks and not selling chips to China all do NOT work.
Perenti | 19 hours ago
phantomathkg | 19 hours ago
Anthropic anti-open-model stance does not mean China is not a threat.
broodbucket | 19 hours ago
stratos123 | 15 hours ago
alightsoul | 6 hours ago
btbuildem | 19 hours ago
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
0xDEAFBEAD | 15 hours ago
joeisnotjane | 10 hours ago
Waterluvian | 19 hours ago
brador | 19 hours ago
Do you accept?
jp0001 | 19 hours ago
jp0001 | 19 hours ago
Glyptodon | 19 hours ago
nlarion | 19 hours ago
neves | 19 hours ago
I think he lives in a different word than me
girfan | 19 hours ago
Good luck preventing distillation and limiting the supply of accelerators to China when Jensen himself is a strong opponent of any such barriers [1].
[1] https://youtu.be/Hrbq66XqtCo?si=VJh2QjOzkqT3iMjl&t=3456
asawfofor | 18 hours ago
paxys | 18 hours ago
txrx0000 | 18 hours ago
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
jamilton | 18 hours ago
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.
txrx0000 | 16 hours ago
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
stratos123 | 14 hours ago
A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.
(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)
¹ Really "all generators of text that went into the training dataset".
² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.
txrx0000 | 13 hours ago
This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.
And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.
FpUser | 18 hours ago
huslage | 18 hours ago
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
sreekanth850 | 18 hours ago
If decades of fighting have failed to stop piracy, I’m sure nobody can stop China from sourcing high end chips. Unlike piracy, I’m happy that Chinese labs are releasing open-source models, so people in developing countries are no longer at the mercy of this capitalist bullshit.
boinkboink78912 | 18 hours ago
nativeit | 18 hours ago
mingqiz | 18 hours ago
ltbarcly3 | 18 hours ago
jsomedon | 18 hours ago
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
dools | 18 hours ago
ProofHouse | 18 hours ago
nullbio | 18 hours ago
Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."
More dishonest framing from the company that constantly lies to everyone. No surprises here.
diebillionaires | 18 hours ago
Also I find it incredibly difficult to hear any company in the US worry about repression of people when financial repression is happening here. I’ve been to China and seen what the services to the public are like. Meanwhile the US funds and employs AI weapons in an ongoing genocide.
mnming | 18 hours ago
cdnsteve | 18 hours ago
wangii | 18 hours ago
breatheoften | 18 hours ago
sleepybrett | 18 hours ago
winterbourne | 17 hours ago
Simboo | 17 hours ago
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
killjoywashere | 17 hours ago
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
parsimo2010 | 17 hours ago
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
fractorial | 17 hours ago
Google catching this stray made me laugh, ha.
parsimo2010 | 17 hours ago
kubb | 14 hours ago
zer0zzz | 17 hours ago
kazinator | 17 hours ago
Our position on Linux
Fud, fuddly, fuddy-duddy fud ...
yanhangyhy | 17 hours ago
Is China an authoritarian government? I’d say yes. Is an authoritarian government worse than a democratic one? Personally, I think so. But these discussions always happen within a perfect, idealized model—reality looks a lot different. Take Japan and South Korea, for instance. Are they democracies? Sure. But Japan is heavily driven by factional and dynastic politics, meaning most lawmakers come from established political families, and regular citizens don't really stand a chance of breaking into that circle. In a recent asset disclosure in Japan, many politicians literally wrote down "$0," and I honestly can't think of a government so broken that even the voters don't see a massive issue with that. Meanwhile, South Korea has its chaebol politics, where massive conglomerates wield incredible influence over the government, to the point where most South Korean presidents end up in prison or meet an untimely end.
Coming back to the US vs. China dynamic: in reality, China’s authoritarian system is actually way more logical and resilient than it sounds from the outside. While openly criticizing Communist Party policies is pretty much banned domestically, the public can still shape the decisions of the government and the Party through public opinion. It ties back to that famous quote: "In China, you can’t change the party, but you can change the policy; in the US, you can change the party, but you can’t change the policy." Even if some policy changes in China happen slowly, compared to the US, it actually works out a lot better most of the time.
As for the article mentioning the use of AI for cyber and biological attacks—they know full well that the US has already deployed AI in actual warfare, which is exactly why they conveniently dodged that topic. It’s incredibly hypocritical. Ironically, the one that hasn't actually engaged in that kind of behavior is the "authoritarian" Chinese government. Sure, you could speculate that China might use AI weapons against Taiwan down the road—especially considering Taiwan likes to build fortifications near schools to create leverage against the PLA—but launching a moral crusade over something that hasn't even happened, coming from American companies whose own country has already done these exact things, is just plain shameless.
Another common misconception is trying to separate the Chinese government from the Chinese people, with arguments like: "The Chinese people are oppressed, so you have to look at them separately; the government is evil, but the people aren't." You only need to look at the US to see the flaw in that logic. The American public voted Trump into office, letting him trigger trade wars and attack other nations. Does that make the American people evil? If the answer is no, then it implies the US isn't truly a democracy, since only an authoritarian state could completely ignore its people's wishes and do whatever it wants. If the answer is yes, then the US really is a democracy—it's just made up of malicious people, much like the company that put out this article.
naiveter | 17 hours ago
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
broodbucket | 17 hours ago
naiveter | 17 hours ago
I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.
k12sosse | 17 hours ago
jongjong | 17 hours ago
When people say "What will be left for us to work on once AI takes over", I say "Ourselves" - We have to stop looking at humans as commodities and strategic pieces and start paying attention to people as individuals, based on the content of their character. We need a society which is attuned to this, which has enough resources and time to pay attention to this. Now we are blind to people's character because it is masked by money, power and status; all of which currently have higher priority. This order of priority is determined by scarcity, which is largely artificial.
We have a dishonest system which tries to control people's behaviors through scarcity-based coercion instead of straight forward laws or simple incentives and clear explanations.
hsaliak | 17 hours ago
cjarrett | 17 hours ago
grwthckrmstr | 17 hours ago
Edit: To add some more context. What I mean is neither look like the good guys or the bad guys, but one of them is spending an awful amount of energy trying to paint the other as the bad guy and themselves as a good guy, which I hope a lot of people aren't buying anymore. Because at the end of the day, I think the honest truth is that everybody is just trying to serve their own interests.
mmmgge3 | 16 hours ago
throwaway27448 | 16 hours ago
stevefan1999 | 16 hours ago
Shut the fuck up
nadermx | 16 hours ago
tachyons | 16 hours ago
SanjayMehta | 16 hours ago
As an example take North Korea. Sanctions didn't stop them from developing nukes and delivery systems.
mrcwinn | 16 hours ago
But the biggest issue is this. Many hate Dario because he’s smug, he caps usage, and because OpenAI effectively ran a counter-positioning campaign to paint Anthropic as undemocratic.
Who is he really and what are his motives? None of you know, really.
crvdgc | 16 hours ago
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
ianm218 | 9 hours ago
raincole | 16 hours ago
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, just like voting eligibility tests aren't not to prevent people you dislike from voting! /s
gscott | 16 hours ago
system2 | 16 hours ago
stkdump | 16 hours ago
habosa | 16 hours ago
palmotea | 16 hours ago
> ...
4. At the end of all this Dario Amodei must become a trillionare, for the good of all humanity.
SubiculumCode | 16 hours ago
regexorcist | 11 hours ago
SubiculumCode | 8 hours ago
Laurel1234 | 10 hours ago
If you think he gives half a shit about safety you've had a dozen lobotomies too many and should reconsider every decision you've ever made.
ianm218 | 9 hours ago
Laurel1234 | 9 hours ago
ianm218 | 9 hours ago
I’m in the camp that words have meanings you can’t just call everyone a fascist who you dislike.
Laurel1234 | 8 hours ago
Would you not agree Palantir is at the forefront of this transition towards fascism?
What would you call someone who, as CEO, chooses to partner with said regime and Palantir? And not just in unrelated, innocuous shit, but in actively helping blow up innocents, including triple tapping a fucking school and killing hundreds of schoolgirls?
> So do you consider everyone who works at defense primes like Raytheon fascists?
They are collaborating with fascism, yes. Of course most of them worked in the industry before the current regime came in and made its fascist turn so they didn't set out to do so. Though the industrial military complex still was the main driver of US imperialism and so anyone who chooses to work in it is absolutely complicit in the mass murder of innocents the US has carried out. No idea about Boeing, can't comment on that.
Not that the agency of some working stiff is comparable in any way, shape, or form to that of the CEO of a private company valued at over a trillion motherfucking dollars.
ianm218 | 7 hours ago
So no the current regime is not fascist in nature. There’s pieces that rhyme for sure but still words have meaning - the opposition isn’t meaningfully oppressed, we don’t have a dictator. Trump is awful but words still have meaning.
> Would you not agree Palantir is at the forefront of this transition towards fascism?
No more towards a surveillance state.
> Of course most of them worked in the industry before the current regime came in and made its fascist turn so they didn't set out to do so
So was Dario in his field?
So yes I don’t think there is a serious argument that Dario is a fascist.
Laurel1234 | 7 hours ago
All of those fit to a fucking T, except being a dictator which he hasn't yet managed. But he's been thoroughly undermining the constitution and limits on his power, has thanked Elon publicly for fixing the election, said he wants to run despite it being unconstitutional, and even that elections won't be needed anymore. He's fired was it Pam Bondi I think because she wouldn't persecute his political opponents as aggressively or vindictively as he wanted.
> No more towards a surveillance state.
My brother in Christ why the fuck would a democratic republic need a surveillance state. You're literally making my case.
> So was Dario in his field?
He was in AI sure. He CHOSE to partner with Palantir and lick the boots of the fascist US regime and its fucking department of war.
No skin off my ass though, it's your country. Enjoy the last dying gasps of your vaunted FrEEdUM. As we say in my country, nobody blinder than he who doesn't want to see.
ianm218 | 6 hours ago
Anyway, you really didn't make a point where it's fair to call Dario a fascist which was the point of this thread.
kfse | 16 hours ago
0xDEAFBEAD | 16 hours ago
kfse | 15 hours ago
ninjahawk1 | 16 hours ago
We all know that this isn’t some higher ground stance, they’re anti-competitive since they’re currently #1. I’ve been seeing this for a while. They’re also the only large AI lab to NOT have ANY open-weight models in the public. Meta, xAI, OpenAI, they all have at least some of their models open sourced from a year or so ago, Anthropic hasn’t even made Haiku 1.0 open-weight.
On top of that, I personally think that they’re upping the price on their models higher than they’re letting on, I think if someone did the actual math on their exact amount of compute and then compared it to their consumer and API prices, it would be astounding.
d5lt5 | 16 hours ago
abotsis | 16 hours ago
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
culi | 14 hours ago
abotsis | 5 hours ago
ianm218 | 9 hours ago
abotsis | 5 hours ago
To continue the analogy, he’s basically asking for some magic guardrails in nmap that doesn’t allow it to be used for scanning a network you don’t own. Of course even if you could add that- you could modify the source to bypass it.
It’s basically “open weight models are a public good if you can guarantee safety” .. which you can’t. So the only viable alternative is a hosted nmap that requires you to prove you own a network before scanning it. Which is impossible. So it’s all the right words and sounds nice, but making an impossible ask.
ianm218 | 3 hours ago
When the risks are as high as discussed here it doesn't really make sense to give up because you haven't found the silver bullet.
Stitch4223 | 15 hours ago
https://en.wikipedia.org/wiki/Regulatory_capture
On processing power, copyright, and capability.
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
Edit: spelling
ianm218 | 10 hours ago
Doesn’t mean that you have to think only Anthropic should be able to make bioweapons or whatever, but it just feels like this ignores the risk
Stitch4223 | 4 hours ago
Using a technology does not make an illegal activity illegal. It already was illegal to begin with.
And there are already guardrails in the form of ethos, law, justice departments and so on. This reality is flagrantly dismissed in their position on open-weights.
Just like cyber crime is just things like extortion etc which are already illegal: following similar reasoning all computer activity should be regulated by their vendors.
Hence the knives factory analogy, which is even more basic to point out this crooked way of reasoning.
tesnorindian | 15 hours ago
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
jacktang | 15 hours ago
pknerd | 15 hours ago
Umm, isn't the US acting like another authoritarian regime by advocating a certain kind of obstacle because only a US regime is allowed[1] to do what it is fearing[1] about:
[1] https://en.wikipedia.org/wiki/United_States_Army_Biological_...
[2] AI models may be misused to carry out cyberattacks or biological attacks
Anyway, Dario. You are more concerned about your business than anything else.
viccis | 15 hours ago
For convenient definitions of "dangerous"
3uler | 15 hours ago
You mean the industrial scale distillation attack you perform on the entire corpus of human knowledge… idk, call me cynical but you reap what you sow.
foo12bar | 15 hours ago
Crafty lawyer speak, saying nothing of substance.
foxylad | 15 hours ago
Maybe the techbros are playing irony roulette, seeing who can get away with the most outrageous hypocrisy.
throwaw12 | 15 hours ago
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
throwaw12 | 15 hours ago
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Why? And how do you design those tests?
For example, bombing girls school in Iran - is this allowed use according to you or not?
If not allowed use, then how do you guarantee that you don't have a separate agreement with DoW which makes it allowed use and only your model passes it?
CMay | 15 hours ago
There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.
Many of these targets were identified before Anthropic or OpenAI even existed.
throwaw12 | 15 hours ago
Also the point is, you (Dario) can't claim morality, when he is fine doing business with entities literally bombing and killing human beings in other countries.
You are either fine with it and continue working - which Dario is doing
Or you say, I will not work with you.
For Dario, main thing is money, everything else in his article is bs
CMay | 15 hours ago
AI has multiple uses. Military attacks can also save lives. As they say, the best defense is a good offense.
The moral element of it largely falls on the people who made the mistake. That said, it is also widely known that civilian casualties are a part of war. The advent of precision strikes has greatly reduced civilian casualties.
Meanwhile, Iran has intentionally promoted the direct targeting of civilians and killed 10s of thousands of their own, while funding proxies that killed many civilians elsewhere.
throwaw12 | 14 hours ago
In your argument have you considered: selling a knife knowingly to gang members just before they are going to another area to kill other gang members
Anthropic did same when it agreed to sell it's tool to DoW. There is no mistake, no misunderstanding of intentions.
He was super clear that he doesn't give a dime to any lives outside of USA, inside USA he was concerned about automatic weapon usage and surveillance of US citizens, because he himself could be accidentally killed, he doesn't care about others.
CMay | 14 hours ago
A similar argument was made for why Microsoft shouldn't sell software or services to border patrol or ICE. The counter-argument was that if it helps them be more precise and reduce errors in their managing of all the people then it could actually help not just the organization, but also the people.
Again, whether it's human or AI, mistakes will be made and civilian casualties will likely remain a part of war. AI can ideally keep civilian casualties low.
8-prime | 13 hours ago
throwaw12 | 12 hours ago
Are you sure?
https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_G...
They even have the cool one: "Where's Daddy?" - which tracks the suspect and then notifies the officials when suspect is at home, so IDF can bomb the whole building.
In this case AI is specifically designed to increase the civilian casualties, why not shoot the suspect independently, why include their neighbours, imagine your neighbour in 10 floor apartment building is a terrorist and your building gets bombed because of single person
CMay | 8 hours ago
Israel has had Iran and its proxies directly and intentionally target Israeli civilians and other civilians in the region, so their perspective on attacks may or may not be more flexible compared to the U.S. approach. I don't know. They have gotten some criticism for their tactics, but nobody argues that they are in a dangerous neighborhood.
There have been more occasions where Israel explicitly went out of its way to avoid civilian casualties than not, so I don't think it's the norm. Based on the extremes I've seen them willing to go to, though, I'm not surprised if it has happened. That said, I'm not saying I know whether that wikipedia page is accurate or not since it relies on an anonymous source and the IDF denied it.
As I said though, the US under multiple administrations had criticized and warned Israel to be more disciplined in its bombing, for what that's worth.
throwaw12 | 7 hours ago
I am not sure about this anymore, for some time I got an impression that Israel became the capital of USA.
> but you're linking primarily to a page about Israeli strikes.
Because you said AI will reduce civilian casualties, I gave you opposite fact. And relevant to the discussion, Dario complained about China being immoral, but didn't say anything against Israel, what message does it give to us? - Dario doesn't care about human rights, he is concerned about himself and how Chinese open AI can impact his company.
> the US under multiple administrations had criticized and warned Israel to be more disciplined
Who cares about criticism with no action, when they literally handing over them bombs:
here is your bomb which can blow up whole building, but please don't use it against buildings with civilians, oops, did you blow up civilians? Is your arsenal depleted? Okay take these bombs, but don't use against civilians, ooops again? Okay take all these bombs
CMay | 7 hours ago
You aren't doomed to only have some opinion impressed upon you by decades of whoever influences you. Break out of it.
Can casualties occur and those casualties still possibly be less than might have occurred otherwise? That's a simple question and the answer is yes. The problem is that every conflict is a bit different and there's almost no way of knowing within such a short period of time if the technology is producing more or less civilian casualties unless it is some unbelievable number that nobody can deny is caused by AI. We simply don't have that. You do not have that, because we don't even have that. It's not so bad that it's obviously not helping, and we won't know how much better it is for many years probably.
AI has been demonstrated to be able to save lives in the case of driving assists in electric cars and the numbers are undeniable. AI can help lend some of that to warfare as well. Doesn't that sound like a well rounded reasonable approach to the issue? We just don't know.
That said, it of course depends how it is used, so if we find that it is only being used to expand out to the maximum number of targets rather than optimizing for minimum civilian casualties then you would have to adjust that policy. As far as I can tell, any president or administration would have difficulty gaining support if they were lax on civilian casualties.
Do I think Israel is less moral than the U.S.? Yes, but I also think they are small and facing far more existential threats than almost any other country, so I also grant that may be where some of their flexibility comes from. I think sometimes they go to extremes that other countries would not, but they don't go to the same extremes that their enemies do which operate more on hate than reason.
bigyabai | 3 hours ago
> Doesn't that sound like a well rounded reasonable approach to the issue? We just don't know.
We could know. It's basic math to subtract the number of lives saved from the number of people killed by driving assists in electric cars, which would give you a "savior quotient" for the technology as a whole. The data is well-preserved for this application: https://www.tesladeaths.com/
Of course, the number wouldn't be very flattering, and would probably undermine your point. Then we'd be back around to "just a few more years until the tech matures" like we were back in the promissory days of FSD.
CMay | 2 hours ago
The website you linked isn't useful, since that's not a reliable way to collect crash data and even then I'm not sure the website supports your claim given that they had sold 9 million Teslas by the latest reported data on their table and in that entire time over 12 years they only recorded 772 deaths. Again, I wouldn't trust their data, but I'm not convinced the data they do have supports your claim.
If you normalize for car type, weight class and include all fatalities involved in a wreck (not only the occupants of the vehicle, which means including pedestrians, motorcyclists or the occupants of other vehicles) the relative safety of a lot of these cars is so much better than older cars now that you essentially get into the range of noise. In statistics so low they blend into noise, you are getting closer to having to use a crystal ball to understand driver psychology and behavior rather than the car itself.
Regardless of Tesla (which was actually the subject of political activist attacks and thus heavy propaganda against it), there are reliable statistics and studies that support these driver assist and even automated driving technologies as provably life saving:
https://www.mitre.org/sites/default/files/2025-01/PR-25-0114...
https://www.iihs.org/news/detail/safety-benefits-stack-up-fr...
https://electriccarsreport.com/2026/07/waymo-driverless-cars...
bigyabai | an hour ago
So, represent them and then subtract the number of people "provably" saved by assistive driving from the number of road fatalities blamed on assistive tech. You can normalize for all of these things without too much trouble, the existence of airbags and roll cages doesn't make your claim impossible to prove.
That would give you a single, easily readable quotient that describes the cost-benefit analysis in human lives perfectly. It's probably not a positive number, but it would be useful to know all the same.
globular-toast | 13 hours ago
throwaw12 | 12 hours ago
siruncledrew | 15 hours ago
> praises Trump's administration and Vance in his letter
sure there, buddy...
muneeer | 15 hours ago
self_awareness | 15 hours ago
We care about specifically Chinese models, because China distills our models. And that's real competition. So we want to ban those.
PS. Oh yeah, in case any open non-Chinese model will one day be good enough to compete with us, we will want to ban you as well, just FYI.
Kthxbye.
jari_mustonen | 15 hours ago
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
isomorphic_duck | 15 hours ago
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
throwaw12 | 15 hours ago
insumanth | 15 hours ago
WhyNotHugo | 15 hours ago
There's also a very strong implicit double-standard in the discourse, along the lines of "it's dangerous if non-US uses this in military fields, but it's fine if the US does so".
TrackerFF | 15 hours ago
Yeah, no thanks.
rarisma | 15 hours ago
moi2388 | 15 hours ago
Just like batteries, the automotive industry and in fact most machining.
We really ought to boycott Chinese products from stolen tech and patents from entering our markets
marhee | 15 hours ago
zdenham | 15 hours ago
kanak8278 | 15 hours ago
I think China building and releasing models to Opensource is a greater good because that is providing equal accessibility to everyone in the world.
By Dario's words authoritarian regime vis a vis China, I think OpenAI and Anthropic are also authoritarian in similar terms.
We are only seeing what they want us to show, they might be creating models which can do more harm.
So claiming that China can do or might do, vs Anthropic will not is just words.
Yeah I agree with the final paragraph that we should have testing agencies mandated world wide for each frontier model testing.
21asdffdsa12 | 14 hours ago
GreenJacketBoy | 14 hours ago
> We should crack down on industrial-scale distillation operations.
So Open-weight models are perfectly fine, but we don't want anyone to be able to make them.
xlii | 14 hours ago
xlbuttplug2 | 14 hours ago
They'd tease us with solutions to hard problems, with code that is orders of intelligence higher than any human or public model can grok.
That'd turn all the whining to begging real quick.
robot_jesus | 14 hours ago
One of two outcomes are true in this scenario: 1) this is unrealistic fear-mongering or 2) we're all fucked.
I just don't see how our solution to this can be export controls or bans. If the fear of a bioweapon engineered by an open weight AI is a legitimate threat, our only option is to develop effective countermeasures (and perhaps the same AI will assist with protecting). Open weights are not going away and pretending like some sort of "ban" will prevent bad actors from accessing them is a fairy tale.
t0bia_s | 14 hours ago
Done by who?
mumin00 | 14 hours ago
cloudie78 | 14 hours ago
Of course you didn’t advocate for the bans of open weight models. I will concede that.
It’s only the at you did literally everything else that you can to eliminate competition because otherwise there is no moat.
And half of the US economy is propped up by this AI bubble.
This isn’t about China doing evil things with models, most of which that you accuse of China - you and the US have already done and are doing.
The most obvious one being double-tapping a girls school.
Anthropic models directly integrated into Palantir’s Maven. The blood is on your hands.
Here’s something to chew on, maybe China is looking to integrate AI into its military and weaponise it as a direct response to the US doing that first.
So please, spare us the moralising.
As a closing thought: who the fuck even gave you the mandate to be the arbiter and judge of right and wrong, evil and good?
felooboolooomba | 14 hours ago
And now, here we are:
elisbce | 14 hours ago
hit8run | 14 hours ago
ricudis | 14 hours ago
saidnooneever | 14 hours ago
this guy is smokin spaceballs and should be institutionalized.
Gud | 14 hours ago
jorisw | 13 hours ago
____mr____ | 14 hours ago
trash_cat | 14 hours ago
1. Make anti-distillation clauses illegal to strenghten western opensource eco-system.
2. Make cyber-defensive models widely available (can detect but won't operationalize vulnerabilities). Otherwise make Fable awailable for everyone. Keeping the cybersecurity in assymetry by withholding cabailities just causes more instability and increases the incentives for powerfull close sourced models. This is a loss for everyone.
ece | 13 hours ago
danbruc | 14 hours ago
madhu_ghalame | 13 hours ago
Havoc | 13 hours ago
parham | 13 hours ago
one33seven | 13 hours ago
bluecalm | 13 hours ago
I just hope that if/when they succeed to lobby for protectionism regulation EU won't follow. I am not very hopeful though. We don't have democracy anymore and as last "vote" on chat control showed American big corps will get what they want out of our bureaucrats and they will be nothing the population can do about it.
neumann | 13 hours ago
What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.
vb-8448 | 13 hours ago
mcfedr | 13 hours ago
vb-8448 | 12 hours ago
Who is trying to define what is "too dangerous" or "not too dangerous"?
mcfedr | 11 hours ago
much as governments have tended to be slow in tech, they actually seem to be making some solid work on this front
tech companies like Anthropic should be collaborating with these programs
ianm218 | 9 hours ago
exolab | 13 hours ago
What Dario does not mention is that concentration in a few states or companies also poses a risk.
belabartok39 | 13 hours ago
Arshad-Talpur | 13 hours ago
maaaaattttt | 13 hours ago
Jackson98Tom | 13 hours ago
drdrek | 13 hours ago
I can already imagine it, a tiny drone carrying a 8x GPU rack thinking about life and deciding to go and build an idyllic society on a pacific island.
shifto | 13 hours ago
feblr | 13 hours ago
encyphilrightus | 13 hours ago
tacone | 13 hours ago
Well, if you ask me, that is dangerous.
paweladamczuk | 13 hours ago
itemize123 | 12 hours ago
ralferoo | 13 hours ago
Translation: "we won't be able to monetise it"
"We should not sell powerful chips or chipmaking equipment to China"
Translation: "we can buy them cheaper when there's less purchasing competition".
"Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable."
Translation: "we've been outsmarted and lost our advantage because their way is faster and cheaper"
I think it's a bit rich to complain about distillation, when they been plundering the internet for years for copyrighted works to train their models on without permission.
pascal-maker | 13 hours ago
jorisw | 13 hours ago
Mirror: https://xcancel.com/Biggest/status/2081891756769952075?s=46
pascal-maker | 11 hours ago
vrganj | 13 hours ago
The incredible hypocrisy to say this while the US itself is doing a sharp turn towards authoritarianism, with armed pro-government troops intimidating the population [0], where pro-government oligarchs openly talk of keeping the population under control with AI [1] all while actively enabling the rise of authoritarianism abroad [2] leaves a really bad taste in one's mouth.
What Dario really is opposed to is not authoritarianism. It's an authoritarianism where he's not part of the ruling class.
[0] https://www.theguardian.com/us-news/2026/jul/15/remove-ice-u...
[1] https://techcrunch.com/2024/09/16/oracle-ceo-larry-ellison-s...
[2] https://www.theguardian.com/us-news/2025/dec/05/civilisation...
alpineman | 13 hours ago
apexalpha | 13 hours ago
Even a completely closed US economy would not merit the current valuations Antrophic and OpenAI have.
tehjoker | 13 hours ago
haritha-j | 13 hours ago
My_Name | 11 hours ago
ranguna | 13 hours ago
I switched from openAI to anthropic because OAI were the bigger clowns in the industry and didn't want to support them, anthropic seemed like the better alternative that didn't cooperate with governamental shenanigans and actually focused on building a better product. But now they are both clowns and anthropic is reaching a ceiling on quality. I'll jump ship as soon as I find a good subsidized Chinese model provider. US companies only path forward is to become retarded instead of competitive.
EbNar | 13 hours ago
Lutger | 13 hours ago
jagadaga | 12 hours ago
wg0 | 12 hours ago
"We are in favour of 3D printers but there should be a body that tests and certifies that a 3D printer cannot print anything that can be used as weapon. Anything pointy or with a spring and recoil or... or..."
skohan | 12 hours ago
> the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This comes off a bit hypocritical, coming from one of the men responsible for the models most likely to be used by the US military, and for repression of US citizens.
> We should crack down on industrial-scale distillation operations ... We should have policy interventions to deter this behavior.
How exactly would you design policy interventions to stop distillation? If Anthropic wants to make their products available around the world, you would need some kind of global regulation to curtail it. And you would need to find some way to enforce it, which is far from trivial.
Honestly I don't see how securing models against distillation is up to anyone but Anthropic, if that's something they want to achieve. Calling for regulation is a bit like crying to mommy and daddy when things aren't going as you would like.
--
Overall, I can understand the argument that advanced AI models can present risks. But I don't see how regulation within the US can mitigate any of those risks. Any bad actor would be able to access the models outside the US, or covertly access the weights.
The only real way to prevent advanced models from being deployed would be to go around the world bombing every significantly powerful data center, and I don't think Dario would call for that any time soon.
The only thing this kind of regulation would achieve would be cutting US companies out of half of the innovation happening in the AI space, putting the US at a disadvantage relative to the rest of the world at everything except maybe frontier AI development.
yamal4321 | 12 hours ago
Its not like current US regime is using AI for: 1) Mass military operations across the globe 2) Mass surveilance of its citizens 3) Removing every possible safety guard from AI/climate regulation 4) Stealing data across the world to train its own closed-source models 5) Is openly antidemocratic, destabilizing EU and economy of whole world
I wonder why company which is actively cooperating with the current regime would push such message
Schlagbohrer | 12 hours ago
ramshorst | 8 hours ago
Schlagbohrer | 12 hours ago
pseudony | 12 hours ago
They have raised billions of dollars and are hoping to justify that by being a monopoly or oligopoly and their worst enemy- the pareto principle, is biting them in the ass. So now they’ll turn to policy to safeguard what they initially hoped could be achieved with money and technology.
It is that simple, they are the very definition of an unreliable source on this topic.
daitangio | 12 hours ago
The strongest competitors or a government law? I prefer the second.
Suppose Chinese model are super-risky: is it better to have them in the open, so everyone can review the problems or have a closed-model?
Security is better at open Linux is open source because development is better.
I understand Anthropic for sure, but trying to limit open weight model seems the opposite direction IF security+safety is the major concert.
The Genie is ALREADY outside the bottle, and there are already plenty of companies offering inference services
realusername | 12 hours ago
- "safe" AI doesn't exist, it's not a thing, AI providers can't distinguish between good and bad use of the AI, the filters they put in place are mostly PR.
- The US government IS an authoritarian government. Whoever wrote this doesn't know the difference between authoritarian and dictatorship
boonzeet | 12 hours ago
"...found that “other global powers’ robust progress in AI is challenging US economic competitiveness" - other countries having a slice of the pie is preventing the US having the whole pie!
"...secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks" - yes, those powerful open-source AI models like ChatGPT... oh wait.
ListeningPie | 12 hours ago
Point 1 is about restricting the sale of chips, not models. Point 2 concerns companies using closed models to train their own models through distillation.
The real issue is that open-weight models can run on much less powerful hardware, making the current AI business model, where companies train a powerful model and gatekeep access to it, less relevant. Once open-weight models become good enough, much of the future revenue for today's leading AI labs could disappear. But just as Microsoft still has a market so will the large AI houses have one, but the moat will not be providing AI responses.
qqt | 12 hours ago
northernsausage | 12 hours ago
regexorcist | 12 hours ago
egorfine | 12 hours ago
> Open-weights models that don’t have dangerous capabilities are a public good
There are no models that don't have dangerous capabilities. There is not a single human in the world that doesn't have a dangerous capability.
> authoritarian government [] build AI models that are more powerful than those built by the US
Yeah, they will do that no matter whether you support the existence of open weight models or don't. In fact, they have probably already done that. I would say "deal with it" but there is nothing you can do actually.
> authoritarian regimes have stolen and used AI
Monkey looking away meme.jpg
> powerful AI models may be misused
Small AI models may be misused just as well. I would say it's the small models that are the problem: cheaply deployable, easily fine-tunable, fast.
> once weights are released they cannot be withdrawn
Exactly. It has been already done. So?
> We should not sell powerful chips or chipmaking equipment to China
I agree. We should have competition and I'm rooting for China to design their own chips. They wouldn't if we let them buy our's.
> We should crack down on industrial-scale distillation operations
Well, at this time it's already irrelevant. We have GLM-2, DeepSeek, K3, which are already SOTA and can be used for distillation.
> All sufficiently capable models [] should go through mandatory safety testing
Problem is, it's unenforceable legally. Models are just math and all previous attempts to ban maths have failed and simply pushed US back. I will not be asking US government permission to calculate matrixes of my choice.
jiaosdjf | 11 hours ago
You can't have it both ways. Either it's fine to outsource to China, fine to use H1Bs to undercut US talent, or, foreign governments are strategically positioning and their espionage is a threat to America.
My_Name | 11 hours ago
And that is all that businesses will see. All the hand wringing over human rights etc will somehow be discovered to just not be important once they read that.
Fokamul | 11 hours ago
pietz | 11 hours ago
mgaunard | 11 hours ago
vasco | 11 hours ago
I don't think the qualifier "that don't have dangerous capabilities" is needed and that single phrase is a trojan horse / escape hatch that will allow them to say they are against any model they want. Smart but weasel behavior.
kome | 11 hours ago
hyperadvanced | 11 hours ago
DrScientist | 11 hours ago
Should the worlds public knowledge be enclosed by a few powerful private companies or should it be available to all?
The original vision for the web was a decentralized, open information sharing space designed to empower humanity. In my view the effort to privatise that via LLM distillation is the antithesis of that vision.
https://medium.com/@timberners_lee/marking-the-webs-35th-bir...
In terms of the risks mentioned in the letter - I agree with the risk of AI enabling a powerful surveillance state - however I don't see that as a solely Chinese problem.
The other risk mentioned - that knowledge can be dangerous - sure - but ultimately here, as there are so many low tech ways to cause mayhem, the ultimate protection is to have a society where people don't want to do it.
And perhaps more importantly I'd note that the primary tool to justify a powerful surveillance state is the fear of terrorism ( and others ).
simgt | 11 hours ago
tripzilch | 11 hours ago
"we're not saying open models are stealing, but we will quote the US vice president, saying this"
shevy-java | 11 hours ago
tripzilch | 11 hours ago
> At Anthropic we’re committed to cracking down on industrial-scale distillation
uhuh
I don't really believe their reasons
Aissen | 10 hours ago
m101 | 10 hours ago
fsuts | 10 hours ago
makingstuffs | 10 hours ago
1. “I’m fine with ‘em, as long as they’re useless”
2. “The only authoritarian regime which can be trusted with such power is the one which benefits my company”
witx | 10 hours ago
titaniumrain | 10 hours ago
amelius | 10 hours ago
And what if I called extracting value from copyrighted works "distillation" ?
It sounds hypocritical.
olalonde | 9 hours ago
bcjdjsndon | 9 hours ago
prmoustache | 8 hours ago
Why? Has Anthropic, or the country it is based in formally declared war against China?
_davide_ | 8 hours ago
ppap3 | 8 hours ago
They all support chasing people because of their origin without following the required legal steps.
It is just a matter of time until they come for you and your opinion
merelydev | 8 hours ago
Lockal | 8 hours ago
- Over the /last few days/ there has been a lot of discussion about open-source systems, especially those that are not made in US. (implying that these discussions appeared just now)
- Reports suggest that /some US officials/ are considering banning the use of these open-source systems by US companies. (implying somebody other than their corrupted representatives support this ban)
And then switching to a complete gaslighting regarding manufacturing capabilities of US vs China, about "open-source can not be trusted", about "GPT-2 is too dangerous to release"
ExoticPearTree | 8 hours ago
Secondly, he's like the Trump of tech: China, China, China.
Thirdly, he needs to cut it out with the guardrails and other safety BS he's peddling. If we're going to get Skynet, we will get it no matter what we do.
softwaredoug | 8 hours ago
Between China and US theres a defacto unequal distillation environment. China has no qualms about distilling off other labs outputs. US labs actively avoid it for legal reasons. Whatever the actual legalities, US labs have a relative hand tied behind their back. If they didn’t, it’d be easier for Grok, Gemini, and Meta to catch up.
If we care about US competitiveness, then one solution could be either OpenAI/Anthropic enter distillation agreements with other US labs. Or we decide to make distillation public domain / legal.
Until then the only models Anthropic/US Govt could realistically regulate would be in the US market. And that’s as much a losing game as tariffs.
irenaeus | 8 hours ago
rnewme | 8 hours ago
irenaeus | 6 hours ago
jimmydoe | 8 hours ago
mark_l_watson | 8 hours ago
I bet via regulatory capture that evaluating Chinese models will be a very slow process if Dario gets his way.
I enjoy using Opus, but I am in the process of ripping it out of my toolkit and leaving it on the ground behind me, as I walk away.
zeezer0 | 7 hours ago
sbiru93 | 7 hours ago
tcldr | 7 hours ago
Either everyone licenses, or nobody does. And if you can't enforce licensing bans for everyone, the de-facto loser is those who you'd probably want to support the most, start-ups and universities, while your adversaries gain the upper hand.
The strongest argument for restricting distillation is arms control – but distillation is the way to defeat GPU embargoes. So, distillation goes on regardless. Only pre-training is seriously attenuated.
If we're honest, the models are compressions of everything society has ever written. A few large corporations can't own that, no more than they can claim copyright for a zip file of the public library.
The genie is out of the bottle, now. So open it up – inputs and outputs, forward-looking – for everyone.
jackcviers3 | 7 hours ago
jatins | 7 hours ago
What is the other hand here? They could do slow/costly illegal think #1 instead of the fast/cheaper illegal thing #2 that they currently do?
monooso | 6 hours ago
pejrich | 5 hours ago
FinchNova12 | 3 hours ago
softwaredoug | 6 hours ago
So really its actively against our interests to prevent distillation.
dominotw | 7 hours ago
So anyone with chips can produce a model ?
nahuel0x | 7 hours ago
timedude | 7 hours ago
man_luke | 7 hours ago
impalallama | 7 hours ago
sagex | 7 hours ago
rootlocus | 7 hours ago
There's not much stopping US from becoming an authoritarian regime, and the US is already using AI against its own citizens.
aranaur | 7 hours ago
ajnin | 6 hours ago
Sure that's simplistic, but human behavior is not that complex, there are a few basic needs and wants that drive everything. In that specific case, those currently in a dominating position (Anthropic, the US) want to put hurdles (regulations) to make it more difficult for others to catch up, and everyone else want to join forces to overtake them.
mapledesk | 6 hours ago
mahmoudilyan | 5 hours ago
gverrilla | 5 hours ago
markmiro | 5 hours ago
Maybe the problem is it’s that Dario is saying it, so it’s easy to assume he has bias, which he does.
I’m not inherently opposed to open models, even at the frontier. But it could make sense for an adversary to give away something for free for some time to create dependence, or to tip the balance of power
QuantumNoodle | 5 hours ago
shivanshuag | 4 hours ago
There is a whole world outside the two countries which see neither as a good actor. As a reminder, USA was the first country which dropped an atomic bomb on civilians.
cdrnsf | 4 hours ago
ktosobcy | 4 hours ago
realo | 4 hours ago
I am Canadian and certainly do not consider the USA to be our benevolent anymore.
USA ... China ... same difference.
throw1234567891 | 4 hours ago
jmull | 4 hours ago
And, to me, this letter comes off as quite insincere. Stopping distillation can only be explained as an anti-competitive measure. Their own explanation is nonsensical -- they say is needs to be stopped to help prevent authoritarian governments from overtaking the US at the frontier of AI. But by its nature distillation lags behind the frontier. Not to mention the US is one of the authoritarian governments we need to be concerned with, and the next thing they advocate for is full, worldwide regulatory control of AI, which is rather heavily authoritarian.
These guys are making a $T gamble and need to screw over a lot of people very badly to make it pay off. You do not want to trust anything they say.
aagha | 4 hours ago
But we should not crack down on the illegal digestion of copyright text used to train our models.
firemelt | 3 hours ago
FloorEgg | 2 hours ago
I can understand how they would be against competitors distilling their models and for having access to competitor open weight models.
At first glance it seems like a pragmatic answer: "banning open weight model use by US companies doesn't help", but to what question? What if those open weight models are poisoned and primed to create backdoors in US companies? If the government has intelligence this is a credible risk, what should they do about it?
Overall it feels like the letter is conflating a bunch of motives, some of which may be opaque, and at face value doesn't seem logically consistent.
olejorgenb | an hour ago