Regulatory Capture by Anthropic Through METR

135 points by agnosticmantis 6 hours ago on hackernews | 65 comments
I think a TL;DR on this whole thing is that Anthropic is controlled by a cult.

I heard someone on a group I'm in humorously say it's the first IPO of an apocalyptic doomsday cult.

If they truly believed this stuff had a meaningful chance of being an X-risk to humans, they should stop building it and stop pursuing a giant IPO to finance building it. They don't even have the Manhattan project excuse of WWII and the Cold War as a forcing function. There's plenty of biologists, for instance, who have refused to work on things they legitimately think are too dangerous (gain of function research being one, biological weapons another). It's not like people with the expertise to develop and code AI systems can't find other fulfilling interesting work that pays well. But these guys are simultaneously crowing about P(doom) while building the doomsday machine with tremendous enthusiasm.

Which means they don't really believe it. I think it's a mix of regulatory capture incentives and a weird cult ideology where the loud part is P(doom) and the quiet part -- the real part, the inner ideology of the cult -- is that humanity's future depends on this cult being at the steering wheel. Their goal is not to stop AI but to make sure the cult and only the cult control it.

The seeming indoctrination of the incoming Democrats from the most recent electoral cycle is really troubling. I've wondered if maybe a good way to fight it is to point out that this cult has some problematic views on race/IQ, biological determinism, and has a lot of figures in it with really troubling histories around sex and consent. Search for "Rationalist sex cult" and you'll find endless articles about "Rationalist houses," BDSM orgies that play loose with consent, employees feeling like they're being pressured to participate in certain extracurricular activities, etc.

I feel like these allegations pouring out of other circles would be (rightly) a five alarm fire, but for some reason the Rationalist orbit around Anthropic and OpenAI and the whole San Francisco Rationalist cult scene gets a pass.

Tarq0n | 5 hours ago

It's more of a muddle of incentives than anything. Using doomsday cult as a model is neither helpful to the debate nor insightful.
You can debate whether cult is too strong a word. There is definitely an ideological base, a number of charismatic leaders behind that ideology, and a "scene" around it with its own peculiar set of social norms and jargon. It's a scene with a lot of the kinds of allegations you often find around cults: gross sexual behavior, "problematic" opinions about stuff like race and gender, etc.

Is it a cult? It's at least cult-ish. It quacks. Maybe not a duck but it's waterfowl.

All I'm really saying is people should look more deeply into this stuff. If they do, I think they'll see what I'm talking about. It is not hard to find.

senordevnyc | 4 hours ago

lol, who are these charismatic leaders in the AI industry?

jubilee33 | 4 hours ago

People don't like looking deeply at uncomfortable things. They really hate thinking they are on the wrong team or the bad guys. It's why most people will go along with the most insane and horrible things that regimes end up doing... And then in the end claim "we never knew!"

They don't what to know so they consciously or unconsciously shut their eyes.

sdellis | 4 hours ago

I grew up in a cult. Anthropic is not a cult and it's ridiculous to make that claim. It's a business with paid employees who may be aligned with the values they share. I want to work for a company who is aligned with my values, don't you?

At least Anthropic is considering safety and forcing the conversation. They are saying "no" to helping develop AI-weapons for the Pentagon. Maybe the doomsday stuff is a marketing scheme, but I don't care because I think that it's better to move slowly and carefully and to regulate the industry. We have learned a lot since the prior "cult" for which the rally cry was "move fast and break things."

Their rally cry is "regulate everyone else but let us move fast and monopolize things."

sandworm101 | 5 hours ago

Humanity is doomed, but cult members are above humanity. As its de facto parents, those in control will be granted demi-god status by the coming omniscient AI. The IPO is everyone's last best chance to buy into that club before the coming storm. We should all be grateful that they are giving us the opportunity to join so easily.

blurbleblurble | 4 hours ago

exactly, that's the plan

  If they truly believed this stuff had a meaningful chance of being an X-risk to humans, they should stop building it and stop pursuing a giant IPO to finance building it.
Counterpoint: the many long papers and posts discussing why this is a flawed strategy.

If you’re going to write multiple paragraphs deriding scientists, you should at least bother to engage with the basics of their findings. Not just spin up bullshit about BDSM orgies and call it a day.

suddenlybananas | 4 hours ago

Do you have to read every work of theology to be an atheist?

nsagent | 4 hours ago

Certainly not, but if you are not apathetic or a nihilist, then it behooves you to at least read some of them so you can be exposed to the arguments.

calmworm | 4 hours ago

One doesn’t need to read arguments to form thoughts.

skeledrew | 3 hours ago

If you're looking to counter-argue, you have to know the arguments you're looking to counter. Otherwise it's just blowing hot air.

bananaflag | 4 hours ago

Actually, I did read some. A lot of intelligent people are religious so I wanted to make sure there isn't any solid argument I'm missing. And generally I read a lot of arguments from the "other side" (and occasionally changing my mind) because I want to be right about things (or "less wrong", as the saying goes).
You should at least be familiar with their arguments and offer basic rebuttals, if you feel confident that they're wrong and respect them enough to give them some credence (i.e. they're not just flat earther types).

Your reply hinges on "every work", which is far from necessary here. You could read a couple blog posts, a wikipedia page or two, or, yes, some books and papers.

In other words: do you have to read every climate paper to vote for candidates interested in tackling climate change?

simoncion | 4 hours ago

> Counterpoint: the many long papers and posts discussing why this is a flawed strategy.

If starving the project of money and resources by immediately stopping work on it and refusing to do more work on it won't work, then regulation sure as fuck won't do anything.

The sort of people who will go ahead full steam on -say- a biosphere annihilator that they've demonstrated that they are incapable of building safely (and admitted their knowledge of their own incapability) are not the sort of people who will closely follow laws and regulations that constrain their activities in ways they disagree with.

As we've seen here recently, when you have trillions of dollars at your disposal, you can buy a lot of "turning a blind eye" to your past, current, and future violations of relevant laws and regulations. And as we saw in the US in 2008, if you get the right people scared enough, you can get your past violations of law made retroactively legal.

  If starving the project of money and resources by immediately stopping work on it and refusing to do more work on it won't work, then regulation sure as fuck won't do anything.
...again, you're just musing about weak strawmen. The arguments tie into the incentives and tendencies inherent to Capitalism and Nationalism. Thinking up unrelated hypotheticals is not a counterpoint.

svara | 4 hours ago

I think a cynical but equally plausible explanation is they do believe it, they think it's inevitable, so might as well make a lot of money in the meantime.

The take I'm increasingly coming around to is that, yes, you can use AI to build dangerous systems, but the most dangerous things is that narrative of inevitability.

Humans actually have a lot of agency in this, and it will take humans to decide to do something really dangerous with AI. The inevitability narrative conveniently absolves them of any responsibility.

hypfer | 4 hours ago

> I heard someone on a group I'm in humorously say it's the first IPO of an apocalyptic doomsday cult.

I think most people in power know that, but they also know that we're in for a heavy hitting recession if this stops - and no one wants to be the one "causing" this by speaking up against it too much.

Which [put on your tinfoil hats now] might also be why the whole jensen story is how it is, because none of it is "active action". It surely makes people angry, but it doesn't make them angry in a way where they show up at someone's door and go "your decisions are responsible for my economic hardship".

It is probably not that and just random chance, but, functionally, it might be a way of letting this crash that doesn't end up with this style of vigilante/lunatic going around.

meowface | 4 hours ago

I am thankful there are at least some politicians who recognize there is a serious possibility AI may get really, really powerful really soon and really fast.

Either people simply do not think AI will ever get that good, let alone get super good very soon, or they think it but are engaging in some sort of magical thinking about the potential consequences.

I can get the former; it is very uncertain how rapidly capabilities might improve. But if you hypothetically take for granted the potential timelines (which one can be totally justified in not doing!), then the rationalists are just obviously completely correct.

The rationalists really worried about this stuff do in fact think Anthropic is deeply evil and is close to causing horrible catastrophes. You might not be aware of the schisms that have been created as a result of this. Of course they want them shut down or halted.

The more moderate rationalists who are less of doomers basically have similar thoughts on AI as the median Anthropic employee and exec. They believe the upsides of AI are so great, and the grave risks manageable and preventable enough, that AI is worth trying to create.

(I've been a member of the rationalist community since around 2014. Meaning, I am someone who found the SSC blog back then and who comments in their comment section; you don't need to live in California to be a rationalist. Being a rationalist basically means "do you read Eliezer Yudkowsky and Scott Alexander and Kelsey Piper and consider them to be more correct than most other people are". No sexual atypicalities required.)

I discovered Rationalist writings around the same time, and Overcoming Bias further back than that, and was initially enamored. But the more I read the less enamored I became. I was put off by a number of what I might call "thought smells."

TL;DR: Know the quality of a tree by its fruit, and I am not impressed by the fruit, and TBH I thought the tree smelled funny anyway.

One thought smell was the way writers would assert premises as if they were axiomatic when they were not and proceed to reason from them to a conclusion without checking the result empirically. This is more like medieval Scholasticism than science. Of course heavier objects fall faster than lighter objects! Why bother doing the experiment? Isn't it obvious?

https://en.wikipedia.org/wiki/Scholasticism

There's a reason science is obsessed with taking small steps and doing a lot of observations and experiments, and relegating anything not so confirmed to the realm of speculation or hypothesis (a more precise and testable speculation). Science does this even if the math or reasoning look pretty because we've learned that it's very easy to make mistakes in either reasoning or more often selection of premises and other inputs. A precise tree of reasoning fails if a single input is wrong. Any programmer knows this.

I think Rationalism is sort of a neo-Scholasticism, not in the specific sense of trying to reconcile ancient thought with Catholicism (though there's a weird neo-Catholic strain in there somewhere) but in the sense of believing in reasoning from asserted truth to correct outcome. "I am smart and my assertions feel right so I'm correct."

Another was a constant odor of biological determinism, crypto-racism, and IQ elitism. I don't even consider myself a "woke leftist" per se. I dislike these things because I legitimately think they're either wrong or irrelevant as the case may be. I don't think it's a coincidence that neoreaction and similar things emerged from these circles. (Yes I'm aware that many today disavow those things.) The soil was fertile.

Same with Longtermism which is just a rebrand of utilitarianism. I can make a solid Rational(tm) argument that killing everyone with a disability or a measured IQ below, say, 80, would improve long term utilitarian outcomes for the human race and would have a high probability of reducing suffering in the long term. I'm absolutely certain I could write a convincing Rationalist-style essay on this. Ready to push the button? There's a reason this kind of pure utilitarian thought has a poor reputation in actual ethics circles.

Combine Scholastic-style "sterile reasoning" (reasoning without empirical feedback) with utilitarianism and weird absolutist reasoning processes and I don't think that leads to a good place.

Lastly, there was this constant smell of what I now call "intellectual edgelording." The way to win points seemed to be to assert something controversial (monarchism is good, actually!) and then show off your superior intellect by arguing that it's good actually, and of course it also means that you are brave and free-thinking enough to look at it and everyone else is just a coward. It's a good way to look smart via the same mechanism that makes shitty music sound more deep and profound to a 14 year old if the musicians claim to worship the devil.

Forgive me for not taking absolute assertions of P(doom) combined with "and we are the only ones who can save you, but that means we must control this hugely profitable powerful technology and ban open models and independent research for your own good" very seriously when they come from this crowd.

meowface | 3 hours ago

These are mostly valid criticisms. I could respond to a lot of the rest, but all I'll say is I align more with the anti-rightist faction of rationalists. (One could say the anti-eigenrobot anti-wanye faction.) This faction also tends to suffer less from the other issues.
I would never argue that the odds of bad AI outcomes is zero. It’s a powerful technology and thus a large lever. I agree that I’m far more concerned with humans doing bad things with AI than sentient autonomous AI.

I don’t rank “doom” very high though. Bad things? Sure. But good things too. Big lever.

The huge issue I have with the safety crowd, especially this one, is that one of the scenarios I fear the most and also think is realistic is monopolization. I’m afraid of what happens if small numbers of people and governments completely monopolize powerful AI and then use it as a force multiplier to become kings of the world more or less.

My kids living under enhanced AI augmented absolute god kings is IMO a “doom” scenario almost on par with mass death.

This isn’t as far fetched as the other scenarios because we already know for a fact that humans behave this way. We have tons of historical examples. Even if they start benevolent, absolute power distorts the human mind and tends to make people lose touch with reality and lose empathy.

The safety measures the safetyists advocate — banning open models, regulating training, forcing research through gates they control — are precisely the measures most likely to lead to this outcome.

meowface | an hour ago

Would you feel less bad about that if the banning of open models and the strict regulation were simultaneously paired with Anthropic and OpenAI being fully nationalized?

(Not exactly ideal under Trump, so let's say it happens in 2029 under a Democratic party administration.)

vkgcr | 4 hours ago

Anthropic Christmas parties must be wild

tclancy | 4 hours ago

Their day to day is wild by some accounts https://www.verysane.ai/p/ai-safety-is-mostly-a-sex-cult-in-...
> but for some reason

Because there's lots of money involved.

droidjj | 5 hours ago

FYI: When he says, "I have conducted an audit of Anthropic's finances," he means he looked at _publicly available_ information about Anthropic, METR, and a bunch of other orgs. And by the looks of the graph attached to the post, when he says, "I," he means, "Claude."

gerikson | 4 hours ago

I would probably use another model when asking pointed questions about Anthropic...
Yeah this is LLM junk. I cant belive this is getting traction.

stingraycharles | 3 hours ago

It’s so silly that people who are so anti-AI are using those same tools to write their spam.
I wonder if these people will apologize after the catastrophe. Or even admit to themselves how wrong and harmful they were :(

crostlybostly | 4 hours ago

I don’t see any world where a high concentration of models in big tech aligned with USG makes the situation better. If a catastrophe is likely, the best things we can have is a bunch of independent models and compute.

We need equilibrium and models playing defense against bad actors. The more models and independent compute we have the better.

kataklasm | 4 hours ago

I don’t see any world where the regulation of arms makes the situation better. If a catastrophe is likely, the best things we can have is a bunch of good men with guns.

We need good men with guns playing defense against bad men with guns. The more good men with guns we have the better.

Si vis pacem, para bellum.

matkoniecz | 4 hours ago

> We need good men with guns playing defense against bad men with guns.

That is called army and is widely considered as sadly necessary.

What other alternative you propose for neighbors of Russia?

Surrender without fight?

gerikson | 4 hours ago

The catastrophe these people worry about is the absolute 100% extinction of the human race. There will be no-one left to care about who was right or wrong.
That's not true. That's one extremely worrying possibility --perhaps the most worrying possible future of them all, by definition-- but it's far from the only one. There's a decent chance that the first catastrophe is somewhat intentional, either as an act of individual nihilistic terror, religious fervor, capitalistic greed, or nationalistic self-interest.

For example, the newest qwen-4-max model is near SoTA on bioinformatics benchmarks. An engineered virus could easily kill millions before being contained; even an unsuccessful one that killed thousands before we managed to contain or treat it would be a catastrophe. 2+2=catastrophe :(

m3kw9 | 4 hours ago

If LLM's can hack computer system, they can hack political systems.

verdverm | 2 hours ago

societies are more resistant to "hacking" than computers, we have thousands of years of experience

semessier | 4 hours ago

this ironically carries written-by-AI cues

jrflo | 4 hours ago

None of this is new or enlightening information. Everyone has known that Anthropic and METR have a close financial relationship, that's been discussed at length if you follow AI development at all. Of course people investing in AI companies will invest in other AI companies, and many of the existential risk people at Anthropic will donate to AI safety causes...

Routine8502 | 4 hours ago

Explaining why there is a conflict of interest, while interesting, doesn't mitigate it. You and I might know the details, but my conversations with friends/family/colleagues indicate that anyone who isn't deeply in the AI space does not. The way METR communicates information, and the impact it has on public discourse around AI is therefore concerning. The author of this article might be a whack job judging by the other comments, but that doesn't mean he is wrong.

jrflo | 4 hours ago

That's true, but his first few lines of the post are so hyperbolic that it's borderline deceiving. There's no need to call for a Congressional investigation when you find very public information that one company is investing in another that it does business with.

Routine8502 | 3 hours ago

The presupposition of guilt is hyperbolic, yes. But let's take anthropic's own hyperbolic public statements (10% chance of human extinction?) at face value. It follows then that any organization that claims to evaluate or regulate such an entity for safety ought to fall under rather heavy scrutiny. It is not unreasonable to ask questions like "is ant/oai/whatever buying off a non governmental regulator of a field that, by their own admission, has a 10% chance of killing everyone?", nor is it hyperbolic to want Congress to step in.

jrflo | 2 hours ago

Congress needs to regulate AI, not investigate METR. It makes no sense for Anthropic to "buy off" METR when they are giving them full access and funding them 100% voluntarily.

eterpstra | 4 hours ago

In other news, this guy also believes the 5 million or so Muslim Americans don't deserve representation in congress. https://nitter.cf/kevinnbass/status/2121849837444297011#m

meowface | 4 hours ago

Everything this person writes everywhere is dumb ragebaitslop nonsense. I flagged the post, personally.

omnimus | 3 hours ago

Yeah it's some AI psyched conspiracy account. Inconsistent posting of what's gonna create attention.

Stuff about Anthropic is funny because it's not some secret that they try to completely capture the regulatory process. At the same time the account is big AI and Jensen Huang booster i guess not realizing that Nvidia is in bed with ant?

vb-8448 | 4 hours ago

From an ethical standpoint is definitely something that need to be addressed.

Probably it's legal, but how can you trust their evaluation of the competition if that can hurt their wallet?

m-hodges | 4 hours ago

Who is this person? His PFP is in a white coat and his bio says, “Kicked out of TTU med for truth-telling — I want Texas Tech to tell the truth about what it did”. This whole thing is clearly written by a language model.

xattt | 4 hours ago

Drama just follows some people. They are usually the reason.

Noaidi | 4 hours ago

He is a grifter who made a bunch of money on the internet years ago doing health experiments on himself.

I have no idea why this dolt spouting this nonsense is of the front page.

walrus01 | 4 hours ago

A casual review of this person's online profile, searching for "kevin n bass" shows some extreme right wing bigotry.

https://www.google.com/search?q=kevin+n+bass+muslims

Whoever the heck this person is they're completely unqualified to comment on anything AI related, or even computer science in general. It's shocking this post has been upvoted so highly without people considering the source.

tstactplsignore | 4 hours ago

He was kicked out of medical school (because I can't confirm the details, I won't say why, but you can google it), and he tried to spin it as being kicked out for his beliefs about vaccines. He then formed a life as a grifter trying to sell his opinions as an "almost doctor" to right wing media sources. More recently, trying to start vibe coded medical startups. He's kind of a source that's untrustworthy by definition, just puts out whatever might get clicks

rafram | 4 hours ago

He isn’t revealing any secret or even scandalous information here.

What he’s describing is Anthropic trying to build a self-regulation mechanism like most industries have.

That’s not “regulatory capture.”

The intent is to show Congress that the industry can regulate itself, but Congress can disagree.

(Hopefully they will.)

What his tweet does have?

Lots of obnoxious,

short

paragraphs.

chilmers | 4 hours ago

What's with these people who write in once-sentence paragraphs.

Like this.

It's weird, like somebody talking to you in an unnatural cadence.

As if they think you're too stupid to read a whole paragraph.

Or what they're saying is so important.

So, every single sentence must be intoned as if it were a command from God.

Starlevel004 | 4 hours ago

Staccato sentences. It's something that was previously contained in LinkedIn Standard English but the LLM explosion has caused it to leak everywhere else.

aslkalska | 4 hours ago

and who's funding him ? I mean the moment the government gave carte blanche to tech bros - the whole thing was doomed. Don't get me wrong I hate regulations as much as the next guy but it's the only tool against greed and even then it's not that hard to buy an election in the US.

flawn | 4 hours ago

AI-generated slop, this diagram was public for longer and has been posted here, iirc.

neomantra | 4 hours ago

I continuously compare the rise of LLMs with the rise of electronic trading. Regulatory capture was upper right on my bingo card card, while bottom left was involving SROs to limit competition and control narrative. Upper left is venture-funded startups pushing politicians and laws to cement their newfound positions. Center free space is the exciting, yet unintended consequences of interactions between complex systems in a highly competitive environment.

aucls1 | 4 hours ago

dannyw | 4 hours ago

The organisation he referenced has been funding efforts in AI safety since 2015, when ResNet was the latest.

People who have long-held beliefs about AI safety may happen to talk to each other; or even invest in startups like Anthropic that become big; but it’s not necessarily a conspiracy.

Deep learning, and scaling it, was a relatively small space.

specked-citrus | 4 hours ago

METR obviously has some mixed incentives, but this writeup is pure conspiracy theory from an actual conspiracy theorist. Is it really so hard to believe that people can be genuinely concerned that AI has risks?