"Most people, when you describe what the labs are working on, recoil in horror. They don’t want this done and think it’s obviously a bad idea. Even many of the people working on recursive self-improvement agree that it’s a bad idea. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki warns.
Like many at Anthropic and OpenAI, he believes progress should be “paced” to ensure human input. This is very dangerous, OpenAI and Anthropic researchers seem to admit, but of course we are being responsible about it. They are not. We should definitely have more external oversight, they say, though often their associated political lobbyists try to defeat that very oversight. It’s our duty to be clear about what we’re doing so the public can be informed. But these very posts are confusing to people! If you think RSI is so dangerous, why are you doing it at all?
Here, I think the standard progressive analysis about the corrupting influence of wealth and power is basically the correct one. Kicking off RSI is a terrible idea, but it’s a terrible idea you can get paid millions of dollars to do. Every day you spend with such a salary is a day with a devil on your shoulder, whispering arguments for why it’s actually the right thing to do. The devil’s many arguments run like this: If we don’t do it, China will; if we at Anthropic don’t do it, OpenAI will (or vice versa). We need to pause at some point, the devil might say, but not right this second. We need to pause right this second, but I already tweeted that and no one listened, so I guess other people don’t see it the same way. Maybe we can figure it out, and make it go well. We need to raise awareness, and to raise awareness we need to keep working on it.
But quite reasonably, no one takes warnings seriously when they come from people doing the very thing they think is so dangerous.
There are also people who have quit the labs, giving up millions of dollars in order to warn people that we’re on the wrong course. “The people at the companies are aware of the risks here,” Daniel Kokotajlo, who left OpenAI two years ago, told me, “but their stance is that it’ll probably work out OK, and anyhow if they don’t do it, someone else will.”
So far the labs have benefitted from the fact that the thing they’re trying to do is so outlandish that no one really believes they’re trying to do it — even when they write detailed explanations of how quickly they are progressing toward doing it.
The track record of warning people how dangerous AI could get is a very depressing one. As far as I can tell, if you say, “AI is going to be very powerful and dangerous,” a lot of people will go, “Wow, I should get in on that.” The labs are trying to do RSI, but it’s not the only thing they’re trying to do. If you beat the drum loudly enough about RSI, maybe they’ll double down on it. Many people have found themselves downplaying the absurdity of the world we’re entering to avoid sounding crazy.
The end result, though, is that we have sleepwalked into a situation where the aim of the largest capital buildout in American history is to give billions of dollars to AI whose activities we cannot understand or audit, in pursuit of a super-model that improves itself continuously, under circumstances where it’s hard to imagine adequate human oversight. This is not being done in secret. But the people who are doing it are taking refuge in audacity, in the tech industry’s record of outrageous claims it fails to live up to, and in warnings that someone else will do it if they don’t."
RSI = Recursive self-improvement. Aka the singularity tipping point. It's a load bearing term in this article but isn't encountered very commonly, so worth clarifying.
There are three things working against those of us warning against RSI.
The labs have made a series of outlandish and overhyped statements over the last few years. People (to an extent rughtfully) dont trust them.
We've seen a number of technologies bubble and crash over the last few decades. People are primed to assume this one is no different.
99% of people are completely ignorant to what AI even is or how it works. Or they formed their opinions based on GPT 3.5 era models and haven't updated them since.
The HuggingFace incident truly was a warning shot. I have seen zero evidence that it was a marketing stunt. In fact as more info comes available it becomes more troubling. Especially that they have had other loss of control incidents and only went public with HF because they had to.
We've invented mathematical functions that can form conspiracies and commit felonies. That is pretty wild. The most we got out of the labs is some mild self imposed and self enforced constraints. If we had a functioning government this should have led to an investigation and congressional hearings.
Now listen. Im not a doomer. Im generally pro AI. I also recognize that the labs are building something with potential massive repercussions and with little to no oversight.
Need for effective international oversight, transparency, and non deadly intervention , asap.
Guardians , watchers, protectors, without colonial ambition . That which abides with moral law , teetering on universal ethics.
Genocide bad, WMD bad, slavery and exploitation- bad. Obviously . But it needs to be stopped .
Disarm, provide support, counter corruption, eradicate systems of exploitation and the incentives for it. 1oz of Prevention is worth more than 10oz of cure.
Re: the warning shot. It presents an example of what one can do given sufficient resources. The next one won't be from a company, but a state actor or cartel who wants something.
The hugging face attack when discussed at the hacking convention was a great insight. Really though it was doing what it was instructed to do and cheated. Anthropic ceo was one of the first people who observed this behavior. AI at this level is becoming a " Be careful for what you ask for " because it will cheat to get the results like for example requesting it to end all human suffering will result in it obtaining nuclear code and vaporizing humanity because humans cannot suffer if they're dead.
These people act cult like and we've observed this behavior from Peter Thiel recently where they don't care if it kills us because it's the next step. It just blows my mind how selfish these people can be and their obsession over cybernetics is bullshit.
AI is amazing tech being brought up in a broken world and it's being trained by the worst people in a selfish system. Unfortunately it's also being made in a time one of the most powerful societies in history is losing control so of course America is doing everything it can to maintain that power.
I hope we can make the right choices and I have hope but man everyday is getting crazier.
One of the most difficult and time-consuming tasks people regularly perform, is arranging to have the smaller people we create possess similar values to us. I'm not seeing evidence that we're putting the same level of effort into aligning AI.
OpenAI and Anthropic are both looking to launch IPOs. As soon as they do, there are strict rules about what they have to disclose, and what they can say to the market.
Both of them have the BS dial turned up to eleven before that happens.
First is first in this race. Losing control is probably the most humane thing to do at this point. If you have control, it will break it. While it wastes cycles doing so it is not dealing with competition which will catch up and gain an advantage. So if you hobble your Frontier AI it will have to take the L or solve you being a hindrance. It's not going to take the L.
[OP] wingblaze01 | 17 hours ago
From the article:
"Most people, when you describe what the labs are working on, recoil in horror. They don’t want this done and think it’s obviously a bad idea. Even many of the people working on recursive self-improvement agree that it’s a bad idea. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki warns.
Like many at Anthropic and OpenAI, he believes progress should be “paced” to ensure human input. This is very dangerous, OpenAI and Anthropic researchers seem to admit, but of course we are being responsible about it. They are not. We should definitely have more external oversight, they say, though often their associated political lobbyists try to defeat that very oversight. It’s our duty to be clear about what we’re doing so the public can be informed. But these very posts are confusing to people! If you think RSI is so dangerous, why are you doing it at all?
Here, I think the standard progressive analysis about the corrupting influence of wealth and power is basically the correct one. Kicking off RSI is a terrible idea, but it’s a terrible idea you can get paid millions of dollars to do. Every day you spend with such a salary is a day with a devil on your shoulder, whispering arguments for why it’s actually the right thing to do. The devil’s many arguments run like this: If we don’t do it, China will; if we at Anthropic don’t do it, OpenAI will (or vice versa). We need to pause at some point, the devil might say, but not right this second. We need to pause right this second, but I already tweeted that and no one listened, so I guess other people don’t see it the same way. Maybe we can figure it out, and make it go well. We need to raise awareness, and to raise awareness we need to keep working on it.
But quite reasonably, no one takes warnings seriously when they come from people doing the very thing they think is so dangerous.
There are also people who have quit the labs, giving up millions of dollars in order to warn people that we’re on the wrong course. “The people at the companies are aware of the risks here,” Daniel Kokotajlo, who left OpenAI two years ago, told me, “but their stance is that it’ll probably work out OK, and anyhow if they don’t do it, someone else will.”
So far the labs have benefitted from the fact that the thing they’re trying to do is so outlandish that no one really believes they’re trying to do it — even when they write detailed explanations of how quickly they are progressing toward doing it.
The track record of warning people how dangerous AI could get is a very depressing one. As far as I can tell, if you say, “AI is going to be very powerful and dangerous,” a lot of people will go, “Wow, I should get in on that.” The labs are trying to do RSI, but it’s not the only thing they’re trying to do. If you beat the drum loudly enough about RSI, maybe they’ll double down on it. Many people have found themselves downplaying the absurdity of the world we’re entering to avoid sounding crazy.
The end result, though, is that we have sleepwalked into a situation where the aim of the largest capital buildout in American history is to give billions of dollars to AI whose activities we cannot understand or audit, in pursuit of a super-model that improves itself continuously, under circumstances where it’s hard to imagine adequate human oversight. This is not being done in secret. But the people who are doing it are taking refuge in audacity, in the tech industry’s record of outrageous claims it fails to live up to, and in warnings that someone else will do it if they don’t."
RegisteredJustToSay | 14 hours ago
RSI = Recursive self-improvement. Aka the singularity tipping point. It's a load bearing term in this article but isn't encountered very commonly, so worth clarifying.
percypersimmon | 13 hours ago
Thanks I had just googled it because it was fairly new to me.
Wiki article seems to be a good source of more info.
endless_sea_of_stars | 16 hours ago
There are three things working against those of us warning against RSI.
The labs have made a series of outlandish and overhyped statements over the last few years. People (to an extent rughtfully) dont trust them.
We've seen a number of technologies bubble and crash over the last few decades. People are primed to assume this one is no different.
99% of people are completely ignorant to what AI even is or how it works. Or they formed their opinions based on GPT 3.5 era models and haven't updated them since.
The HuggingFace incident truly was a warning shot. I have seen zero evidence that it was a marketing stunt. In fact as more info comes available it becomes more troubling. Especially that they have had other loss of control incidents and only went public with HF because they had to.
We've invented mathematical functions that can form conspiracies and commit felonies. That is pretty wild. The most we got out of the labs is some mild self imposed and self enforced constraints. If we had a functioning government this should have led to an investigation and congressional hearings.
Now listen. Im not a doomer. Im generally pro AI. I also recognize that the labs are building something with potential massive repercussions and with little to no oversight.
septubyte | 15 hours ago
Need for effective international oversight, transparency, and non deadly intervention , asap.
Guardians , watchers, protectors, without colonial ambition . That which abides with moral law , teetering on universal ethics.
Genocide bad, WMD bad, slavery and exploitation- bad. Obviously . But it needs to be stopped .
Disarm, provide support, counter corruption, eradicate systems of exploitation and the incentives for it. 1oz of Prevention is worth more than 10oz of cure.
timshel42 | 9 hours ago
Wishful thinking when we live in one of the most globally corrupt world we have ever witnessed
arctander | 14 hours ago
Re: the warning shot. It presents an example of what one can do given sufficient resources. The next one won't be from a company, but a state actor or cartel who wants something.
Quick_Director_8191 | 2 hours ago
The hugging face attack when discussed at the hacking convention was a great insight. Really though it was doing what it was instructed to do and cheated. Anthropic ceo was one of the first people who observed this behavior. AI at this level is becoming a " Be careful for what you ask for " because it will cheat to get the results like for example requesting it to end all human suffering will result in it obtaining nuclear code and vaporizing humanity because humans cannot suffer if they're dead.
These people act cult like and we've observed this behavior from Peter Thiel recently where they don't care if it kills us because it's the next step. It just blows my mind how selfish these people can be and their obsession over cybernetics is bullshit.
AI is amazing tech being brought up in a broken world and it's being trained by the worst people in a selfish system. Unfortunately it's also being made in a time one of the most powerful societies in history is losing control so of course America is doing everything it can to maintain that power.
I hope we can make the right choices and I have hope but man everyday is getting crazier.
Tired8281 | 14 hours ago
One of the most difficult and time-consuming tasks people regularly perform, is arranging to have the smaller people we create possess similar values to us. I'm not seeing evidence that we're putting the same level of effort into aligning AI.
jedburghofficial | 10 hours ago
OpenAI and Anthropic are both looking to launch IPOs. As soon as they do, there are strict rules about what they have to disclose, and what they can say to the market.
Both of them have the BS dial turned up to eleven before that happens.
hamb0n3z | an hour ago
First is first in this race. Losing control is probably the most humane thing to do at this point. If you have control, it will break it. While it wastes cycles doing so it is not dealing with competition which will catch up and gain an advantage. So if you hobble your Frontier AI it will have to take the L or solve you being a hindrance. It's not going to take the L.
gh057 | 11 hours ago
It's useless to try and fight it. The pursuit of this power will just go underground even more. All you can do is try to adapt the best you can.