so so tired of these myopic takes which minimise concerns about extinction risks. leading AI research scientists are saying we might all be gone in 5-10 years and all you can go is “ahhh MaRkEtInG hYpE arggghhh” - it can be both things. please stop shutting down serious, credible whistleblowers the rest of us want to hear from bc you can’t compute that something can be two things at the same time
Alright, that's kinda funny. Sounds like something my younger and dumber brain would have invented as my permission to get high. I'm sure deep down they know it's probably not gonna happen so swiftly.
I would like the government to take this risk seriously. Of course, the serious response to a company which claims that it will, on average, cause ~800 million deaths is a series of cruise missiles.
I drank some coffee today which made me smarter. This allowed me to get out of bed and eat breakfast, which made me even smarter. My IQ will be 700 by tomorrow.
>“[…] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’”
I’m totally okay with people getting rich. But I like to mandate that everyone who is financially involved with a company will be physically tied to the epicenter of the apocalypse that their investments might foster.
/s - because I’m so much less afraid of the AI apocalypse than the very real harm, that the skewed ethics and selfishness of financial elites cause every day.
Just a few days ago I'd be here and say the same as everyone else, sales pitch, hype, IPO.
But very recently, researchers noticed that the latest openAI models are able to directly modify weights of other smaller models, improving their performance a lot: https://rikiyat.github.io/blog/weight-space-programming/ others confirmed the same with opus 5.5. https://www.reddit.com/r/singularity/s/wemgzwnbbb
From that, it doesn't feel like a big step to have next generation models, developed internally right now, being able to improve the weights of larger models, such as themselves. Which, I think we can all agree, would very much count as RSI.
To temper the hype a bit, note that some say that the results are not THAT impressive: https://www.seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/
So yes, possibly IPO hype, but RSI is not as far fetched as it was just a few days ago.
I think we are past the era of a random twitter guy making breakthroughs, any breakthrough will come from a lab with infinite tokens. So when you see a random guy on twitter making big claims, it's usually just a nothingburger (and before you ask, yes, a PhD at MIT does qualify as a random twitter guy).
I agree. RSI should be pushing the frontier. One (already better) model making the performance of a different (smaller or “worse” model) better, perhaps to better match its own performance, isn’t RSI, IMO. It’s one model optimizing another according to some fitness function, probably based on the “better” model’s “judgment.” At no point was a model universally “better” than either on any dimension created. To do that, you would need a fitness function to describe that better model. I believe you still call that “training data,” and these labs have been using synthetic training data for a while, i.e., you trust a model to make it up. I don’t have a citation, but having worked in ML (admittedly not in the AI era), that approach can only get you so far, because if your models were good enough to create new training data of sufficient quality, you wouldn’t need the training data to begin with, since the model would already be that good, and we know it is not, or else you wouldn't be trying to train it. At best, one model can make another more efficient/a better fit against existing judgment/training data — basically a better, more attentive trainer — and that would potentially be a good explanation for why Opus 5.5 is so much more “useful” than Opus 5. It’s better optimized. But it doesn’t actually “know” more.
I think it's also worth mentioning that there are a lot of places where humans employ "judgment" that people -- even experts -- don't agree on the right answer. (This happens frequently in deep design and knowledge work divorced from a clear spec.) Therefore, the training data doesn't really exist, in a sense. It's hard to imagine how you could push any model into that "fuzzy" area where experts disagree because even just face validity ("Do I personally agree with the model?") suffers. The goal or "destination" is not well-defined. If training data for a task doesn't exist, then I don't think you can really teach a model to perform that task. Therefore, I think that's an upper bound to how good models can really ever be, at least quickly. This point might be more controversial, so if anyone disagrees, I'd be interested to hear!
The specific case where this worked was just resolving a reasoning loop for a micro qwen 3 model. Quite impressive, but it was just tweaking some weights to resolve the loop, which let the small model reach it's real potential. This technique for now seems to only address low hanging fruits when it comes to improving the model.
Very marginal improvements can compound... Like no one is setting a timescale on this, are we talking weeks or years? It doesn't really matter, minor self improvements over any amount of time is a significant step into uncharted territory that will probably lead to faster and faster and larger and larger gains.
You can already write self-healing, self-writing code, so the challenge here is probably more on the quality side than any technical limitation, and what this means for the model or the harness and how that’s evaluated.
~~This is not what rsl is. Recursive self learning is the ability of the model to change its own weights based on its own evaluation of the outcome of a task to improve itself. It is something researchers have been trying to do with ML for decades, without the self part and it has never worked.~~
~~The problem is you don't know what weights matter, so updating them to improve one outcome might improve that one thing but make everything else worse. It is like if practicing piano could cause you to forget how to write.~~
~~This is the Holy Grail of AI research, btw. Once you can do it you have a reenforcing feedback loop and you would rapidly beat everyone.~~
That's the first time I hear that term as a synonym for RSI.
But in any case - that's not what RSI is.
RSI is just generally the AI system being able to design an improvement for itself and that improvement increasing its capability so that it can find another improvement. It doesn't have to change its own weights directly.
Yeah same lol — i honestly hadnt heard of rsL before this comment.
I had heard about using synth data to train LLMs, but at least my impression was that was still considered under the umbrella of RSI.
I guess it is sort of its own thing now, although I’m struggling to find any OSS projects that specifically focus on RSL. It seems all the ones mainly speak of RSI, and then have synthetic data generation to help with the training of RSI.
Reddit is such a dumpster fire of mistitling and people not understanding at all what they're reading. Or well, most people obviously not actually even reading what they are commenting on.
Reddit promotes and rewards the first person to reply to a post with visibility, which drives more upvoting and replies, which drives more visibility.
It was redesigned to fill the gap left by original Twitter, where a broader mix of people that don't lean hard right can be influenced by single sentence takes that are reinforced through conversations about a topic's percieved truth rather than learned information.
lol. Lmao even. Anthropic is smoking the glass dick if they thinks this is even remotely true. You’d sooner get a a garbage self regurgitating system than a recursive one.
I expect claims like this and more doom trolling as the IPO draws closer.
The same people who cant fix autocorrect on phones and cant figure out how to fix many of the bugs that have been plaguing software for years need you to believe theyve laid the foundation for a machine god that can do everything and kill everyone.
They know that newsmedia has a massive blindside when it comes to technology and they also know that things meant to scare people will get extra coverage. They also know that social media functions on the same incentives. So rather than emphasizing literally anything useful or good the technology is doing, theyre making all kinds of claims that nobody with a big enough platform cares enough to verify and apparently no government agency exists to audit and shut them down if they're actually a genuine threat to safety.
This is what advertising looks like in the 21st century. Not "here's something that will improve your life and all the things its already doing to improve the quality of life for the average person and resolve one or more persistent problems" its just spectacle and claims of any kind.
Tesla is the most valuable car company in the world, yet it has no minivans, no small vehicle, no all weather 4 wheel drive truck, is out of the price range of most people, and has a CEO that spends all his time online promoting white supremacy and then flat out lying to shareholders. It makes absolutely no sense based on the logic of classic economic theory, but when you understand that facts and evidence doesnt actually matter anymore to investors and the media that represents them, then it starts to make sense why other CEOs and company reps are coming out and just saying anything as well.
"Recursive self-improvement" is nonsense for stupid people. It's like saying I will buy supplements and objectively speaking I will be improved cognitively from them. So then I'll be even smarter once I have the supplements and improve myself more. And then I'll be even smarter and improve more! "Recursive self-improvement"! It's just conflating a bunch of things together. I have noticed that "singularity" type people don't have any clear thinking mechanism, every concept works in a fuzzy way to them.
they’re pretty clear what they mean by RSI. It’s the point at which new models are developed by old models, and at that point we can’t claim to really have control over it
RSI is like a teacher training a student in everything they know, and the student surpasses them because they started from a higher base line. This is how humanity got where it is: We just keep passing on what we know, and the next generation learns even more.
Currently, Ai does most of the work for these companies in terms of building and training the models, but they arent smart enough to do it unsupervised. So while it saves a lot of time, its still a human led effort. But if the ai starts being able to make the next slightly better model without a human in the loop, then it can just pass the job on to that slightly better model. This this newest model can get put to work making the next next model. And so in.
Does that mean we should go over and suck Dario's dick and give him our ram? Nah, hes probably hyping up how good their models are before the IPO so that maybe the bubble wont burst just yet. He needs everyone to believe that he's right about to solve the ai equvilant of cold fusion "real soon bro. Just keep investing."
EverythingScience-ModTeam | an hour ago
Your post has been removed for the following reason:
>2) Maintain scientific integrity
>We do not expect peer review, but we do expect a level of scientific rigour in submissions
oddmetre | 7 hours ago
I’m sure this has nothing to do with the upcoming IPO
newpua_bie | 7 hours ago
Recursive self-inflation
NikkoE82 | 5 hours ago
“We’re so close! Just imagine how that will impact our value in the future!”
gloerkh | an hour ago
you can see everything
Radiant_Plantain_127 | 5 hours ago
Oh yeah don’t forget the 20% chance it might kill us all!
Key_Home_2141 | 4 hours ago
so so tired of these myopic takes which minimise concerns about extinction risks. leading AI research scientists are saying we might all be gone in 5-10 years and all you can go is “ahhh MaRkEtInG hYpE arggghhh” - it can be both things. please stop shutting down serious, credible whistleblowers the rest of us want to hear from bc you can’t compute that something can be two things at the same time
Street_Anxiety2907 | 4 hours ago
Our whole team has started smoking because they're so confident AI will cure all disease and addiction in 5 years. These are 22 year old guys.
Paddyputthepipedown | 3 hours ago
Your team sounds like a bunch of morons.
Own_Unit_6170 | 2 hours ago
Maybe AI will miraculously cure stupidity, too.
nothingInteresting | an hour ago
Is your team up close to unreleased models or are they basing it on where publicly available models are at?
Ssspaaace | an hour ago
Alright, that's kinda funny. Sounds like something my younger and dumber brain would have invented as my permission to get high. I'm sure deep down they know it's probably not gonna happen so swiftly.
Own_Unit_6170 | 2 hours ago
We will not be gone in 5-10 years.
Alarmed-Outside-8683 | 4 hours ago
They never said it wasn't both things. They said it was one of the things.
Membership-Great | 3 hours ago
Intentionally obtuse is never convincing
panna__cotta | 4 hours ago
You weren’t around for Y2K huh?
Membership-Great | 3 hours ago
Well said. I truly wonder what is the motivation for someone to blindly parrot “they’re lying for investor $$$”
I think I’m just going to start blocking those people when I see it because I don’t really want to see their input on anything again
DeltaRofl | 4 hours ago
I would like the government to take this risk seriously. Of course, the serious response to a company which claims that it will, on average, cause ~800 million deaths is a series of cruise missiles.
Brrdock | 7 hours ago
Can we stop reporting these fuckasses' sales pitches?
Atheizm | 7 hours ago
Gasoline is close to becoming the best energy source says gasoline executive.
oracleofnonsense | 3 hours ago
Solar and batteries are clearly a superior technology. — Say it in Elon musk voice.
rei0 | 7 hours ago
Those private islands aren’t going to buy themselves.
hobopwnzor | 7 hours ago
Wait I thought they achieved RSI like a year ago, and then 6 months ago, and then 3 months ago
NikkoE82 | 5 hours ago
They are recursively achieving it.
tzaeru | 4 hours ago
Idk what news that has been, but can't remember Anthropic making such a claim.
king_crizzy | 4 hours ago
I drank some coffee today which made me smarter. This allowed me to get out of bed and eat breakfast, which made me even smarter. My IQ will be 700 by tomorrow.
Cephell | 7 hours ago
Convenient, right before IPO.
tzaeru | 4 hours ago
I mean...
It's from their IPO registration statement.
>“[…] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’”
DanimalPlays | 8 hours ago
Bullshit.
DevilsInkpot | 5 hours ago
I’m totally okay with people getting rich. But I like to mandate that everyone who is financially involved with a company will be physically tied to the epicenter of the apocalypse that their investments might foster.
/s - because I’m so much less afraid of the AI apocalypse than the very real harm, that the skewed ethics and selfishness of financial elites cause every day.
account22222221 | 5 hours ago
Again, as they have been for the last 2 years it seems like
ancientevilvorsoason | 5 hours ago
Any minute now...
sf-keto | 4 hours ago
Dario will claim whatever boosts his IPO. What’s new?
Aliktren | 8 hours ago
wait, I've seen this movie...
Aggressive_Roof488 | 6 hours ago
Just a few days ago I'd be here and say the same as everyone else, sales pitch, hype, IPO.
But very recently, researchers noticed that the latest openAI models are able to directly modify weights of other smaller models, improving their performance a lot: https://rikiyat.github.io/blog/weight-space-programming/ others confirmed the same with opus 5.5. https://www.reddit.com/r/singularity/s/wemgzwnbbb
From that, it doesn't feel like a big step to have next generation models, developed internally right now, being able to improve the weights of larger models, such as themselves. Which, I think we can all agree, would very much count as RSI.
To temper the hype a bit, note that some say that the results are not THAT impressive: https://www.seangoedecke.com/llms-are-not-superhuman-at-reading-and-writing-neural-network-weights/
So yes, possibly IPO hype, but RSI is not as far fetched as it was just a few days ago.
Icy-Concentrate2076 | 4 hours ago
I think we are past the era of a random twitter guy making breakthroughs, any breakthrough will come from a lab with infinite tokens. So when you see a random guy on twitter making big claims, it's usually just a nothingburger (and before you ask, yes, a PhD at MIT does qualify as a random twitter guy).
parallax3900 | 4 hours ago
Is it RSI though? It could just be LLM technical debt and training data whacking on piles of LLM debt and older training data.
aboothe726 | 4 hours ago
I agree. RSI should be pushing the frontier. One (already better) model making the performance of a different (smaller or “worse” model) better, perhaps to better match its own performance, isn’t RSI, IMO. It’s one model optimizing another according to some fitness function, probably based on the “better” model’s “judgment.” At no point was a model universally “better” than either on any dimension created. To do that, you would need a fitness function to describe that better model. I believe you still call that “training data,” and these labs have been using synthetic training data for a while, i.e., you trust a model to make it up. I don’t have a citation, but having worked in ML (admittedly not in the AI era), that approach can only get you so far, because if your models were good enough to create new training data of sufficient quality, you wouldn’t need the training data to begin with, since the model would already be that good, and we know it is not, or else you wouldn't be trying to train it. At best, one model can make another more efficient/a better fit against existing judgment/training data — basically a better, more attentive trainer — and that would potentially be a good explanation for why Opus 5.5 is so much more “useful” than Opus 5. It’s better optimized. But it doesn’t actually “know” more.
I think it's also worth mentioning that there are a lot of places where humans employ "judgment" that people -- even experts -- don't agree on the right answer. (This happens frequently in deep design and knowledge work divorced from a clear spec.) Therefore, the training data doesn't really exist, in a sense. It's hard to imagine how you could push any model into that "fuzzy" area where experts disagree because even just face validity ("Do I personally agree with the model?") suffers. The goal or "destination" is not well-defined. If training data for a task doesn't exist, then I don't think you can really teach a model to perform that task. Therefore, I think that's an upper bound to how good models can really ever be, at least quickly. This point might be more controversial, so if anyone disagrees, I'd be interested to hear!
JarPicklez | an hour ago
LLMs are already a big pile of technical debt as it is.
parallax3900 | an hour ago
Been saying this for two years and I keep getting told "skill issue" or some shit
xkalibur3 | 2 hours ago
The specific case where this worked was just resolving a reasoning loop for a micro qwen 3 model. Quite impressive, but it was just tweaking some weights to resolve the loop, which let the small model reach it's real potential. This technique for now seems to only address low hanging fruits when it comes to improving the model.
IndependentLog6441 | an hour ago
Very marginal improvements can compound... Like no one is setting a timescale on this, are we talking weeks or years? It doesn't really matter, minor self improvements over any amount of time is a significant step into uncharted territory that will probably lead to faster and faster and larger and larger gains.
JarPicklez | an hour ago
It’s very very hard to predict this kind of thing
fluffyp0tat0 | 7 hours ago
More like recursive self-destruction, considering LLM's autonomous coding capabilities.
YumDood | 7 hours ago
You can already write self-healing, self-writing code, so the challenge here is probably more on the quality side than any technical limitation, and what this means for the model or the harness and how that’s evaluated.
joshocar | 6 hours ago
~~This is not what rsl is. Recursive self learning is the ability of the model to change its own weights based on its own evaluation of the outcome of a task to improve itself. It is something researchers have been trying to do with ML for decades, without the self part and it has never worked.~~
~~The problem is you don't know what weights matter, so updating them to improve one outcome might improve that one thing but make everything else worse. It is like if practicing piano could cause you to forget how to write.~~
~~This is the Holy Grail of AI research, btw. Once you can do it you have a reenforcing feedback loop and you would rapidly beat everyone.~~
tzaeru | 4 hours ago
That's the first time I hear that term as a synonym for RSI.
But in any case - that's not what RSI is.
RSI is just generally the AI system being able to design an improvement for itself and that improvement increasing its capability so that it can find another improvement. It doesn't have to change its own weights directly.
tigerhuxley | 2 hours ago
Yeah same lol — i honestly hadnt heard of rsL before this comment.
I had heard about using synth data to train LLMs, but at least my impression was that was still considered under the umbrella of RSI.
I guess it is sort of its own thing now, although I’m struggling to find any OSS projects that specifically focus on RSL. It seems all the ones mainly speak of RSI, and then have synthetic data generation to help with the training of RSI.
tigerhuxley | 6 hours ago
RsL?..
joshocar | 6 hours ago
recursive self learning. different word for recursive self improvement
elmo298 | 6 hours ago
Anthropic tried to be a good guy and quickly turned into one of the worst players
tzaeru | 4 hours ago
Reddit is such a dumpster fire of mistitling and people not understanding at all what they're reading. Or well, most people obviously not actually even reading what they are commenting on.
UnpluggedUnfettered | 3 hours ago
Reddit promotes and rewards the first person to reply to a post with visibility, which drives more upvoting and replies, which drives more visibility.
It was redesigned to fill the gap left by original Twitter, where a broader mix of people that don't lean hard right can be influenced by single sentence takes that are reinforced through conversations about a topic's percieved truth rather than learned information.
Tada.
The aristocrats.
Alarmed-Outside-8683 | 4 hours ago
That's because everyone is pushing a narrative to get others on their side. The comment is a post in itself.
That-Advance-9619 | 8 hours ago
...A-hah.
frazorblade | 8 hours ago
Singularity intensifies
raydditor | 7 hours ago
I thought AGI was here.
Random_182f2565 | 6 hours ago
Just like me!
cr0wburn | 6 hours ago
Go for it, lets see what happens!
DeerOnARoof | 5 hours ago
Completely unrelated to the IPO.
maniacleruler | 5 hours ago
lol. Lmao even. Anthropic is smoking the glass dick if they thinks this is even remotely true. You’d sooner get a a garbage self regurgitating system than a recursive one.
SweetNeo85 | 4 hours ago
More fantasy.
curiouscuriousmtl | 4 hours ago
This guy looks like a secondary character in a Goof Troop direct to TV movie voiced by Rick Moranis
Old_Satisfaction_233 | 4 hours ago
Things are in the news now, the secrecy money drain is visible; desperation is showing…
ComicsEtAl | 4 hours ago
Oh, so despite the self-serving warnings, everybody’s just pushing ahead, yeah?
golfstreamer | 3 hours ago
The term recursive self improvement is ambiguous enough for this to have been true for years.
Vanillas_Guy | 3 hours ago
I expect claims like this and more doom trolling as the IPO draws closer.
The same people who cant fix autocorrect on phones and cant figure out how to fix many of the bugs that have been plaguing software for years need you to believe theyve laid the foundation for a machine god that can do everything and kill everyone.
They know that newsmedia has a massive blindside when it comes to technology and they also know that things meant to scare people will get extra coverage. They also know that social media functions on the same incentives. So rather than emphasizing literally anything useful or good the technology is doing, theyre making all kinds of claims that nobody with a big enough platform cares enough to verify and apparently no government agency exists to audit and shut them down if they're actually a genuine threat to safety.
This is what advertising looks like in the 21st century. Not "here's something that will improve your life and all the things its already doing to improve the quality of life for the average person and resolve one or more persistent problems" its just spectacle and claims of any kind.
Tesla is the most valuable car company in the world, yet it has no minivans, no small vehicle, no all weather 4 wheel drive truck, is out of the price range of most people, and has a CEO that spends all his time online promoting white supremacy and then flat out lying to shareholders. It makes absolutely no sense based on the logic of classic economic theory, but when you understand that facts and evidence doesnt actually matter anymore to investors and the media that represents them, then it starts to make sense why other CEOs and company reps are coming out and just saying anything as well.
GeorgeSThompson | 2 hours ago
"RSI" could be coming imminently.
Could is doing a lot of heavy lifting here surely if its imminently they will know
NIRPL | 2 hours ago
Yayyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
thatsalotofrhinos | 2 hours ago
Elon Musk said we would have fully self driving cars by 2016.
manystripes | 2 hours ago
Doesn't "recursive self improvement" describe pretty much all machine learning?
Own_Unit_6170 | 2 hours ago
RSI isn't possible. Continual human input is required to keep AI systems on track.
whatidoidobc | 2 hours ago
Anyone fooled by this deserves it.
beigechrist | 35 minutes ago
Quick, late-term abortion for Ai now!
CountryOk6049 | 5 hours ago
"Recursive self-improvement" is nonsense for stupid people. It's like saying I will buy supplements and objectively speaking I will be improved cognitively from them. So then I'll be even smarter once I have the supplements and improve myself more. And then I'll be even smarter and improve more! "Recursive self-improvement"! It's just conflating a bunch of things together. I have noticed that "singularity" type people don't have any clear thinking mechanism, every concept works in a fuzzy way to them.
GeorgeSThompson | 2 hours ago
I mean yeah, we have recursive self improvement. AIs dont yet
Bat_Shitcrazy | 2 hours ago
they’re pretty clear what they mean by RSI. It’s the point at which new models are developed by old models, and at that point we can’t claim to really have control over it
IndependentLog6441 | an hour ago
Yeah exactly, humans have had RSI for thousands of years, that's literally how we're having this conversation.
ChaosMilkTea | an hour ago
Thats a sever misunderstanding.
RSI is like a teacher training a student in everything they know, and the student surpasses them because they started from a higher base line. This is how humanity got where it is: We just keep passing on what we know, and the next generation learns even more.
Currently, Ai does most of the work for these companies in terms of building and training the models, but they arent smart enough to do it unsupervised. So while it saves a lot of time, its still a human led effort. But if the ai starts being able to make the next slightly better model without a human in the loop, then it can just pass the job on to that slightly better model. This this newest model can get put to work making the next next model. And so in.
Does that mean we should go over and suck Dario's dick and give him our ram? Nah, hes probably hyping up how good their models are before the IPO so that maybe the bubble wont burst just yet. He needs everyone to believe that he's right about to solve the ai equvilant of cold fusion "real soon bro. Just keep investing."