Isn't Sam Altman accused of being a habitual liar by a lot of people, including the richest man on the planet, the arguably best (certainly longest-lasting) AI researcher, the most famous AI ethics researcher, and half of his own board?
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.
TLDR; someone used some fake graphs and probabilities pulled out of thin air to justify an unwarranted conclusion based on a false analogy to a situation that is far from clear.
Dwarkesh has been skeptical about AI claims for a long time. In his interview with Dario earlier this year, he pressed him hard. So this is a fairly recent change of opinion.
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
That also matches my observation, particularly in economics once you factor in a scarce resource. But it's not like I have a convincing metamodel to say whether that or the other distribution is more common. But neither do people who fervently believe in the exponential...
Taking a step back, it's basically getting very good in one arena, with who knows how much money is being spent refining things. But nobody appears to be working on the other aspects of intelligence that I (as a lifelong sf reader) would expect from an AI, let alone a superintelligent one. There is no permanence of anything except at the "build the next model" level and some notes it keeps and can scan quickly. Instead of an intelligence that focuses on you for a moment, you're getting a new copy every time you interact with it after a certain amount of time. There is no "game loop". It's just there to respond to you and be ordered about. It will never reply in a text chat hours later wondering how your latest meeting went. It doesn't have a concept of concepts. It regurgitates words and code in helpful ways, but there is no understanding there. It doesn't know what it means when it says things, it's all just a (very impressive!) act.
I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.
I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)
It's coming for everyone: there's a startup that introduced robot arms that promise to learn anything 1 shot: if it holds - and there's little wiggle room for one shot - it would be a good indicator that manual labor is also soon gone.
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
This is jumping the real world concerns into the realm of science fiction. How about we engage with the premise that before the AGI god comes into being and annoints the Rationalists as it's chosen people, there will be people who will try and use this to enrich themselves and immiserate others.
Every year we are making the "science fiction" of the previous year come true. The trajectory has been clear, the insider info from lab employees themselves are that the scaling laws continue unabated. The coding of the last 40 years is gone in a flash. Unanswered math problems are being solved every week and the top mathematicians of the world see crisis. The only "science fiction" is hyperbole about anointing rationalists, it's a strawman no one holds, X-risk concern is about disregarding everyone.
"there will be people who will try and use this to enrich themselves and immiserate others."
This is already happening from Big Tech and VC's, all the big money. You should try listening to the people that have shirked enormous fortunes in order to tell the truth about things. Some examples:
Sarcastically quipping that the AGI god would anoint the Rationalists as it's chosen people is like saying that the global warming god is going to come along and anoint the Climatologists as its chosen people. It's like saying Al Gore is a reptile and is excited for the universally hot, humid future.
Everybody who thinks they're "the chosen people" predicts the end of the world is coming, but not everybody who predicts the end of the world is coming thinks they're "the chosen people". Maybe the end of the world is coming and it's Just Bad.
Not a chance: if there was even a 1% chance of this being true you'd see something entirely different happening. If AI research self-reinforces, as Anthropic publishes (but doesn't actually believe, it can't), then it will win from all the economic activity on the planet. The major consequence of that is that money (as a source of labor, ie. to get people to do stuff for you, to get stuff for you) is now worthless.
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all just turned more valuable than their weight in pure gold (assuming they're more useful than gold, which is all of them). Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance, but not super great. When it comes to strength it is beaten by 2-day old chewing gum. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, not something else. So they are either stupid or they know 99%+ sure they do not have superintelligence. Because if they do let superintelligence loose, mining countries will have them by the balls and will squeeze. Hard. Yet the money all these people are collecting boatloads of will be useless. And yet, they want the money, not the raw materials! That tells you what they believe: that the money is more valuable than raw materials ... AND more valuable than superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
Not really. You are assuming that ASI will just win irrespective of anything else, 2 seconds after it starts. How would it do that? By launching the nukes? And if it takes time, everyone else has time to catch up ... and raw materials will be the determining factor.
So that's not what happen. Yes, someone will be first, but the second will not be far behind.
Once you have ASI, raw material owners and the governments they're in will PICK the winner from the choices available. Just like they did in the 20th century.
To say nothing of the consequences of what is currently happening with Chinese models only 2 months behind frontier labs. Even if it's 8 months or 2 years or whatever. ASI will not build an insurmountable advantage in the world economy in 2 months, especially not if some raw materials players sabotage it (which seems likely in the extreme, and is simply what raw materials players are already doing, so they don't even have to realize what's happening and start doing it)
You are assuming that ASI won't just win, simply because you can't imagine it? Examples:
- Sophisticated cyberattacks to disable or quietly sabotage competing labs
- Undermining nuclear second-strike capability by locating submarines and infiltrating command-and-control networks
- Fast-replicating nanotechnology
- Superpersuasion (I am dubious on this)
etc., backed by global surveillance to preempt sabotage efforts. If you haven't read Superintelligence you should, all the leaders of the labs certainly have.
I just totally exclude that, because that would be a totally uncontrollable AI, which would be "huge meteor strikes the earth" type of scenario. Extinction level event stuff. If that happens, we're all dead, including Altman and Amodei, Google and Meta management.
So this level superintelligence also falls under the statement "if management believed that, this isn't what you'd see them do".
Plus your statement doesn't explain why they're after money.
You cannot control or align something 10x your intelligence. That would be like a dog controlling a human. For one thing, it won't even work after a short while because the dog's requests are going to be stupid. Except Altman and Amodei will be the dogs.
I'm with the people who really doubt alignment matters.
Besides, one thing keeps coming up in the open models: removing alignment makes them better, more interesting and more creative (difference for fiction, games, and ... well look at civitai is enormous).
And alignment also opens up an avenue to beat these models. And this will be a new level of moral dilemma: you can't judge, because the AI is too complex, too fast, too smart, ... and you cannot even react in time.
> ... they are after money because it allows them to scale and hopefully control AGI/ASI ...
Makes no sense for companies that are already making their own chips. If they truly believed they could figure it out FAST, they'd be buying sand (and it's not everywhere: apparently the purity is a big deal and not available in many places)
I feel you skipped the step, that explains how this works in democracies. Musk was just not able to buy an election, and decidedly so.
It might all end very badly, but we should not be so prone to presume bad outcomes without explanations, that are just as good, as we would expect for good outcomes.
> Musk was just not able to buy an election, and decidedly so
This time he wasn't able to. But the time before (2024) he seemed to have some effect. I'm sure that he and the other oligarchs are going to learn from this midterm election and try different approaches in the future. Think of the 2026 election as one of his crashed rockets. They learn from those crashes - iterative improvement. They'll figure out ways to be more sneaky next time.
This exchange reads to me like a conversation from 1938 in Berlin. How dangerous are they, those sneaky Jews?
If you had millions of followers and billions of dollars, and you were convinced that America will go down if, say, Trump wins an election, would you not use the influence you have?
One could argue that you’d be a coward if you didn’t - having the conviction, having the means, but not wanting to stick out your neck.
Humans amassing money in an AI world is like squirrels amassing nuts when humans are logging the forest.
If we want to have a planet left we need to make it so laughably inefficient to run AI ground-side that no one bothers.
Extremely high temperature semiconductor research to run datacenters in dyson swarms around stars/black holes is fundamental research for human survival at this point.
I feel like this is why more distributed control or open ownership of AI models is such an important goal for regulators. As AI accelerates its own development, the private companies that are initially maybe only a few months ahead of open-source will widen the gap more and more.
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
> When is superdupermegaintelligence going to be announced?
Isn't it already here? Infallible God-King Trump and all his ultra-genius tech-bro buddies delivered superintellgence already, and due to exponential growth it should already exist by this time, no? By this time next week it will have solved all problems and made humans obsolete.
>majority of this acceleration today is likely attributable to Anthropic’s extensive use of AI to do R&D
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
He's making an argument and you're free to engage with it or not, if you don't read anything by anyone who doesn't meet your credentials quota then save your time and skip it.
it is reasonable to ask if a person has done something relevant to their opinion - lots of people have opinions on things that they don't understand. is there a benefit in giving credence to unsubstantiated opinions?
So? Are we all supposed to stop using the term, despite it being around since like the late 90s? Sorry, I don't let politicians control my life like that.
I understood that the author was using the actual term, before it was hijacked by a malignant narcissist as a perversion of the meaning of words and an indicator of fealty (which only applies to Federal employees in the Executive Branch, by the way).
If AI is a verifiable domain, how do you get AIs to construct benchmarks to apply to themselves?
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
There are hundreds of verifiable tasks in AI development as simple benchmarks on the models themselves, such as
1. Performance engineering ability
2. Ability to reconstruct visuals in code alone
3. Ability to orchestrate cheaper models towards a given task
4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
A lot of AI research can be already automated. The experimental part of work can be tonally outsourced to agents (at least for the topics that I am interested in). I had some success with outsourcing theoretical work as well. These days I use quite advanced orchestration. Interestingly, the most difficult part still is writing good stories for research papers. There I can still beat any AI. And when it comes to research we have been already covered by an avalanche of AI slop. There was always slop in AI research but now it is the fully automated slop. A few years ago there was some human-made slop, now it is being factory produced. AI can do good research on its own, but good research is rare in general.
I think the next step for developers is to combine existing technologies. For example, in the medical imaging space that are thousands of published algorithms that were implemented once (by the researcher who published the paper), but never used. AI can now read the paper and one-shot an implementation. I do wonder if there are breakthroughs to be had by combining such algorithms.
Almost certainly yes - I’ve tried this for some imaging problems - but what tends to happen is you find that the method is fragile and only works well on a narrow task, or you need some specific data that’s a pain to label, or the results are strongly overfit. Reproducibility crisis type stuff. Which makes sense when you think about how narrowly scoped many papers are, and the fallibility of the average grad student. Also survivorship bias is strong. Lots of papers are forgotten about because the methods didn’t really work for other things. Sometimes it doesn’t matter, maybe it’s fine to have a method that only works on some niche type of data that you have.
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
I work as a research engineer within an Alphabet company, and I have been in the industry for almost 15 years. I am in the same place as Dwarkesh. I have seen some things that have absolutely shocked me within the last two months. A team of agents rewriting a model from scratch that had had many years worth of R&D spent on it, beating the original in just a few days. An agent that anyone in the company can just message an experiment id and say 'improve this', and it will come back a few hours later with like 20 issues, having examined checkpoint weights and gradient steps.
These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
What options does humanity have? All I can think of is
1. Pause AI development and allow humans to stay in control a bit longer while we work on making AI safer, allowing the field to advance bit by bit (and hoping this plan even works)
2. Allow AI to progress to superintelligence and hope that it is aligned and will treat everyone fairly
3. Stop AI development altogether and create a world government panopticon which ensures it never progresses towards human obselesence?
Obviously there is the general gradient between full speed ahead <-> full stop, but what's more important is the communication and assurances between all relevant parties (US/Chinese governments, major AI labs). If everyone is forced to make a deal without being aware of the risks, then there is no point as they will imagine that these safeguards are overly stringent.
We are somewhat lucky that your option 1 will just happen by accident. And by this I mean the fall from grace of Jensen Huang will put a bit of brakes on the AI movement. It will be temporary, for sure, but it will give us some time to adjust.
One person's scandal won't put a break on AI progress itself, at worst it is viewed as an arbitrage opportunity to exploit short term chaos such that leadership restructuring puts different people in charge.
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
> These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
For math/cs research I found that agents alone doesn't replace a harness that gives them a strict sense of reality to have a better "compass". It's not only Lean but a plethora of tools like Mathematica, SageMath, SAT/SMV solvers, and very niche ones.
I am not in an alphabet lab, feel free to hire me though, and using off the shelf models with a few tweaks to the harness I used the same models to build I beat all the published benchmarks by substantial margins.
We need people at this point to tell the AI to do better and to stop disastrous side quests that kill its productivity.
And yes, rsi is here already. And its small enough that it can fit on a local gpu.
The tsunami is coming for everything, I now think that AI is the first democratising technology since the musket. And the results will probably be the same.
Why is there a limit from the chips? If AI is better than all the humans in every field, then it should also be able to design a better chip no problem. It should be able to figure out how to make new manufacturing processes etc to make chips smaller and faster and overall more capable.
"The avalanche that hit math is coming for X" is already a tired meme. It's telling that the impressive success of applying LLMs to the entirely abstract domain of math is being used to make extrapolations about all kinds of domains that aren't amenable to 10k agents (intelligently) throwing shit at the wall.
What I find fascinating about AI predictions is the human nature reflected behind them. There seem to be two camps: the first includes the author of this essay, people who are beginning to believe that unknown capabilities yet to exist will soon become possible and are impossible to predict. This camp seems to be winning an increasing number of people due to previous unknowns being solved very rapidly. Therefore, belief has been increasing in this group that AGI and other big AI questions such as massive human job losses can't be far around the corner.
On the other side of the spectrum we see people who still want evidence and kick their AI predictions out to a more distant future by pointing at current limitations. This group seems to move the goalposts as needed to support their argument with each new advancement. As the author of the article pointed out, he has moved from this camp to the first one. I have also moved to the first one.
the author bet everything on AI explosion and it paid off for him. Sometimes it is easier to be a believer and hope you end up right than endlessly discuss.
Future capabilities are impossible to predict, therefore the timeline must be accelerated to an attention grabbing year such as 2030. All the same camp
It's human nature and all to common to move goalposts when a belief is entrenched or ideologically motivated. It can be as simple as the belief is aligned with the interests of that person.
Anthropic would not be IPO’ing if the first case were coming anytime soon. They’d have no need. Their current behavior looks more like a company who wants to cash out while they can.
The behavior of these frontier labs is much more telling than anything else.
It's not only about verifiable or not. It's also about how high the fruit hangs vs how good your ladder (search heuristic) is. NP stuff is readily verifiable for example.
Absolutely plausible that this math avalanche was 1 cm deep, and harder problems are not solvable without new math theories that have to be constructed with a lot of understanding needed and taste (human heuristic).
So if AI is shortly (Dwarkesh suggests maybe next year) going to be better than any expert at anything, I guess I'll just wait until it's there and ask it to make me rich.
Seems far simpler than asking it to make super-human intelligence since nobody appears to know how to do that, and tons of people already know how to successfully start companies and make money.
My backup plan is to still be here in a years time, not rich, reading about the latest forecasts about how the singularity is coming real soon now.
I've come to the conclusion that the only thing LLMs are good for is writing code and ONLY things they will excel at are things that can be solved by writing code.
All that fuss about the Mathematics problems solved was just LLMs writing Lean code. And besides that, agents are not anything remote to human in terms of behavior and accountability. They are like silly toddlers with a lot of knowledge.
I think AI is not coming for all white collar jobs, it's only coming for jobs that can be solved with code.
One can argue that they will be able to do brain surgery by writing code that execute instructions in a robot arm scalpel. Only time will tell.
I’m not into math. Certainly not professional math. Judging by the headlines here I know that AI (or AI + shepherds) solved some math. Some weeks later I saw headlines that mathematicians were psychologically in shambles. Um, okay? Maybe this was a long-term development, that AI has gradually made mathematicians (and not just the AI-pilled ones) realize that it’s better than them? That’s the only thing that makes sense. Because it doesn’t make sense to have a breakthrough in one week and then a little later have a collective depressive episode. No human institution is upended just like that.
Nothwithstanding genuine one-shotting, mathematics is certainly a great propaganda angle for the AI Bros. Ivory tower image. Not obviously directly applicable to the real world. Theoriticians solving theoretical problems because the problems are there and tenure. That’s the perception, anyway; highly-educated whiteboard writers, not salt of the earth workers. Then you frame that crowd as jealously guarding their theorycrafting fiefdom. But it’s no match for the great AI. And then they go into group therapy next week.
rsoto2 | 4 hours ago
bananaflag | 4 hours ago
Trump's mistake is using the word to denote AI in general when it already had an established meaning.
spwa4 | 4 hours ago
Strange how this guy can be CEO of anything. All you hear management types ever blubber about is integrity ... Clearly there's a problem here.
quicklywilliam | 4 hours ago
platinumrad | 4 hours ago
aerhardt | 4 hours ago
12983701 | 4 hours ago
https://www.dwarkesh.com/p/advertise
qw34ahg | 4 hours ago
aerhardt | 4 hours ago
What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.
As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.
The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.
jeremyberemy | 4 hours ago
In my experience, s-curves are much more prevalent in nature than exponentials.
timschmidt | 3 hours ago
aerhardt | 3 hours ago
godshatter | 3 hours ago
I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.
I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)
mentalgear | 4 hours ago
The only question that remains is if 99.9% of the world population will let themselves be fooled by UBI & VR to keep the 'masses content; while the 0.1% hold all the wealth and AI power.
And this is decided now, because it is too late once AI is consolidated under one bi/tri-llionaire.
reducesuffering | 4 hours ago
Is the equivalent of ants trying to outsmart Google that wants to build a datacenter on their land. ASI will unfathomably outsmart the wealthy and intelligent humans like Stockfish 13 does Magnus Carlsen
Avicebron | 4 hours ago
reducesuffering | 3 hours ago
"there will be people who will try and use this to enrich themselves and immiserate others."
This is already happening from Big Tech and VC's, all the big money. You should try listening to the people that have shirked enormous fortunes in order to tell the truth about things. Some examples:
https://x.com/hilbertspaess/status/2097476196791709843?lang=...
https://fortune.com/2026/09/26/geoffrey-hinton-godfather-of-...
https://yoshuabengio.org/en/blog/faq-catastrophic-ai-risks
https://ai-2027.com/
rcoveson | 3 hours ago
Everybody who thinks they're "the chosen people" predicts the end of the world is coming, but not everybody who predicts the end of the world is coming thinks they're "the chosen people". Maybe the end of the world is coming and it's Just Bad.
dominotw | 4 hours ago
But we've all seen robots fold clothes to do that sort of thinking easily. Oh maybe we need to hangout on ai twitter more.
spwa4 | 4 hours ago
Why? It is now easy to get a few motors to do that stuff for you. It is now easy to get whatever manufactured ... except for raw materials, and energy if you see that as separate.
Think about that again: if superintelligence exists the huge change is that money is now worthless and raw materials have all just turned more valuable than their weight in pure gold (assuming they're more useful than gold, which is all of them). Steel, just raw ingots, will increase in price BY A FACTOR ONE HUNDRED THOUSAND, minimum. At this point your dog can probably reason towards the investment opportunity, no?
You would buy farms, mines, mining rights, water, energy (and not for DCs, actual energy sources), rivers, raw materials ... that sort of thing. A superintelligence is not going to be exempt from the means of production. Raw materials are mostly monopolized in the current economic system, so this would in fact be absolutely critical to your success. And hoards of raw materials, anything, will be worth more than Gold (gold being pretty useless except for looking pretty and having a somewhat favorable electrical resistance, but not super great. When it comes to strength it is beaten by 2-day old chewing gum. So if economic activity is 100x-ing every raw material actually used for something will go into extreme shortage)
So no, sorry. Obviously this cannot be how Amodei, Altman, Google, Meta, and others are seeing things: they're going after money, not something else. So they are either stupid or they know 99%+ sure they do not have superintelligence. Because if they do let superintelligence loose, mining countries will have them by the balls and will squeeze. Hard. Yet the money all these people are collecting boatloads of will be useless. And yet, they want the money, not the raw materials! That tells you what they believe: that the money is more valuable than raw materials ... AND more valuable than superintelligence.
In other words: AI is a bubble, a hype, according to the management of the big AI firms. It's a way to get people to give them money, not an actual path to superintelligence. Either that, or these people are complete idiots.
morkalork | 4 hours ago
https://www.forbes.com/sites/arielshapiro/2021/01/14/america...
ryeights | 4 hours ago
Your argument only makes sense in a world with one AI lab. Labs are pouring all of their money into scaling because there is no alternative.
spwa4 | 3 hours ago
So that's not what happen. Yes, someone will be first, but the second will not be far behind.
Once you have ASI, raw material owners and the governments they're in will PICK the winner from the choices available. Just like they did in the 20th century.
To say nothing of the consequences of what is currently happening with Chinese models only 2 months behind frontier labs. Even if it's 8 months or 2 years or whatever. ASI will not build an insurmountable advantage in the world economy in 2 months, especially not if some raw materials players sabotage it (which seems likely in the extreme, and is simply what raw materials players are already doing, so they don't even have to realize what's happening and start doing it)
ryeights | 3 hours ago
- Sophisticated cyberattacks to disable or quietly sabotage competing labs
- Undermining nuclear second-strike capability by locating submarines and infiltrating command-and-control networks
- Fast-replicating nanotechnology
- Superpersuasion (I am dubious on this)
etc., backed by global surveillance to preempt sabotage efforts. If you haven't read Superintelligence you should, all the leaders of the labs certainly have.
spwa4 | 2 hours ago
So this level superintelligence also falls under the statement "if management believed that, this isn't what you'd see them do".
Plus your statement doesn't explain why they're after money.
ryeights | an hour ago
Consider that they are after money because it allows them to scale and hopefully control AGI/ASI
spwa4 | 46 minutes ago
I'm with the people who really doubt alignment matters.
Besides, one thing keeps coming up in the open models: removing alignment makes them better, more interesting and more creative (difference for fiction, games, and ... well look at civitai is enormous).
And alignment also opens up an avenue to beat these models. And this will be a new level of moral dilemma: you can't judge, because the AI is too complex, too fast, too smart, ... and you cannot even react in time.
> ... they are after money because it allows them to scale and hopefully control AGI/ASI ...
Makes no sense for companies that are already making their own chips. If they truly believed they could figure it out FAST, they'd be buying sand (and it's not everywhere: apparently the purity is a big deal and not available in many places)
ryeights | 16 minutes ago
That's what you (and I mostly) believe. AI researchers do not believe this.
jstummbillig | 4 hours ago
It might all end very badly, but we should not be so prone to presume bad outcomes without explanations, that are just as good, as we would expect for good outcomes.
UncleOxidant | 3 hours ago
This time he wasn't able to. But the time before (2024) he seemed to have some effect. I'm sure that he and the other oligarchs are going to learn from this midterm election and try different approaches in the future. Think of the 2026 election as one of his crashed rockets. They learn from those crashes - iterative improvement. They'll figure out ways to be more sneaky next time.
leobg | an hour ago
If you had millions of followers and billions of dollars, and you were convinced that America will go down if, say, Trump wins an election, would you not use the influence you have?
One could argue that you’d be a coward if you didn’t - having the conviction, having the means, but not wanting to stick out your neck.
You might not be that different from “them”.
root_axis | 3 hours ago
lll-o-lll | 3 hours ago
kridsdale1 | 3 hours ago
noosphr | 3 hours ago
Humans amassing money in an AI world is like squirrels amassing nuts when humans are logging the forest.
If we want to have a planet left we need to make it so laughably inefficient to run AI ground-side that no one bothers.
Extremely high temperature semiconductor research to run datacenters in dyson swarms around stars/black holes is fundamental research for human survival at this point.
zaler | 4 hours ago
It seems likely we end up with an even more extreme tech oligarchy than we already have when superintelligence arrives. And that's only if we manage to solve alignment and it doesn't do something more horrible.
orphereus | 4 hours ago
Ridius | 4 hours ago
itronitron | 4 hours ago
blooalien | 4 hours ago
Isn't it already here? Infallible God-King Trump and all his ultra-genius tech-bro buddies delivered superintellgence already, and due to exponential growth it should already exist by this time, no? By this time next week it will have solved all problems and made humans obsolete.
dominotw | 4 hours ago
Lots of companies think they are reaping similar rewards from ai coding and research but it has failed to lift bottomlines or create any new products. My last two employers are gungho on ai and ppl have been producing unfathomable amount of ai output. To a causal observer it would look like we have really accelerated but this has had no visible impact on any of the products that employers produce other than make them shittier.
Whats different about product development at athropic that ai is particular suited for not generating slop.
j2kun | 4 hours ago
drcode | 4 hours ago
platinumrad | 4 hours ago
timschmidt | 4 hours ago
viklove | 3 hours ago
timschmidt | 3 hours ago
Seemingly from you, as a strawman. He doesn't seem to make such a claim in the article.
viklove | 3 hours ago
So he's not an expert. Just a guy off the street. Got it, I'll pass on the podcast then thanks.
timschmidt | 3 hours ago
“Great minds discuss ideas; average minds discuss events; small minds discuss people.” -- Eleanor Roosevelt
viklove | 3 hours ago
timschmidt | 3 hours ago
avanwyk | 4 hours ago
dumberquestions | 4 hours ago
wnmurphy | 3 hours ago
If someone has interesting ideas, or a reasoned argument, do you require their credentials before considering what they've written?
thin_carapace | an hour ago
dyauspitr | 4 hours ago
wilkystyle | 4 hours ago
dyauspitr | 4 hours ago
https://www.politico.com/news/2026/09/29/trump-signs-executi...
baq | 4 hours ago
wilkystyle | 3 hours ago
dyauspitr | 3 hours ago
avanwyk | 4 hours ago
hackermailman | 4 hours ago
dumberquestions | 4 hours ago
dyauspitr | 4 hours ago
reverius42 | an hour ago
wnmurphy | 3 hours ago
zarzavat | 4 hours ago
It seems that the only benchmarks that could be automatically constructed are those for tasks that are themselves verifiable (theorem proving, coding, etc).
Non-verifiable tasks require humans to define what exactly a good job is.
The RSI hypothesis is dependent on an assumption that progress on verifiable tasks generalizes to non-verifiable tasks.
atleastoptimal | 4 hours ago
1. Performance engineering ability 2. Ability to reconstruct visuals in code alone 3. Ability to orchestrate cheaper models towards a given task 4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
OtherShrezzing | 4 hours ago
It’s a small distinction, but I think it creates a meaningfully less challenging requirement.
jazzyjackson | 4 hours ago
iaanix | 4 hours ago
aabajian | 4 hours ago
throwaway219450 | 4 hours ago
But models are really good at being told “here is a paper, check the implementation against the claims in the paper and munge my data to work with it”. On the plus side, many seminal papers have really simple implementations. Like the meat of many CVPR papers is a derivative repo + one function modified. Often they can fix things or backport improvements from more recent work. And they can definitely re-implement things that ought to work, but the authors never published.
afgfaa | 4 hours ago
https://sponsorradar.com/channels/dwarkeshpatel
And that does not include Substack bulk subscriptions.
FLeXMurphy | 4 hours ago
throw-away-mle | 4 hours ago
These agents know every paper, every trick, have amazing instincts, and can code up 20 ablations and start them running before you finish your commute. I'm finding guiding them is becoming less and less useful. I have never felt my obsolescence rushing towards me as much as now.
atleastoptimal | 3 hours ago
1. Pause AI development and allow humans to stay in control a bit longer while we work on making AI safer, allowing the field to advance bit by bit (and hoping this plan even works)
2. Allow AI to progress to superintelligence and hope that it is aligned and will treat everyone fairly
3. Stop AI development altogether and create a world government panopticon which ensures it never progresses towards human obselesence?
asa123 | 3 hours ago
atleastoptimal | 3 hours ago
reducesuffering | 3 hours ago
credit_guy | 3 hours ago
atleastoptimal | 2 hours ago
oconnor663 | 3 hours ago
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
ant6n | 3 hours ago
wslh | 3 hours ago
For math/cs research I found that agents alone doesn't replace a harness that gives them a strict sense of reality to have a better "compass". It's not only Lean but a plethora of tools like Mathematica, SageMath, SAT/SMV solvers, and very niche ones.
noosphr | 3 hours ago
We need people at this point to tell the AI to do better and to stop disastrous side quests that kill its productivity.
And yes, rsi is here already. And its small enough that it can fit on a local gpu.
The tsunami is coming for everything, I now think that AI is the first democratising technology since the musket. And the results will probably be the same.
lousken | 3 hours ago
SirMaster | 3 hours ago
root_axis | 3 hours ago
lavendermon | 3 hours ago
On the other side of the spectrum we see people who still want evidence and kick their AI predictions out to a more distant future by pointing at current limitations. This group seems to move the goalposts as needed to support their argument with each new advancement. As the author of the article pointed out, he has moved from this camp to the first one. I have also moved to the first one.
newyankee | 3 hours ago
Dzugaru | 3 hours ago
lavendermon | 3 hours ago
thewanderer1983 | 3 hours ago
String theory is one that comes to mind.
popkkksidhd | 3 hours ago
The behavior of these frontier labs is much more telling than anything else.
Dzugaru | 3 hours ago
Absolutely plausible that this math avalanche was 1 cm deep, and harder problems are not solvable without new math theories that have to be constructed with a lot of understanding needed and taste (human heuristic).
HarHarVeryFunny | 3 hours ago
Seems far simpler than asking it to make super-human intelligence since nobody appears to know how to do that, and tons of people already know how to successfully start companies and make money.
My backup plan is to still be here in a years time, not rich, reading about the latest forecasts about how the singularity is coming real soon now.
jcmontx | 3 hours ago
All that fuss about the Mathematics problems solved was just LLMs writing Lean code. And besides that, agents are not anything remote to human in terms of behavior and accountability. They are like silly toddlers with a lot of knowledge.
I think AI is not coming for all white collar jobs, it's only coming for jobs that can be solved with code.
One can argue that they will be able to do brain surgery by writing code that execute instructions in a robot arm scalpel. Only time will tell.
Sorry for the rambling, just puking my thoughts.
keybored | 2 hours ago
Nothwithstanding genuine one-shotting, mathematics is certainly a great propaganda angle for the AI Bros. Ivory tower image. Not obviously directly applicable to the real world. Theoriticians solving theoretical problems because the problems are there and tenure. That’s the perception, anyway; highly-educated whiteboard writers, not salt of the earth workers. Then you frame that crowd as jealously guarding their theorycrafting fiefdom. But it’s no match for the great AI. And then they go into group therapy next week.