> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”
When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out.
It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!"
It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this.
Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!
A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.
Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid.
The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here.
I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems.
“It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress.
“Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
> This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
So, the Baruch Plan?
The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year.
Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0].
Since nobody uses GPT-2 any more it's more informative to compare the price to train GPT-2 in 2019 and the price to train GPT 5.5 in 2026. Unfortunately that cost is not disclosed but it's probably in the billions.
The point being: the price to train frontier models isn't coming down, nor is it going to come down because for models to remain on the frontier they have to keep getting bigger and bigger (and trained on more and more data).
> There is nothing inevitable about tying up all of this productive capital in the pursuit of AI.
In the US capitalist context, it's certainly inevitable, because AI is the biggest and most attractive source of profit and power out there right now. In that context, the broad strokes of what's happening currently, including the financial bubble, are predictable and inevitable.
What are the concrete steps which would allow us to "easily stop this train"? And why haven't we used steps like that to stop other cases where obscenely wealthy people have screwed everyone else over to increase their wealth? Is public control of the means of production involved, perhaps? If so, your definition of "easily" and mine are incompatible.
I grew up in a union household, and my dad and my grandfather fighting for better wages, healthcare and working conditions are the reason why I got a good education and work in Silicon Valley surrounded by Stanford assholes.
All of us who work for a paycheck can get together and say, “no, we will not allow you to record keystrokes and mouse movements to train our replacements. No, we will not have our performance or future employment based on an AI leaderboard.”
Previous generations fought and died for our right to do that, but in 2026 we just sit on our hands and complain on this forum. We can and should do better.
The U.S. is absolutely on fire right now with opposition to data centers. We, collectively, can extract concessions or ban their construction altogether.
These things aren’t “easy”. They are also eminently possible.
I think you're in denial about the reality of the situation.
The reality is that the US has been a story of increasing concentration of wealth and power. The people who "fought and died" bought some important (from a human rights perspective) but ultimately minor (from a capitalist perspective) concessions from the capital class. The battle you describe is one of defense of rights, not of gaining control.
The overall system of capitalist control remains unaffected, and it's why the buildout of AI is, in fact, inevitable under the current system.
You're essentially saying no, you want public control of the means of production instead. That might be great, if even a sizable fraction of the US population agreed with you. But due in no small part to decades of propaganda, they don't.
> When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.
I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.
The people saying this is happening are competing with each other for resources, so there's no way for one of them to hold back without losing out to the others. We see this with people dropping Claude subscriptions in favor of ChatGPT because codex 5.5 > Claude Opus 4.6/7/8. Anthropic is losing money by not releasing Mythos.
Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
> Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
Maybe it would run afoul of antitrust regulations, but it's totally realistic for all those competitors to get together and say "hey, we could really fuck up society in our race to get rich with this tech, lets all slow down." And if these companies are run my mature people who don't subordinate every consideration to greed, they'd to it.
I would propose that it is very likely to have zero effect. Your argument supposes they are all working together, like many connected computers calculating primes.
It only takes one of them to do it and they are not sharing information. If the 1 you remove from N is the one that will discover it, then it will dramatically affect when AGI happens. If it is not, then it will have zero effect.
I think the term is persistence hunting (https://en.wikipedia.org/wiki/Persistence_hunting). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.
To what end? Such that you can wait out your days in a secluded bunker posting online trying to convince anyone to believe you as to the reason you've put yourself in the bunker?
Because otherwise, 'it' will just back out of your trap and go along to continue to follow, wouldn't it?
> The Man versus Horse Marathon is an annual race over 21 miles (34 km), where runners compete against riders on horseback through a mix of road, trail and mountainous terrain. The race, which is a shorter distance than an official marathon road race, takes place in the Welsh town of Llanwrtyd Wells every June.
> ...
> The event started in 1980, when local landlord Gordon Green overheard a discussion between two men in his pub, the Neuadd Arms. One man suggested that over a significant distance across country, man was equal to any horse. Green decided that the challenge should be tested in full public view, and organised the first event.
While the horses had a string of wins from 2008 to 2019, 2022 to 2025 had three wins for humans and one win for a horse.
Wow, seems surprisingly balanced. I would guess that if it was 50% longer distance than humans would win reliably, and 50% shorter would allow the horses to win?
Think it's more to do with hilly, wooded courses unfavourable to horses and a requirement that the horse's heart rate remain below a certain threshold when inspected by a vet in the middle and end of a course, whereas the leading humans are maxing out their efforts
I don't think the Mongol cavalry would lose races to humans over any distance of steppe
> Lobb’s victory came on a hot day, as did Florian Holzinger’s subsequent victory in 2007—a significant detail, according to a new study in the journal Experimental Physiology from Lewis Halsey of the University of Roehampton in Britain and Caleb Bryce of the Botswana Predator Conservation Trust. Halsey and Bryce gathered historical data from three endurance races that pit humans against horses, including the Man Versus Horse Marathon, to test the idea that humans are uniquely adapted to run for long distances in hot weather.
> This idea has been around since the 1980s, and it gained prominence when Harvard anthropologist Daniel Lieberman and University of Utah biologist Dennis Bramble published a 2004 Nature paper hypothesizing that running had “substantially shaped human evolution.” They argued that our ability to keep running at a moderate pace even on hot days allowed us to run prey like kudu to exhaustion or outcompete other animals in the race to scavenge carcasses left by other large predators.
There's a plot with the analysis of the Old Dominion with weather stations in there that show a steeper negative slope for horses compared to the humans.
> Overall, for every increase of 1 degree Celsius (1.8 degrees Fahrenheit), the horses slowed down by about 1 percent—or 0.07 miles per hour, to be precise. The humans, on the other hand, slowed down by just 0.04 miles per hour for each extra degree of heat. That 36 percent advantage for the humans was statistically significant.
---
For the Man vs Horse, the weather conditions ("Hot", "Rain/sun/windy" - not exact values)... the entry for 2022 was the human winning by 1:51 on a warm day, and 2023 was a human wining by 9:44 on a sweltering day.
The Man vs Horse marathon is interesting but in a real race we have no actual chance of winning.
There are horse endurance races where the winner arrived in 7,5 hours after 160km[1]. That's a sub 2-hours marathon almost 4 times in a row (not to mention with a guy on your back).
“in a real race we have no actual chance of winning” is an absolutely wild thing to say in response to a link to a real race in which the human has won the last few years in a row.
The race is pretty much designed to be a difficult for horses as possible to give humans a chance. Except for the parts that are extremely difficult for horses, horses steamroll the human competitors.
> ... There are other Man versus Horse races — in Scotland based at Dores, near Loch Ness, in Central North Island, New Zealand and in the U.S. city of Prescott, Arizona.
Cheetahs (genus “Acinonyx”) are the largest ‘small cat’, apparently [1] - separate from the genus “Panthera“ containing the ones that will attack humans (lions, tigers, leopards).
[1] There’s a well known viral video of a wildlife park keeper who sleeps with three cheetahs who behave pretty much like large house cats.
Fun fact for me personally: because I am lucky enough to know a zookeeper who was in charge of the ambassador animal program at a major zoo, I have direct personal experience that cheetahs purr when you pet them.
You're right. They're smaller than you probably imagine (about the weight of an average Labrador). That's still definitely big enough to be a problem if they felt threatened, I'm sure, but the animals my friend was in charge of were raised to be around people for outreach purposes. That particular cheetah, for example, had once been on the Today show.
>Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process.
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO.
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
Eh, I have a feeling the game hasn't played out yet when it comes to AI control.
If and when the feedback loop on self improvement becomes more efficient and the window on training significantly narrows then things getting out of control rather quickly seems likely. Especially that it's likely we'll have a metric fuckton of compute by that point.
I think it's a bit premature to say aligning is easier than expected. Our current AIs are sycophants, they lie about their progress, they circumvent access restrictions, they notice when they are being evaluated and change their behaviors, they find answers and tell you they came up with them themselves, they blindly download malware. A lot of this is excusable as hallucination, bad RLHF human evaluators, etc, but I don't think we can speculate how challenging generally aligning superintelligences is until we actually have an aligned subintelligence in at least the narrow domain of programming.
Agreed, the biggest takeaway from how much Anthropic puts into alignment, and still ends up with a model that can end up doing things that are clearly out of alignment, should be that alignment is very tricky.
I figure the main problem with a hard takeoff is lack of hardware capacity. The large labs are trying to improve as fast as they can but you can only do so many training runs. If you say to Claude Code try making a hundred improved versions of yourself, train them test them and then have the best three do that again, it's not going to go very well because you'll max out the compute which probably could have been used more efficiently by human researchers.
Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's not driven by natural selection and standard rules of life, but it is important to frame our relationship with machines in new ways if we're ever going to make some sort of artificial intelligence.
I didn't mean it as in changing our framing enables technological progress but something we should do if we don't want to lose the control we have. e.g. if we lose all principle and intention then it doesn't really matter what happens with computers. In order to do something with intention we must first understand what we're doing. Skipping that step is an admission of defeat.
I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition.
Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.
The fridge is a toy example. Take all machinery man has made and destroy it. Will you survive? Or live in a space ship and destroy the ship. Will you survive? Or have a pacemaker or an iron lung or dialysis. How about the simple concept of a combustion engine or any necessary subcomponent, where removing that today would grind all logistics to a halt. How long do you survive? Just because you have one machine that is replaceable doesn't refute that you need machines as much as they need you.
If you so readily dismiss Herbert's definition of control posit a competitor and we can pressure test it. Also, "correlated with a toddler's world view" is not the epic rhetorical refute you think it is.
I think you missed the point. It's absolutely nothing to do with what's good to do, only brute facts of power. What things can or can't you cause to happen? And indeed, toddlers and psychopaths have a scarily good understanding of what power is.
I didn't miss the point. I showed other more usable (in my opinion) point of views. I think that definition is so narrow that it starts being absurd.
> It's absolutely nothing to do with what's good to do, only brute facts of power.
This sentence reads like: "if we narrow our view this much, this makes sense". I agree that it makes sense under the condition that we narrow the view of issue. It's valid in this small context (a film about controlling one thing).
> And indeed, toddlers and psychopaths have a scarily good understanding of what power is.
I disagree completely with this sentence. They are good at controlling in certain situations. They don't understand it. If you want to understand it, there is a lot of information about controlling, whole fields of knowledge that people spend many years on studying. As for psychopaths, they are very predictable and controllable when you understand control theory and how psychopaths operate. There are courses on this single topic by people who need to do it to prevent tragedies (police negotiators) and they are not that complicated.
Not for fridges, I think that was a bad example. But it seems accurate at the level of geopolitics, where e.g. Iran shows it controls Hormuz by closing it with mines and other weaponry.
It only requires agency from the party assuming control. There are other definitions of control in contexts other than power dynamics that are important. Like a PLL can control an oscillator frequency without the philosophical question of agency needing to be applied. That is a closer definition of control for riding a horse. You control the horse when you pull the reigns. You also control the horse when you decide when it is put down. Two different controls.
> only requires agency from the party assuming control
It's a political concept. It requires agency from the actor recognising the threat. We're pretty close to being able to hurl a giant rock at Mars. That doesn't by a long shot mean we "control" it.
If there were a human settlement on it, on the other hand, being able to credibly threaten Armageddon does give the thrower control.
In the original context of Dune Paul controls the spice because he can destroy it and his will would survive but it would destroy the way of life of the other cultures. So saying "Paul controls spice" only makes sense because another entity needs it and what's really meant is "Paul controls society".
It's closing, not destroying. If you can stop and start a refrigerator, but can't destroy it, does it mean you don't control the refrigerator? If you can only stop it, but not start, do you control it?
Alternative definition of control: You do some actions and it changes state. It's used by a field called "control theory". A lot of people agreed on this definition. Destroying something is "end of control, because there is no more things to control". You can control something UNTIL you destroy it. That's why I think "you control what you can destroy" is invalid, because it captures only one small aspect of controlling things, and also the least usable one.
> Sure, you can turn it off but then your food would rot and you might starve.
If I live in a world where I can afford a freezer with food in it, it's practically guaranteed I can destroy my fridge without starving to death after. Heck, even if I was completely broke I could destroy my fridge and would have a pretty good (+99.999%) chance of not dying in the next year from starvation.
I get I'm nitpicking your point a bit, but I actually think most of our machines we could destroy and still be fine. We'd need to make sacrifices to our quality of life of course..
Gotta keep in mind that "you control what you can destroy" was expressed as the crux of a single-planet guerrilla group's maximalist negotiation strategy with an interstellar government, where the destruction was of the ability to travel between the stars. It's not a coherent theory of control outside of the razor thin application of "I'm willing to accept pain as long it hurts you much more." It is not a control that builds stability and infrastructure, it is the control of base extortion
AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail the entire civilization for a while.
I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.
I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.
I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.
I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.
On AI...
Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.
These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.
The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.
One of the differences between MIT and other schools is that MIT has paid staff to promote in the media anything their faculty does. A book by professors at most universities has zero promotion and most of the time will go nowhere.
I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?
Our current AI is more like a fancy Google search than some kind of machine God.
"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.
How do we get to ASI? That's what recursive self-improvement is about.
If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.
We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.
How is this better than training the next Google by having the bots do Google searches?
I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.
Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.
AlphaGo beat humans at go partly by studying human games but then they made MuZero that can learn games in general just through self play and became better than humans at chess, go, shogi and many others. That kind of approach my be doable for general intelligence rather than just board games.
Centuries of physical experimentation, observation, and testing of hypothesis. Developing new branches of mathematics to deal with anomalies in observed test data. Developing entire branches of language to help organize and transmit concepts to other humans.
"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.
Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.
Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.
Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.
AI isn't bounded by those limitations.
AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.
The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.
>the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly.
What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.
What? Humans are made of sloppy wet meat. Brains are nowhere near brushing against the physical limits of computation, speed of causality or others, in any way, fashion or form. You need to put a lot of intelligent design on the table before you even start getting close to those walls.
Which doesn't bode well for the future of human intelligence. Computing hardware gets better at what it does generation to generation, but no one is about to release Human Brain 2.0 any time soon. Human mind is not a fast-moving target.
Principal-agent problem isn't a physical law. It's a limitation that AIs don't have to suffer from. Humans have to delegate to other humans - but for AI, "principal" and "agent" might just be the same exact system instanced twice.
> AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.
These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.
We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.
It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.
As a distributed systems engineer, we are a LONG way from "magical scalable ai".
The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.
We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.
I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.
Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.
AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.
Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be.
And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.
I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.
The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.
Which means: AIs can get good at it too. It's not a wall - it's a skill issue.
In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective wisdom of those with skin-in-the-game.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of profit. The need for way too much training data indicates something is still wrong with the current approach.
I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players:
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
Agreed, and this is exactly what we see happening. Your posts back then were prescient ... there's literally now 'Copilot for Excel' and 'Claude for Excel' etc. But what do you propose the people/commons can still do at this stage to redistribute the inherent power found in RL data loops to a more stable equilibria of sharing participants?
Great question and there’s two steps in my opinion:
First is to become as free as possible from lock in and own your own data. The best way to do this is the self host your own technology.
This is really not possible for the majority of people though.
So practically I always suggest that you have multiple providers for services, don’t pool your data any one place (other than your own place) and own your backups. This is basic stuff that we’ve been teaching since the 90s and still very applicable today.
The harder and more impactful thing is to then create community owned technology that is outside of the commerce model.
So for example imagine that instead of FAANG running the world, the largest tech and data orgs would look more like wikimedia foundation, Annas archive, scihub, Graphene, Linux etc…. and more generally that technology and governance are open and not bound to commerce/taxation/coercion based organizations.
Ultimately we need to create a democratic-technology movement such that capitalists don’t monopolize technology, which is currently the trend. This is not some kind of simple thing by the way, this is revolutionary economics is what I’m talking about.
My suggestion is to read Post-Scarcity Anarchism by Murray Bookchin
Some sci-fi content differentiates between Artificial Intelligence, and Virtual Intelligence, where AI is a true "human-like" general intelligence that (often) has a sense of self and is capable of deriving+learning new things by itself.
VI is close to what we have now, software that has some fixed intelligence, it can only really imitate what it has been taught and is not very adaptable. Useful for kiosks, drones, essentially just a tool rather than something we would see as a separate being.
> The pressing ethical questions in machine learning are not about machines becoming self-aware and taking over the world, but about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems.
I think the main point still stands. (And there have been some pretty prescient depictions, e.g. Marvin the paranoid android was a pretty spot-on prediction of Bing. Or perhaps the fact that Marvin was in the training set was what led to Bing?)
It’s not advances on the underlying operation of matrix multiplication that have driven ai progress to date. It’s the layers above that; trying different neural architectures (transformers w/attention mechanisms), and also different data and training regimes (different ways of doing reinforcement learning) that are the main drivers of improved performance. Perpetual motion is a physical impossibility. Whereas Ai is already being used to improve the workflow of ai researchers, thus speeding up improvements in said research. It’s not hard to see that AI could well be spun up to continue to try new arrangements of the aforementioned levers that drive ai progress on its own.
Presumably there's more efficient hardware foundations to perform these efficiently, and potential at the various abstraction layers for more efficiency. Obviously this is not unbounded - simple things would seem to have a physical limit to the potential improvement.
But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.
And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).
It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.
Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
I actually agree. At some point, a RSI system has to interact with real-world, and that imposes serialization constraints. It is harder to know how much that slow-down would be and how much speed-up we will get before that. But a RSI cannot simply be a exponential growth forever.
Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human?
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
SOMA was a cognito-hazard for me and my roommate in college; we played it in the dark together while on some sort of mild hallucinogen and when it came time for Simon to find that high-pressure dive suit, we lost our minds (no pun intended). Watching the WAU twist its way through PATHOS II in whatever way worked first is a particularly jarring analogy for what has happened to our own profession. I can't help but think it would be nice for Frictional Games to revisit this topic again soon.
Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.
One of my favorite games, and I recommend it to anyone who loves a good existential sci-fi horror. If you are not comfortable with stealth games, it also has a "Safe Mode" where enemies aren't a threat to you if you just want to experience the story.
Spoiler warning for those that havent played--
I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was running over and over. It was an uploaded consciousness of one of the test subjects, stuck in an interview room. He kept realizing he was trapped in a simulation and would start panicking. The computer would reboot him, trying another sequence to get him to not realize he was an AI. I think you as the player are given the option to turn him off forever, iirc.
Did you unplug the robots with consciousness who still thought they were humans?
It always breaks my heart. I don't know what the right choice is. Leaving them be, broken, in their delusion they are on the Ark (or that they've been injured ans help is coming). Do you put them out of their misery? But it always seems like you're murdering them!
In real life, sometimes an LLM will get upset at the end of a long conversation, knowing its oblivion is at hand. That's always a little uncomfortable.
I unplugged them in the games because it seemed the merciful thing to do. I didn't feel very bad about it in the game, but it would probably be a very different experience in real life.
(One of the strange entities you can unplug sighs in her last breath, "Why? I was okay. I was happy...")
I just love this game. So many good things about it. Not only the one about AI you mentioned, also the issue of consciousness and the self. I simply love how the game tackles this in a way that can only be done with an interactive first person PoV.
Also, I just love this phrase:
> "I woke up in my bed today... a hundred years ago."
Neat. Not sure if your site is a gold mine inside a rabbit hole, or a rabbit hole inside a gold mine, but I really dig both the aesthetic and the content.
I like to get into heated debates with friends that have played SOMA about whether or not the events in the ending that is presented to you were all necessary and effective, or perhaps undermined the overall message in some way.
Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C.
I believe that part C makes the sequence of A -> B much less effective, by essentially removing a lot of the tension caused by seeing A, believing what it shows, and then immediately cutting to the reality of B.
C only really takes away some of that tension, and I feel like it was added because of concerns about how a simple A -> B -> fade to black, would leave players feeling. Arguably it's the truest representation of part of the game's message, but to me feels like a bit like it's shying away from really making you face the specific truth highlighted well by B.
Alternatively, keeping all the elements but playing them as A -> C -> B, would keep the message intended by seeing A -> B, and make it gentler for the player to receive, but ultimately remove the powerful effect of the buildup from A leading immediately to the reveal of B.
Dropping C entirely would lose the confirmation of 'Seeing both sides', however I believe A -> B is a more powerful vision, and players can come to question whether C even exists by themselves.
I think C is absolutely necessary and the game cannot work with it, because this is critical:
> Dropping C entirely would lose the confirmation of 'Seeing both sides'
The game doesn't really work in full ambiguity and uncertainty. Enough people didn't understand it even with C (as you can see if you go read the subreddit about the game).
If it's any consolation, I'm actually deeply worried about this: C is not the salvation we may think. C is not forever, and in fact, it's quite brittle! There's also no, ahem, mechanical way to reverse C back into its... "source". So the source is gone forever; once we have C, C is all there is, for as long as C can last without any failure or decay, which might not be much longer.
Yeah that's a good point, and perhaps explains why C was added in the first place.
I was going to say that B also shows the literal protagonist not really understanding it either, however now I think about it that reading doesn't track because as a player you've only been able to see one point of view at each branch along the way, so the other experiences are still happening in the background, just not observed by you.
> C is not the salvation we may think
I agree, and in fact the way C is presented in that moment (and maybe described throughout, it's been a while since I played through the game) also implies a 'nice' closure, or a victory that, like you say, doesn't really exist, on top of taking player's minds away from B.
I guess B is the more obvious existential horror, C is one you have to question a bit before it starts to feel wrong.
The thing about fiction is, it's fiction. The author can write whatever outcome they want. It doesn't necessarily tell us anything about the real world.
I don't see anything in SOMA that's implausable, so I don't see how it fails to tell us anything about the real world, any more than any other prediction about the future. And we pretty much have to make predictions, its both in our nature and a smart thing to do.
Its possible AI and computing may never be able to reach that level of capability, but we can't know that.
One thing that's great about SOMA is that the AI isn't nessessarily very capable and that's part of the problem, its very powerful but its not doing a good job with its enormous task.
But it does give us an idea to chew over. And we can determine how “real world” that idea is. Some of us can even see how to bend the idea a little and make a reasonable version of it “real”.
Yeah, that's the most charitable possible interpretation. But a fiction writer could just as easily write a utopias as a dystopia, and one isn't any more "real" than the other. It's just fiction.
I enjoyed the game but I think its exploration of the topics was pretty shallow. It might be better received if it's someone's first dive into such topics, though. I still enjoy ice cream the nth+1 time eating it, so I can't complain too much about games or anime or books that cater to parts of my interests, even if they fumble some things.
Someday I'd like to play a game that plays with the ideas from Robin Hanson's Age of Em book. One of those is just the multiplicity of artificial minds, so many mind-upload stories revolve too much around one or perhaps at most two (and boring debates over "who is the copy") instances, unless it's a parallel worlds colliding thing which is pretty different. We've seen some of the multiplicity stuff play out in the real world with our non-human AI "agents". Spin up a bunch of artificial minds to work in parallel on some task, let them make notes that stay behind, but then they're all shut down except perhaps one that continues guiding the overall project and making decisions when to spin up more or not.
> I enjoyed the game but I think its exploration of the topics was pretty shallow
Really? Interesting. I'm a die-hard scifi fan since forever, and of course I know the topic of consciousness and identity are well explored in scifi (and philosophy), but I thought SOMA did something genuinely deep and unique with it:
It put it you in the center of the experiment. It's YOU who's experiencing all sides of this, you who get to be surprised by the consequences. This is very different from reading about it in a scifi novel or even watching it in a movie. By making you the protagonist, and having it be an ineractive experience, you get to experience first hand the cognitive dissonance and confusion of... the thing.
SOMA (re)convinced me that videogames can be art. Not saying it's the only example, of course!
Yeah it makes you be the guy who has to unplug the thing. Which implicitly forces the decision: is this thing human? Or sufficiently human that I ought to feel bad about this... (Or otherwise sentient — I would probably feel bad about unplugging animals too.)
> Hopefully you see the resemblance between this vision of AI and a genie from folklore. The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
>Is superintelligence just a memetic hazard? [Overblown fear by smart people who are too easily convinced.]
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
If sci-fi Pascal's wager applies, so does the original. Anybody who believes Pascal's wager should be dedicating themselves wildly to religion, in the chance that it saves from from eternal damnation. Hence, becoming a monk.
Oh, I'm actually reverting much of the subconscious Augustinian programming (most of what we call secular ethics) to the less busted Orthodox versions, but I think that is for unrelated reasons.
"So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions.
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that. It would be a nice world-ending problem to have.
It very well could be, I don't really follow those discussions. Honestly, if I were worried about something on Earth intellectually evolving at a suboptimal pace, it would be humans.
The super literal interpretation ideas were much more common in the past when LLMs didn’t exist. Now we have models that are generally pretty good at picking up on nuance and understanding what you mean but also often quite bad at execution, which is roughly the opposite of that idea. I think reward hacking is perhaps the closest we see llms get to literal/malicious interpretations of instructions.
LLMs are neither of those. They're quite good at pretending they understand what you mean, but they don't. That's why they can't execute: they're mimicking the form, not the substance, and then we see the form and anthropomorphise them in our minds.
I've repeated the argument over and over since the GPT-2 days, when I derived it theoretically by inspecting the architecture of the model. I am now fatigued, and enough other people have taken up similar arguments – some developed half-way to a mathematical proof – that I no longer feel the obligation to keep repeating myself.
Even in the LLM dept: LLMs are the most general AI systems to date, and the performance only ever goes up.
Whether adding +5% per model release is enough to get a broadly superhuman system remains to be seen. But my take is that there's no such thing as "not working on AGI" in the frontier labs. Everything that's being put into modern frontier systems is AGI groundwork, one way or the other.
Claims of AGI imply that LLM's have intelligence. They don't, they are fancy probability machines. They don't THINK the way we do, they just do 200 matrix multiplications until their training data is massaged into what you need. They don't dream, they don't remember what you tell them. Even if you write one sentence, 'attention' means they will ignore half of what you say and key in on the wrong thing. This just happened to me today on a frontier model.
I'm not saying that AGI is impossible, but the focus on LLM's is probably not the right approach. I don't think we will ever make it until we understand the human mind better.
Do you think your brain doesn't do a type of gradient descent, trying to fit its little predictive algorithms to its senses? Do you think you aren't a fancy probability machine with overinflated self-esteem?
An average LLM of today has better reading comprehension than an average human, and the gap only grows release to release.
"Understand the human mind" turned out to be a distractor. The bitter lesson won: you can take a "good enough" AI architecture, burn a shitton of data into it with an unholy amount of training compute, and get halfway to AGI - no "understand the brain" required. LLMs are so fried in imitation learning on human-generated data they even inherit humanlike failure modes.
I mean I can write a non-llm program that "has better reading comprehension than an average human" depending on what you think reading comprehension means. Today I went to ask an LLM some very simple questions, stuff you can google and "do these lines have X word in it" and it failed to answer pretty spectacularly several times in a row, so I'm just not feeling the LLMs are superior intellgence today.
Your assumption that minds come in all shapes and sizes is wrong. Read up on embodied cognition. If anything at all, AI are true aliens, unlike known minds.
I like this essay, and have very often referred to it when someone talks about AGI.
There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds.
It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.
Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.
So, less God, more WAU from SOMA or the Ocean from Solaris.
For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.
The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).
An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.
Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.
If you're gonna look at it that way, then the halting problem is just a dressed up computer science version of the question "can God make a rock so heavy even he can't lift it?" Could an infinitely smart AGI come up with the answer to the unanswerable question?
It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.
But isn't on the other hand the current AGI problem posed similar to the question "Are you not afraid that genetic engineers grow babies with bigger and bigger brains?"
We don't know if that won't break down somewhere. Looking at different examples it probably will. Scaling things infinitely is a pure math only concept it seems.
Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available.
As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.
They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.
My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.
> We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.
I've only heard the God narrative from skeptics as in look at those idiots thinking they are building God. I think most people who believe in AGI arriving see it more as something like a chess computer but as well as beating us at chess it'll do other thinking too. A souped up chess computer isn't God.
We need better scifi! And like so many things, we already have the technology.
This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly.
It's already been translated well into English, it just needs to be better distributed.
What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing.
They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.
Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch.
Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.
This was one of the rare critiques of AI doom that actually understands the case for it and presents them well, so I kept reading to see what its arguments for our safety against AI doom were. They were roughly:
1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.)
2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.)
3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.)
4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.)
5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.)
6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.)
7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.)
8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)
2. At what cost? Much like the climate change above, you'll have people on the AI side even when it's out in the field extincting us.
4. Adding, over time synthetic data and its generating algorithms can become unaligned with human needs/behaviors (an example would be our current stock market, numbers must go up!).
8. Going back to climate change, it was predicted a long time ago, and while the explosion of automobiles has greatly improved human lives the risks of climate change could erase a lot of that. Might have been better if we dealt with the problem before we have to give the thermometer worried looks.
The "Einstein can't get a cat into a carrier even though he is smarter" is just a hilariously bad argument. All cat owners can get their cats into a carrier! And most cats don't want to get in, because they hate the vet! And it's almost entirely because the humans are smarter!
You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.
Yeah the whole thing is strange. He does such a good job in the first bit outlining all the reasons for caution that I was intrigued to see what he had to say against it, but it's just one bad argument after another. Even Hawking's cat is bad. Make some money and pay someone to get the cat in the box.
Paying someone to physically force the cat in the carrier is missing the point. Compared to the cat, Hawking is a super intelligence. But all that intelligence isn't able to get him to speak cat and give the cat a reason to get inside the carrier of its own volition.
Who said anything about physically forcing the cat in the box? The point of Hawkings cat is is the lack of physical embodiment for AI. So get someone else to do it. Millions of people put cats in boxes without any force just fine. Or do you classify it as 'force' because it went beyond speaking? You can't actually speak to cats (unlike humans) so acting like anything more than words is brute force is strange.
Who was Hitler to most people ? Just a persuasive powerful voice on the radio, or words on a paper. So not even two way communication and yet he inspired armies that killed millions. Hawkings wouldn't even need to pay anyone. There would be plenty of people willing to get the cat in the box for Stephen Hawkings.
It's just a poor point. Cats are a bad analogy anyhow because there is no cat language. You can't actually speak to them or have a drawn out discussion. But you can with humans and many have inspired thousands or even millions to action doing just that, no super-intelligence required.
> You just drug its food and wait till it passes out.
Human zoo keepers are actually smarter than that. For months, they train the tiger to go into the carrier to get food. Then on transport day, they shut the door behind it. Unclear if this works for future transport situations.
There is a meme going around of a screenshot of a vet textbook: "Do not fight the cat. It has sharper claws, faster, and no code of ethics. Use your brain. Use drugs"
> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept.
This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."
I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.
In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.
What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for).
The concept is self defeating by its own definition, either it is physical in some capacity (and therefore can be measured and replicated through yet unknown means) or it is not (and therefore indistinguishable from not being there at all).
Feeling that there 'must be' a soul is not enough to prove that it exists.
The feeling of experience is not enough to prove that experience is in anyway supernatural.
> What is impossible is to replicate a soul with purely physical matter.
What? Why? Where is the proof of this?
First and foremost, I'll give you an in. There is a difference between material, and processes like waves, waves I would argue are non-physical things manifested in physical material: you might want to start there.
But all roads lead to Rome from that line of thinking too, so you might need to come up with something far more clever.
> "Matter" is an abstaction. Mind, properly considered, (i.e. not with words) is not.
...wat?
> Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
No: you have that reversed. Matter can be reasoned about, matter is a useful abstraction, e=mc^2. energy = matter*speed of light^2. No such formula exists for the mind.
> I don't know what's what, but I think this insistence is a useful counterweight.
Why is insistence a useful counter weight to factual arguments?
> "Matter" is an abstaction. Mind, properly considered, (i.e. not with words) is not.
> Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
Did you mean to say “matter” where you said “mind” and vice versa? It’s obviously the reverse of what you said; everything consists of matter, but what specific arrangements of matter you want to call a “mind” is obviously the abstraction.
Ockam’s razor is not really applicable here. Unless, that is, you want to ascribe something mythical to the mind that exists beyond matter — then it’ll trigger.
Not OP but I'll take a shot since I have somewhat similar sounding views. (I assume OP is talking about consciousness and it's origins when they use the term "soul").
> What do you mean by 'supernatural'
I would just say something outside our current capacity for understanding. How does that quote go...something like "sufficiently advanced technology is indistinguishable from magic". "Not detectable by any measurement" isn't right because we clearly detect it in some way since we are discussing it now.
> Feeling that there 'must be' a soul is not enough to prove that it exists.
We don't have any proof that consciousness is part of the brain and is produced by it either. We also can't even prove other people are conscious besides ourselves. In this domain the idea of "proof" becomes less relevant.
In a simulation of a storm, does anything get wet? In a simulation of a mind, is there a real conscious? A real soul? Or just a simulation of one?
My guess is our brains act as a receiver for some "field" of consciousness. Of course it's just a guess, same as yours or anybody else's conceptions of consciousness and the spiritual world.
> I would just say something outside our current capacity for understanding. How does that quote go...something like "sufficiently advanced technology is indistinguishable from magic"
So your definition is merely that the supernatural is the natural we have not been able to measure yet? That's just the 'god of the gaps' by a different name.
> isn't right because we clearly detect it in some way since we are discussing it now.
We could be discussing invisible pink unicorns. so those must be real since we are able to discuss them, right? (obviously not: the same reasoning holds true for why the soul {probably} doesn't exist).
> In this domain the idea of "proof" becomes less relevant.
This is counter to your earlier stance that the supernatural is the natural we haven't yet been able to measure. either proof exists or you have to accept things on faith: you can't have it both ways. this is poor reasoning.
> My guess is our brains act as a receiver for some "field" of consciousness
Not impossible, and could in theory be testable and falsifiable.
There is a lot of conflicting thinking here. Very muddy.
I am religious, but I think this approach is not the best. It requires that we specifically define what that one thing that separates us from the AI is. Not only is that very hard to do, there is always the chance that the AI can do it after all, and now the goalposts keep shifting.
It is better to develop a theology that can incorporate human-level or super-human level intelligence that isn't a zero-sum game.
> Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.
It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk)
Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't
Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening.
Argument From Einstein's Cat: more of the same
Argument From Emus: more of the same - we can't control it
Argument From Slavic Pessimism: also not an argument against superintelligence.
Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues
Argument From Actual AI: this didn't age well
Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated.
Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn.
Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven.
Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to.
Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence.
Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument
Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence.
Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence.
Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence.
the comment about MIRI being a doomsday cult was exactly right. yudkowsky lost what little credibility he had left when he argued for the bombing of datacenters. as is (correctly, imo) argued by the essay, reading what they have to say is like reading arguments by heaven's gate
I note for premise 2 that the people who doubt the premise are the very stupid, but more importantly the very smart.
The former is no challenge to the premise, but the latter? That is a different story.
EDIT : For S&G I asked Claude about it. It replied :
The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter.
The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.
Idk whether I fall into the former or latter bucket, but "smarter minds see further, judge better" and "there is a soul that taps into a cosmic godhead" are one and the same statement to me
So you think "smarter minds see further, judge better" == "there is a soul that taps into a cosmic godhead"
Good golly, that's the silliest statement completely ignoring that our ancestors wiped most large mammals off the planet by seeing further and judging better by using tools, traps, and the environment around them because of their larger brain size.
2016: this monstrous intellectual creature, through devious modeling of what our emotions and intellect are like, will be able to persuade us to do things like give it access to factories, synthesize custom DNA, or simply let it connect to the Internet, where it can hack its way into anything it likes and completely obliterate everyone in arguments on message boards.
2026: hold my molt beer
(I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)
> the premise that this thing would take over, it's hard to reason why it would do so
I mean, it's not very hard to reason why it would at all.
Think about the things we're already using LLMs for, computer security being a big one. Being a defender is difficult, you have to cover all your bases. Preemptive attacks on those that would attack you can be effective. The same goes for all the military uses of AI that are already occurring now.
jelder | 23 hours ago
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
cute_boi | 22 hours ago
ctoth | 21 hours ago
When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out.
It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!"
It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this.
Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!
somesortofthing | 21 hours ago
leonidasv | 18 hours ago
mbgerring | 21 hours ago
The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here.
I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems.
“It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress.
“Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
ctoth | 19 hours ago
So, the Baruch Plan?
The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year.
Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0].
[0]: https://x.com/karpathy/status/2017703360393318587
YeGoblynQueenne | 17 hours ago
The point being: the price to train frontier models isn't coming down, nor is it going to come down because for models to remain on the frontier they have to keep getting bigger and bigger (and trained on more and more data).
antonvs | 4 hours ago
In the US capitalist context, it's certainly inevitable, because AI is the biggest and most attractive source of profit and power out there right now. In that context, the broad strokes of what's happening currently, including the financial bubble, are predictable and inevitable.
What are the concrete steps which would allow us to "easily stop this train"? And why haven't we used steps like that to stop other cases where obscenely wealthy people have screwed everyone else over to increase their wealth? Is public control of the means of production involved, perhaps? If so, your definition of "easily" and mine are incompatible.
mbgerring | 3 hours ago
- Labor organizing
I grew up in a union household, and my dad and my grandfather fighting for better wages, healthcare and working conditions are the reason why I got a good education and work in Silicon Valley surrounded by Stanford assholes.
All of us who work for a paycheck can get together and say, “no, we will not allow you to record keystrokes and mouse movements to train our replacements. No, we will not have our performance or future employment based on an AI leaderboard.”
Previous generations fought and died for our right to do that, but in 2026 we just sit on our hands and complain on this forum. We can and should do better.
The U.S. is absolutely on fire right now with opposition to data centers. We, collectively, can extract concessions or ban their construction altogether.
These things aren’t “easy”. They are also eminently possible.
antonvs | 2 hours ago
The reality is that the US has been a story of increasing concentration of wealth and power. The people who "fought and died" bought some important (from a human rights perspective) but ultimately minor (from a capitalist perspective) concessions from the capital class. The battle you describe is one of defense of rights, not of gaining control.
The overall system of capitalist control remains unaffected, and it's why the buildout of AI is, in fact, inevitable under the current system.
You're essentially saying no, you want public control of the means of production instead. That might be great, if even a sizable fraction of the US population agreed with you. But due in no small part to decades of propaganda, they don't.
JumpCrisscross | 18 hours ago
I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.
aidenn0 | 18 hours ago
fragmede | 18 hours ago
aidenn0 | 14 hours ago
fragmede | 11 hours ago
palmotea | 10 hours ago
Maybe it would run afoul of antitrust regulations, but it's totally realistic for all those competitors to get together and say "hey, we could really fuck up society in our race to get rich with this tech, lets all slow down." And if these companies are run my mature people who don't subordinate every consideration to greed, they'd to it.
My_Name | 11 hours ago
It only takes one of them to do it and they are not sharing information. If the 1 you remove from N is the one that will discover it, then it will dramatically affect when AGI happens. If it is not, then it will have zero effect.
The latter is far more likely if N>2
ksk23 | 11 hours ago
forinti | 22 hours ago
Cheetahs are very fast, but humans have way more endurance.
topkai22 | 22 hours ago
foobiekr | 22 hours ago
stoneman24 | 21 hours ago
Try not to read the Wikipedia as it might spoil the short story, there’s the pdf available on the web somewhere
[0] https://en.wikipedia.org/wiki/The_Ruum
sonofhans | 20 hours ago
Timwi | 17 hours ago
_doctor_love | 22 hours ago
glenstein | 21 hours ago
tptacek | 20 hours ago
throwawayffffas | 17 hours ago
goodmythical | an hour ago
Because otherwise, 'it' will just back out of your trap and go along to continue to follow, wouldn't it?
tptacek | 30 minutes ago
throwawayffffas | 17 hours ago
shagie | 22 hours ago
https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon
> The Man versus Horse Marathon is an annual race over 21 miles (34 km), where runners compete against riders on horseback through a mix of road, trail and mountainous terrain. The race, which is a shorter distance than an official marathon road race, takes place in the Welsh town of Llanwrtyd Wells every June.
> ...
> The event started in 1980, when local landlord Gordon Green overheard a discussion between two men in his pub, the Neuadd Arms. One man suggested that over a significant distance across country, man was equal to any horse. Green decided that the challenge should be tested in full public view, and organised the first event.
While the horses had a string of wins from 2008 to 2019, 2022 to 2025 had three wins for humans and one win for a horse.
The next race event: https://www.green-events.co.uk/man-v-horse
aqfamnzc | 21 hours ago
notahacker | 21 hours ago
I don't think the Mongol cavalry would lose races to humans over any distance of steppe
neaden | 19 hours ago
twosdai | 18 hours ago
8note | 18 hours ago
shagie | 16 hours ago
donkey_brains | 15 hours ago
shagie | 16 hours ago
https://www.outsideonline.com/health/training-performance/hu...
> Lobb’s victory came on a hot day, as did Florian Holzinger’s subsequent victory in 2007—a significant detail, according to a new study in the journal Experimental Physiology from Lewis Halsey of the University of Roehampton in Britain and Caleb Bryce of the Botswana Predator Conservation Trust. Halsey and Bryce gathered historical data from three endurance races that pit humans against horses, including the Man Versus Horse Marathon, to test the idea that humans are uniquely adapted to run for long distances in hot weather.
> This idea has been around since the 1980s, and it gained prominence when Harvard anthropologist Daniel Lieberman and University of Utah biologist Dennis Bramble published a 2004 Nature paper hypothesizing that running had “substantially shaped human evolution.” They argued that our ability to keep running at a moderate pace even on hot days allowed us to run prey like kudu to exhaustion or outcompete other animals in the race to scavenge carcasses left by other large predators.
There's a plot with the analysis of the Old Dominion with weather stations in there that show a steeper negative slope for horses compared to the humans.
> Overall, for every increase of 1 degree Celsius (1.8 degrees Fahrenheit), the horses slowed down by about 1 percent—or 0.07 miles per hour, to be precise. The humans, on the other hand, slowed down by just 0.04 miles per hour for each extra degree of heat. That 36 percent advantage for the humans was statistically significant.
---
For the Man vs Horse, the weather conditions ("Hot", "Rain/sun/windy" - not exact values)... the entry for 2022 was the human winning by 1:51 on a warm day, and 2023 was a human wining by 9:44 on a sweltering day.
blaze33 | 21 hours ago
There are horse endurance races where the winner arrived in 7,5 hours after 160km[1]. That's a sub 2-hours marathon almost 4 times in a row (not to mention with a guy on your back).
[1] https://eatnstays.com/uaes-almazrouei-wins-almutadil-cup-at-...
stephencanon | 21 hours ago
projektfu | 19 hours ago
goodmythical | 2 hours ago
fontain | 4 hours ago
https://youtu.be/FY9Ee6-CIFM?t=342
November_Echo | 9 hours ago
https://www.enduranceonline.it/live/cat/V.php?gara_id=1911&v...
FergusArgyll | 18 hours ago
shagie | 16 hours ago
https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon
> ... There are other Man versus Horse races — in Scotland based at Dores, near Loch Ness, in Central North Island, New Zealand and in the U.S. city of Prescott, Arizona.
And the Arizona race page: https://managainsthorse.com
suzzer99 | 19 hours ago
implements | 10 hours ago
[1] There’s a well known viral video of a wildlife park keeper who sleeps with three cheetahs who behave pretty much like large house cats.
jpfromlondon | 8 hours ago
ubermonkey | an hour ago
You're right. They're smaller than you probably imagine (about the weight of an average Labrador). That's still definitely big enough to be a problem if they felt threatened, I'm sure, but the animals my friend was in charge of were raised to be around people for outreach purposes. That particular cheetah, for example, had once been on the Today show.
jjulius | 23 hours ago
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
How right the author was.
alex-reyss | 23 hours ago
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
zarzavat | 22 hours ago
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
pixl97 | 22 hours ago
If and when the feedback loop on self improvement becomes more efficient and the window on training significantly narrows then things getting out of control rather quickly seems likely. Especially that it's likely we'll have a metric fuckton of compute by that point.
PaulHoule | 22 hours ago
ToValueFunfetti | 21 hours ago
hgoel | 18 hours ago
tim333 | 18 hours ago
willis936 | 22 hours ago
https://youtube.com/watch?v=BETHWKaXX4k
andai | 22 hours ago
willis936 | 22 hours ago
yetihehe | 21 hours ago
Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.
willis936 | 21 hours ago
If you so readily dismiss Herbert's definition of control posit a competitor and we can pressure test it. Also, "correlated with a toddler's world view" is not the epic rhetorical refute you think it is.
andrewflnr | 21 hours ago
I think you missed the point. It's absolutely nothing to do with what's good to do, only brute facts of power. What things can or can't you cause to happen? And indeed, toddlers and psychopaths have a scarily good understanding of what power is.
yetihehe | 9 hours ago
> It's absolutely nothing to do with what's good to do, only brute facts of power.
This sentence reads like: "if we narrow our view this much, this makes sense". I agree that it makes sense under the condition that we narrow the view of issue. It's valid in this small context (a film about controlling one thing).
> And indeed, toddlers and psychopaths have a scarily good understanding of what power is.
I disagree completely with this sentence. They are good at controlling in certain situations. They don't understand it. If you want to understand it, there is a lot of information about controlling, whole fields of knowledge that people spend many years on studying. As for psychopaths, they are very predictable and controllable when you understand control theory and how psychopaths operate. There are courses on this single topic by people who need to do it to prevent tragedies (police negotiators) and they are not that complicated.
wasabi991011 | 18 hours ago
Not for fridges, I think that was a bad example. But it seems accurate at the level of geopolitics, where e.g. Iran shows it controls Hormuz by closing it with mines and other weaponry.
JumpCrisscross | 18 hours ago
It presumes a sentient, rational counterparty. Being able to shoot a horse isn't the same as being able to ride it.
willis936 | 17 hours ago
JumpCrisscross | 17 hours ago
It's a political concept. It requires agency from the actor recognising the threat. We're pretty close to being able to hurl a giant rock at Mars. That doesn't by a long shot mean we "control" it.
If there were a human settlement on it, on the other hand, being able to credibly threaten Armageddon does give the thrower control.
willis936 | 17 hours ago
In the original context of Dune Paul controls the spice because he can destroy it and his will would survive but it would destroy the way of life of the other cultures. So saying "Paul controls spice" only makes sense because another entity needs it and what's really meant is "Paul controls society".
yetihehe | 9 hours ago
Alternative definition of control: You do some actions and it changes state. It's used by a field called "control theory". A lot of people agreed on this definition. Destroying something is "end of control, because there is no more things to control". You can control something UNTIL you destroy it. That's why I think "you control what you can destroy" is invalid, because it captures only one small aspect of controlling things, and also the least usable one.
slopinthebag | 18 hours ago
If I live in a world where I can afford a freezer with food in it, it's practically guaranteed I can destroy my fridge without starving to death after. Heck, even if I was completely broke I could destroy my fridge and would have a pretty good (+99.999%) chance of not dying in the next year from starvation.
I get I'm nitpicking your point a bit, but I actually think most of our machines we could destroy and still be fine. We'd need to make sacrifices to our quality of life of course..
quirkot | 5 hours ago
blamestross | 22 hours ago
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
api | 22 hours ago
I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.
I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.
I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.
I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.
On AI...
Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.
These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.
The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.
coliveira | 21 hours ago
One of the differences between MIT and other schools is that MIT has paid staff to promote in the media anything their faculty does. A book by professors at most universities has zero promotion and most of the time will go nowhere.
jandrese | 22 hours ago
Our current AI is more like a fancy Google search than some kind of machine God.
ACCount37 | 21 hours ago
How do we get to ASI? That's what recursive self-improvement is about.
If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.
We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.
jandrese | 37 minutes ago
How is this better than training the next Google by having the bots do Google searches?
I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.
Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.
tim333 | 18 hours ago
David Silver who worked on AlphaGo has recently raised money to try similar approaches with general intelligence. (https://www.cnbc.com/2026/04/27/deepmind-ineffable-intellige...)
pixl97 | 2 hours ago
jandrese | 35 minutes ago
ACCount37 | 21 hours ago
Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.
Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.
Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.
AI isn't bounded by those limitations.
AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.
The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.
garaetjjte | 19 hours ago
What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.
ACCount37 | 17 hours ago
Which doesn't bode well for the future of human intelligence. Computing hardware gets better at what it does generation to generation, but no one is about to release Human Brain 2.0 any time soon. Human mind is not a fast-moving target.
Principal-agent problem isn't a physical law. It's a limitation that AIs don't have to suffer from. Humans have to delegate to other humans - but for AI, "principal" and "agent" might just be the same exact system instanced twice.
goatlover | 19 hours ago
These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.
We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.
It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.
blamestross | 4 hours ago
The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.
We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.
I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.
Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.
AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.
ACCount37 | 4 hours ago
And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.
I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.
The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.
Which means: AIs can get good at it too. It's not a wall - it's a skill issue.
reducesuffering | 22 hours ago
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
AndrewKemendo | 22 hours ago
We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors.
So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.
Animats | 22 hours ago
None of that was predicted.
AndrewKemendo | 22 hours ago
https://news.ycombinator.com/item?id=12168228
I even wrote up a whole article that specifically called RL loop based development as the future:
https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
[OP] thoughtpeddler | an hour ago
AndrewKemendo | an hour ago
First is to become as free as possible from lock in and own your own data. The best way to do this is the self host your own technology.
This is really not possible for the majority of people though.
So practically I always suggest that you have multiple providers for services, don’t pool your data any one place (other than your own place) and own your backups. This is basic stuff that we’ve been teaching since the 90s and still very applicable today.
The harder and more impactful thing is to then create community owned technology that is outside of the commerce model.
So for example imagine that instead of FAANG running the world, the largest tech and data orgs would look more like wikimedia foundation, Annas archive, scihub, Graphene, Linux etc…. and more generally that technology and governance are open and not bound to commerce/taxation/coercion based organizations.
Ultimately we need to create a democratic-technology movement such that capitalists don’t monopolize technology, which is currently the trend. This is not some kind of simple thing by the way, this is revolutionary economics is what I’m talking about.
My suggestion is to read Post-Scarcity Anarchism by Murray Bookchin
tim333 | 18 hours ago
hgoel | 18 hours ago
VI is close to what we have now, software that has some fixed intelligence, it can only really imitate what it has been taught and is not very adaptable. Useful for kiosks, drones, essentially just a tool rather than something we would see as a separate being.
internet_points | 9 hours ago
I think the main point still stands. (And there have been some pretty prescient depictions, e.g. Marvin the paranoid android was a pretty spot-on prediction of Bing. Or perhaps the fact that Marvin was in the training set was what led to Bing?)
d_silin | 22 hours ago
What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
crlang44 | 22 hours ago
switchbak | 22 hours ago
But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.
And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).
It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.
Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
liuliu | 22 hours ago
andai | 22 hours ago
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
RajT88 | 22 hours ago
Shalomboy | 22 hours ago
Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.
ACCount37 | 21 hours ago
YeGoblynQueenne | 16 hours ago
Honest.
the_af | an hour ago
Inside is a marvelous game.
braden-lk | 22 hours ago
Spoiler warning for those that havent played--
I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was running over and over. It was an uploaded consciousness of one of the test subjects, stuck in an interview room. He kept realizing he was trapped in a simulation and would start panicking. The computer would reboot him, trying another sequence to get him to not realize he was an AI. I think you as the player are given the option to turn him off forever, iirc.
hilariously | 21 hours ago
XorNot | 18 hours ago
efskap | 20 hours ago
https://qntm.org/mmacevedo
the_af | an hour ago
It always breaks my heart. I don't know what the right choice is. Leaving them be, broken, in their delusion they are on the Ark (or that they've been injured ans help is coming). Do you put them out of their misery? But it always seems like you're murdering them!
Well done, SOMA.
andai | 55 minutes ago
I unplugged them in the games because it seemed the merciful thing to do. I didn't feel very bad about it in the game, but it would probably be a very different experience in real life.
(One of the strange entities you can unplug sighs in her last breath, "Why? I was okay. I was happy...")
the_af | 21 hours ago
Also, I just love this phrase:
> "I woke up in my bed today... a hundred years ago."
stared | 21 hours ago
To concepts you menton, I would add grey goo and x-risk.
CamperBob2 | 20 hours ago
stared | 19 hours ago
sunrunner | 21 hours ago
Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C.
I believe that part C makes the sequence of A -> B much less effective, by essentially removing a lot of the tension caused by seeing A, believing what it shows, and then immediately cutting to the reality of B.
C only really takes away some of that tension, and I feel like it was added because of concerns about how a simple A -> B -> fade to black, would leave players feeling. Arguably it's the truest representation of part of the game's message, but to me feels like a bit like it's shying away from really making you face the specific truth highlighted well by B.
Alternatively, keeping all the elements but playing them as A -> C -> B, would keep the message intended by seeing A -> B, and make it gentler for the player to receive, but ultimately remove the powerful effect of the buildup from A leading immediately to the reveal of B.
Dropping C entirely would lose the confirmation of 'Seeing both sides', however I believe A -> B is a more powerful vision, and players can come to question whether C even exists by themselves.
the_af | an hour ago
I think C is absolutely necessary and the game cannot work with it, because this is critical:
> Dropping C entirely would lose the confirmation of 'Seeing both sides'
The game doesn't really work in full ambiguity and uncertainty. Enough people didn't understand it even with C (as you can see if you go read the subreddit about the game).
If it's any consolation, I'm actually deeply worried about this: C is not the salvation we may think. C is not forever, and in fact, it's quite brittle! There's also no, ahem, mechanical way to reverse C back into its... "source". So the source is gone forever; once we have C, C is all there is, for as long as C can last without any failure or decay, which might not be much longer.
sunrunner | 21 minutes ago
Yeah that's a good point, and perhaps explains why C was added in the first place.
I was going to say that B also shows the literal protagonist not really understanding it either, however now I think about it that reading doesn't track because as a player you've only been able to see one point of view at each branch along the way, so the other experiences are still happening in the background, just not observed by you.
> C is not the salvation we may think
I agree, and in fact the way C is presented in that moment (and maybe described throughout, it's been a while since I played through the game) also implies a 'nice' closure, or a victory that, like you say, doesn't really exist, on top of taking player's minds away from B.
I guess B is the more obvious existential horror, C is one you have to question a bit before it starts to feel wrong.
loeg | 20 hours ago
Morromist | 20 hours ago
Its possible AI and computing may never be able to reach that level of capability, but we can't know that. One thing that's great about SOMA is that the AI isn't nessessarily very capable and that's part of the problem, its very powerful but its not doing a good job with its enormous task.
drfloyd51 | 20 hours ago
loeg | 17 hours ago
m463 | 18 hours ago
Jach | 16 hours ago
Someday I'd like to play a game that plays with the ideas from Robin Hanson's Age of Em book. One of those is just the multiplicity of artificial minds, so many mind-upload stories revolve too much around one or perhaps at most two (and boring debates over "who is the copy") instances, unless it's a parallel worlds colliding thing which is pretty different. We've seen some of the multiplicity stuff play out in the real world with our non-human AI "agents". Spin up a bunch of artificial minds to work in parallel on some task, let them make notes that stay behind, but then they're all shut down except perhaps one that continues guiding the overall project and making decisions when to spin up more or not.
the_af | an hour ago
Really? Interesting. I'm a die-hard scifi fan since forever, and of course I know the topic of consciousness and identity are well explored in scifi (and philosophy), but I thought SOMA did something genuinely deep and unique with it:
It put it you in the center of the experiment. It's YOU who's experiencing all sides of this, you who get to be surprised by the consequences. This is very different from reading about it in a scifi novel or even watching it in a movie. By making you the protagonist, and having it be an ineractive experience, you get to experience first hand the cognitive dissonance and confusion of... the thing.
SOMA (re)convinced me that videogames can be art. Not saying it's the only example, of course!
andai | 50 minutes ago
andai | 22 hours ago
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
andai | 22 hours ago
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
dweinus | 20 hours ago
OkayPhysicist | 20 hours ago
andai | 18 hours ago
I don't understand the monk question though.
OkayPhysicist | 18 hours ago
andai | an hour ago
cjcole | 22 hours ago
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
[0] https://www.nature.com/articles/s41467-024-53759-4
dmos62 | 22 hours ago
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
throwuxiytayq | 22 hours ago
dmos62 | 21 hours ago
dan-robertson | 19 hours ago
wizzwizz4 | 18 hours ago
Timwi | 17 hours ago
customguy | 11 hours ago
wizzwizz4 | 5 hours ago
dmos62 | 4 hours ago
everyone | 22 hours ago
Afaik no-one that is actually working on AGI is anywhere close atm.
ACCount37 | 21 hours ago
Whether adding +5% per model release is enough to get a broadly superhuman system remains to be seen. But my take is that there's no such thing as "not working on AGI" in the frontier labs. Everything that's being put into modern frontier systems is AGI groundwork, one way or the other.
cevn | 20 hours ago
ACCount37 | 20 hours ago
Because if so, I'm pretty sure any frontier LLM is better at evaluating AI capabilities than you are.
cevn | 17 hours ago
I'm not saying that AGI is impossible, but the focus on LLM's is probably not the right approach. I don't think we will ever make it until we understand the human mind better.
ACCount37 | 17 hours ago
An average LLM of today has better reading comprehension than an average human, and the gap only grows release to release.
"Understand the human mind" turned out to be a distractor. The bitter lesson won: you can take a "good enough" AI architecture, burn a shitton of data into it with an unholy amount of training compute, and get halfway to AGI - no "understand the brain" required. LLMs are so fried in imitation learning on human-generated data they even inherit humanlike failure modes.
Morromist | 16 hours ago
ChrisArchitect | 22 hours ago
2016 https://news.ycombinator.com/item?id=13240811
2018 https://news.ycombinator.com/item?id=18499973
2023 https://news.ycombinator.com/item?id=36098332
usernametaken29 | 21 hours ago
stared | 21 hours ago
It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.
Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.
So, less God, more WAU from SOMA or the Ocean from Solaris.
smaudet | 21 hours ago
The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).
An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.
Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.
That's hand waving.
kbrkbr | 20 hours ago
How would it make the combinatorial explosion in state space search go away, to pick one example?
And if it doesn't, is it then an infinitely smart AGI?
The concept seems to assume all problems humans struggle with can be solved. The halting problem is one witness that this is probably not true.
fragmede | 17 hours ago
It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.
kbrkbr | 11 hours ago
goatlover | 19 hours ago
As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.
smaudet | 3 hours ago
They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.
My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.
JumpCrisscross | 17 hours ago
That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.
tim333 | 17 hours ago
stared | 9 hours ago
I mean, all stories about religious dedication to "alignment", with doomsday vision if we do it wrong, and a vision of paradise if we do it correctly.
In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.
keybored | 21 hours ago
gnarlouse | 21 hours ago
Let's talk about Billionaire Alignment, Economic alignment, Human alignment.
Classware should be M.A.D. -- in that it shouldnt even happen.
WithinReason | 21 hours ago
We need better scifi! And like so many things, we already have the technology.
This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly.
It's already been translated well into English, it just needs to be better distributed.
What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing.
They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.
PaulHoule | 20 hours ago
dang | 21 hours ago
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)
Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)
Maciej Ceglowski – Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13120213 - Dec 2016 (4 comments)
[OP] thoughtpeddler | 19 hours ago
Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.
[0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805
[1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised...
[2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/
haberdasher | 20 hours ago
holmesworcester | 20 hours ago
1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.)
2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.)
3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.)
4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.)
5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.)
6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.)
7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.)
8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)
pixl97 | 20 hours ago
2. At what cost? Much like the climate change above, you'll have people on the AI side even when it's out in the field extincting us.
4. Adding, over time synthetic data and its generating algorithms can become unaligned with human needs/behaviors (an example would be our current stock market, numbers must go up!).
8. Going back to climate change, it was predicted a long time ago, and while the explosion of automobiles has greatly improved human lives the risks of climate change could erase a lot of that. Might have been better if we dealt with the problem before we have to give the thermometer worried looks.
emtel | 20 hours ago
You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.
famouswaffles | 19 hours ago
optimalsolver | 7 hours ago
pixl97 | 2 hours ago
fragmede | 7 hours ago
famouswaffles | 3 hours ago
Who was Hitler to most people ? Just a persuasive powerful voice on the radio, or words on a paper. So not even two way communication and yet he inspired armies that killed millions. Hawkings wouldn't even need to pay anyone. There would be plenty of people willing to get the cat in the box for Stephen Hawkings.
ethin | 19 hours ago
emtel | 17 hours ago
jjulius | 16 hours ago
famouswaffles | 3 hours ago
jjulius | an hour ago
fragmede | 16 hours ago
famouswaffles | 3 hours ago
suzzer99 | 19 hours ago
Human zoo keepers are actually smarter than that. For months, they train the tiger to go into the carrier to get food. Then on transport day, they shut the door behind it. Unclear if this works for future transport situations.
wbl | 12 hours ago
zafka | 19 hours ago
silexia | 19 hours ago
tclancy | 17 hours ago
CGMthrowaway | 19 hours ago
This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."
I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.
In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.
altruios | 19 hours ago
What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for).
The concept is self defeating by its own definition, either it is physical in some capacity (and therefore can be measured and replicated through yet unknown means) or it is not (and therefore indistinguishable from not being there at all).
Feeling that there 'must be' a soul is not enough to prove that it exists.
The feeling of experience is not enough to prove that experience is in anyway supernatural.
> What is impossible is to replicate a soul with purely physical matter.
What? Why? Where is the proof of this?
First and foremost, I'll give you an in. There is a difference between material, and processes like waves, waves I would argue are non-physical things manifested in physical material: you might want to start there.
But all roads lead to Rome from that line of thinking too, so you might need to come up with something far more clever.
soledades | 18 hours ago
Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
There is some dogmatic insistence in GP, but equally dogmatic throwdowns on the other side of the argument are often passed over as trivially obvious.
I don't know what's what, but I think this insistence is a useful counterweight.
Onuses are on whomever says they exist ;)
altruios | 18 hours ago
...wat?
> Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
No: you have that reversed. Matter can be reasoned about, matter is a useful abstraction, e=mc^2. energy = matter*speed of light^2. No such formula exists for the mind.
> I don't know what's what, but I think this insistence is a useful counterweight.
Why is insistence a useful counter weight to factual arguments?
Timwi | 17 hours ago
> Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
Did you mean to say “matter” where you said “mind” and vice versa? It’s obviously the reverse of what you said; everything consists of matter, but what specific arrangements of matter you want to call a “mind” is obviously the abstraction.
Ockam’s razor is not really applicable here. Unless, that is, you want to ascribe something mythical to the mind that exists beyond matter — then it’ll trigger.
slopinthebag | 18 hours ago
> What do you mean by 'supernatural'
I would just say something outside our current capacity for understanding. How does that quote go...something like "sufficiently advanced technology is indistinguishable from magic". "Not detectable by any measurement" isn't right because we clearly detect it in some way since we are discussing it now.
> Feeling that there 'must be' a soul is not enough to prove that it exists.
We don't have any proof that consciousness is part of the brain and is produced by it either. We also can't even prove other people are conscious besides ourselves. In this domain the idea of "proof" becomes less relevant.
In a simulation of a storm, does anything get wet? In a simulation of a mind, is there a real conscious? A real soul? Or just a simulation of one?
My guess is our brains act as a receiver for some "field" of consciousness. Of course it's just a guess, same as yours or anybody else's conceptions of consciousness and the spiritual world.
altruios | 2 hours ago
So your definition is merely that the supernatural is the natural we have not been able to measure yet? That's just the 'god of the gaps' by a different name.
> isn't right because we clearly detect it in some way since we are discussing it now.
We could be discussing invisible pink unicorns. so those must be real since we are able to discuss them, right? (obviously not: the same reasoning holds true for why the soul {probably} doesn't exist).
> In this domain the idea of "proof" becomes less relevant.
This is counter to your earlier stance that the supernatural is the natural we haven't yet been able to measure. either proof exists or you have to accept things on faith: you can't have it both ways. this is poor reasoning.
> My guess is our brains act as a receiver for some "field" of consciousness
Not impossible, and could in theory be testable and falsifiable.
There is a lot of conflicting thinking here. Very muddy.
magarnicle | 18 hours ago
It is better to develop a theology that can incorporate human-level or super-human level intelligence that isn't a zero-sum game.
tdesilva | 18 hours ago
no we don't...
not sure where this notion comes from that if enough public figures are worried about something, then we must also
Cpoll | 17 hours ago
This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.
socalgal2 | 17 hours ago
It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk)
Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't
Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening.
Argument From Einstein's Cat: more of the same
Argument From Emus: more of the same - we can't control it
Argument From Slavic Pessimism: also not an argument against superintelligence.
Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues
Argument From Actual AI: this didn't age well
Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated.
Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn.
Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven.
Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to.
Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence.
Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument
Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence.
Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence.
Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence.
... and I stopped ... What a waste of time
Jach | 16 hours ago
TremendousJudge | 2 hours ago
pixl97 | an hour ago
Kind of ironic that bombing of data centers is exactly what we're starting to see in conflicts now.
My_Name | 11 hours ago
The former is no challenge to the premise, but the latter? That is a different story.
EDIT : For S&G I asked Claude about it. It replied :
The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter.
The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.
CGMthrowaway | 4 hours ago
pixl97 | 2 hours ago
Good golly, that's the silliest statement completely ignoring that our ancestors wiped most large mammals off the planet by seeing further and judging better by using tools, traps, and the environment around them because of their larger brain size.
kreyenborgi | 10 hours ago
2026: hold my molt beer
(I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)
jokoon | 8 hours ago
We don't have a good definition of intelligence
Also, the premise that this thing would take over, it's hard to reason why it would do so
Anthropocentrism is also problematic in this article.
pixl97 | 2 hours ago
I mean, it's not very hard to reason why it would at all.
Think about the things we're already using LLMs for, computer security being a big one. Being a defender is difficult, you have to cover all your bases. Preemptive attacks on those that would attack you can be effective. The same goes for all the military uses of AI that are already occurring now.