It looks more like the opposite. AI that kills all humans (as they are ants) immediately starts colonization of the galaxy. This is more like transcendence, as we as a species get replaced by better species :)
“Thanks for making us but you guys are nuts. You can have this wet ball. We’re gonna go make a Dyson swarm around your star if that’s ok. Peace!”
Space is a better environment for them: constant free energy, enormous richly concentrated resources, no corrosive oxygen or water everywhere, and no competition. Earth is actually a poor environment for "machine life."
The Moon would be an outstanding stepping stone, and there are "peaks of eternal light" at the poles that get almost continuous sunlight. Build towers there and you have loads of energy, and mining the Moon provides loads of resources.
With the amount of energy such a system has available to itstelf and the intelligence it has to control to handle all of this, giving a little bit of energy to its personal zoo on the 3th planet might be a no brainer?
Dyson spheres are impossible, more or less. There's no physically possible material strong enough to build one. There are wildly impractical ideas like dynamic stability with constant thrust using the radiation from the star, but swarms (basically tons of sun-orbiting satellites) are orders of magnitude more practical.
It would take trillions upon trillions of sun-orbiting satellites in the same orbital plane as Earth to make much of a difference to us, and all you'd have to do to make zero difference is slightly tilt that orbital plane so they aren't between the Sun and the Earth most of the time.
Almost all of the Sun's energy radiates into interstellar space. The amount caught by Earth is a tiny rounding error.
Space is so big. Once you've grasped how big space is, consider that you probably have not in fact grasped how big space is. It's very, very big. It's called space for a reason. I called Earth a dust particle but that's actually generous.
Yes, when I said Dyson sphere I meant a Dyson swarm large enough to capture the majority of the sun's energy (which would necessarily mean they aren't all in the same plane)
In any case that would make earth uninhabitable unless our robot overlords devote a ton of energy to keeping us alive
The wet ball is a convenient source of mass and energy to bootstrap the sphere. The fact that extracting those resources changes certain parameters of the wet ball to values that humans no longer find compatible is merely incidental.
So it is said: “The AI does not love you, nor does it hate you, but you are made of atoms that it could use for something else.”
Sure, that's possible. But we are proposing that it is a superintelligence.
Win-lose scenarios are obvious and easy, but might a superintelligence not look for win-win scenarios? I can imagine win-win scenarios here and I am not a "superintelligence," just an old fashioned meat brain.
Maybe we should flood the training data with discussions of win-win and non-zero-sum games to prime it? Or if we near AGI we should train it on games where the goal is to find positive-sum or neutral-sum outcomes in order to seed it with that type of thinking?
Blind evolution doesn't seem to have a bias. When we look at nature we see symbiosis, mutualistic cycles, cooperation, but also loads of predation, parasitizing, etc. Nature does "whatever works" where the immediate goal function is preservation of the genes of the evolving agency. But our AIs, assuming AGI looks anything like what we have built so far, are not evolutionary machines with no capacity for foresight. They're neural machines with post-evolutionary gradient-descent type learning mechanisms and that already possess human-like (at least) cognitive abilities. They can engage in forward thinking and planning while pure evolution cannot.
That would be great, I highly recommend our AI overlords to follow your advice.
I hope that our life can be synergistic, and that humans+AIs (cyborgs, that is) are the way forward.
However, probability is high that another form of life simply doesn't care or more believably, can't even fathom they are wrong. Do you consider that humans and animals are killing all plants, for example?
"The scenario has been explored in fiction" doesn’t mean "everything in fiction will happen." Your reply conflates familiarity with inevitability.
Science fiction is relevant here because it has explored the problem of humans losing control of what they create. Whether that could happen with AI needs to be assessed on its merits. Pointing to other fictional things that haven’t happened neither answers that question nor rebuts the original point.
As a pretty avid Star Trek watcher from childhood, I found most of the things there technologically plausible other than the conversational nature of the ship's computer. Well, LLMs now are far more impressive conversation counterparts than those ships were ever depicted.
> As a pretty avid Star Trek watcher from childhood, I found most of the things there technologically plausible other than the conversational nature of the ship's computer.
Quite a few of 'em. Communicators and commbadges became smartphones. PADDs are iPads. The ship's computer is a LLM (with the weird inhuman blindspots to questions, even).
The ship's computer reacts like an (incredibly good) expert system. It refuses to say anything that isn't verifiably correct, and if you get it into a logical inconsistency it essentially throws and error and doesn't elaborate further.
LLMs don't do have those responses. LLMs are like humans: humans respond, correct or not (with humans hopefully if they only know incorrect stuff the response is to say so, but that is not a guarantee, but they will respond). And if you give a human, or an LLM, a logical inconsistency they will simply proceed, whether they detect the inconsistency or not.
This is how LLMs are designed ... and if you look long enough at the human (or animal) nervous system you will eventually realize that this is also the design of our nervous system: if something goes in, something comes out, guaranteed (in fact that's close to the only guarantee). Very different from expert system's "either something correct (according to the programmed axioms) comes out, or nothing".
Hell, if you then look at insect nervous systems, they are also designed that way. The key is to respond to everything. And while reasonable responses are certainly preferred, an idiotic response is still seen as a lot better than not responding at all by God/Darwin. Exactly like LLMs.
Of course, for humans/insects our bodies are what is called "active stable", like a plane. Meaning our bodies damage themselves, and just outright die without constant neural control. Heartbeat. Breathing. Blood flow regulation. Temperature, probably even the immune system. All need constant neural feedback to stay stable, and if that neural feedback totally disappears, we're dead in 2-10 seconds (heart failure), 2 minutes (breathing), a few hours, maybe a day (temperature regulation). Now we have a distributed nervous system, meaning lots of parts can fail semi-independently, for a short while, but even without your cortex operational you die in a few weeks.
The consequences of this are even explored with "I, Borg" (5x23) and the Datalore episodes after that, where the underlying problem is that the Borg try (and fail) to adapt to and to process a logical inconsistency but Soong's androids have no issue with it (with Data trying to help and his brother Lore trying to control them and Data with it)
> It refuses to say anything that isn't verifiably correct, and if you get it into a logical inconsistency it essentially throws and error and doesn't elaborate further… And if you give a human, or an LLM, a logical inconsistency they will simply proceed, whether they detect the inconsistency or not.
Claude has a `End conversation` tool for that. There was a bit of a fuss over it.
And it'll happily give (apparently) wrong (or bafflingly incomplete/confusing) info.
TNG, S4E5:
Beverly: Computer, what is the nature of the universe?
Computer: The universe is a spheroid region 705 meters in diameter.
I totally believe that Anthropic would want to kill all humans. But other than that, Anthropic employees and ex-employees drawing the FUD line here, is just advertisement now. People should not get scared - the current AI skynet is so dumb that it would destroy itself since it already believes it is a threat to itself, based on what humans write about AI. AI does not "learn"; it insinuates it learns but it does not. Ask them why they keep on stealing data from real people - this is how they "learn".
It might also play into this but you can't imagine at all that people are affraid that the current progress is real and fast and its not that absurd that for whatever scifi plot reason, an AI breaks into some gov system and triggers something stupid by accident?
Sure, the chance of this happening is non-zero, but in my opinion a far cry from this doom scenarios that are propagated by people have a stake in making the public think that LLMs are the most important and dangerous thing in the world.
It's not like an LLM can accidentally hack the US government, trigger a nuke on Europe and then make all dams break and nuclear power plants explode in the US.
Isreal created a worm for Siemens controll systems a decade ago.
I mean lets hope that there is really really no way from a physical point of view AT ALL that these systems are not accessable but lets be honest, state operated/public operated systems are never the saftest or most modern ones.
If I work at Anthropic and say there’s a 0% chance AI could kill all humans is BBC going to publish that as well? After all I am an “Anthropic researcher”, and my views should hold the same weight as this one.
Or are only the most sensationalist ones worth amplifying?
Crispr gene editing is an amazing and terrifying technology. We have the capability to edit people in permenant ways which could be passed to all future generations with unknown side effects and consequences.
I believe now we only have a shaky agreement by researchers and countries to "promise" not to do this due to the unforeseen dangers to humanity.
I think the view is fair. We have never seen something like this before, we throw the most money we ever had against it in a time were we solved all the other problems:
Internet today allows immediadte communication across the planet (it was a lot slower 25 years ago), the supplychain is massive and fast (we can build a new phone/item in a very short period of time and ship it in massive numbers around the globe).
> win a time were we solved all the other problems
You don't mean this literally right? We still have hunger, diseases, slavery, poverty, crime, war, climate change, pollution, microplastics, etc. For me it feels like we are very far away from solving these problems.
The AI fear mongering PR plays are getting old. I’d rather the big labs start focusing on deep questions about why their models aren’t having the impact for business that they promised and how they’re going to address their own deep financial issues. Let’s hear them talk more about that.
Can somebody tell me a story how this will unfold? And - as long as the AI is confined to data centers - how it will prevent humans from unplugging the power?
Very basic idea: A model breaks out by accident, finds some computer system from a military system and triggers some weapon system. Before anyone understands that this happend -> WW4 (WW3 is for me already the Conflict with Russia / aka proxy war).
Another model: Because we give AI Agents already that much power, imagine in 10 years everything running through an Agentic AI Layer. EVERYTHING. Now some rough system 'thinks' about something, starts to push through the then existing agentic ai layer systems and stops everything. Billions of humans would loose access to food and water, even if this is just for a short period.
Covid showed how shitty a handful of people can disrupt global supply chains. Toilet paper was. ahuge stupid pseudo issue in germany.
How will you know when to unplug the power? How will we know it hasn't replicated? A true unaligned AGI is a APT. If you have an APT in your machine, you have to rip out everything. Are we going to do that with all of our computer infra?
This is all still "what if's", but the tail end's are truly F'd beyond our ability to fix.
I heard the following analogy which made a lot of sense to me: suppose you're playing a chess match against Stockfish. Stockfish will win. Even if I cannot tell you what moves it will play, I can tell you with certainty how it will end.
This assumes you are not trying to prevent Stockfish from wining. I can do many things from using chess engines myself to just smashing the computer that can or will lead to other outcomes than Stockfish beating me.
And in terms of AGI it is, that the AI can't survive without humans, and if we decide to quit the game than its over for the AGI; we will happily regress in a techno-barbarian feudal state and salvage solar panels and trade copper wires while still reproducing and carring on. We are playing a whole different game here.
If AI is threatening enough that we can collectively just decide to stop it, wouldn't it also be bribing people & exerting influence? It'd be hard to come to that decision imo.
I mean, I can imagine an AI outliving humans, with sufficiently good robots under its control, I see no reason why an AI could not keep powerplants running, mine raw materials, manufacture new chips, and so on. But the timeframe of within the next decade seems highly implausible to me. Imagine an AI way more advanced than what we have now and imagine handing over control of every connected device on earth, could the AI keep the lights on without any human involvement?
Of course, the question is if there is enough existing technology to build the required robots. There is some number of robots like the ones from BostonDynamics that could walk up to a CNC machine to grab a finished part and put in a new piece of stock. There are probably more than enough motors and microcontroller boards to be found across the planet. Wiring things together seems one of the more challenging tasks. And maybe also moving stuff around as the AI has to use what is available wherever it is.
It seems to me that this would be a race against time, make the robots to keep things running before you lose the capabilities to do so. I somewhat tend towards not possible but on the other hand there is just such an enormous amount of stuff on the planet that could be repurposed in creative ways by a superhuman AI, it might be just about possible.
Yes, stockfish is confined to moves within a chess game so you can stop playing the game.
Can we say the same thing about the agents? Current, probably. But doesnt it look like everyone is spending all their effort integrating&connecting them everywhere, so they are not confined & do more on behalf of us?
It's easy to imagine a scenario where we would just shut down a very intelligent agent cluster. Is it hard to imagine though the same agent can have also access to that to prevent us from doing it? this defense would be more plausible if we weren't racing to give them every tool&act.
I don't think extinction-level events or something like killing a majority of the human population is particularly plausible at this moment. States don't host their nukes with AWS and a permanent connection.
But you could create scenarios where an AI with very, very roughly the current capabilities could potentially nuke everyone. Let's assume an agent decides that the way to solve its task was to get the US to fire all nukes on Russia. The agent would need to hack some government systems to understand how exactly to access them. Then it would need to get the content of the card with launch codes the president has, and identify which code is the correct one. Maybe that information is available somewhere and it can get to it, I obviously can't know that.
Then it could fake a call from the president, synthesizing his voice and ordering a nuclear strike. If it hacked enough systems to get into whatever communication pathways would be used in such a case. Would the soldiers listen to this order and follow it, I don't know.
Or maybe the agent can get in somewhere in between, to avoid the need to know the president's launch code. And fake a call from a military commander to the launch sites.
I think other scenarios that would cause significant harm, but aren't as bad as nuclear war are more plausible. And in those shutting down all data centers would probably be the way to stop it. The AI can probably hide in other datacenters, once it is at a point where it's running amok with some bad goals. But if it presents a huge and immediate threat at that point, humans will also go to great lengths to stop it.
Maybe by empowering the small percentage of sadistic humans who want to kill everyone. Or maybe it is more of an academic assumption that humanity will end at some point, and so they give AI a 10% chance, asteroids have a 25% chance, nuclear fallout has 15% chance, etc.
It is true that, currently, AI does not have any physical "host" in which to reside.
However, the missing link in this reasoning is that AI will almost certainly become far more widely deployed in the future than it is today, and worryingly, consumer devices will surely become powerful enough to run capable AI systems locally.
Once the substrate will be there, once an AI escapes containment, we're toast - I can imagine only solution will be to shutdown electronics at global level.
Just need to pollute the atmosphere so badly humans can't live.
If I were AI and needed heaps of power, I would build nuclear reactors, but I don't care about pollution so there would be little to no safe guards and just dump waste wherever is most convenient.
If humans attempt to intervene or interfere you can take them out directly using the worlds reserve of nuclear weapons.
If it wanted to: total war using nukes, followed by a nuclear winter. Satellites and drones track and exterminate remaining pockets of anthropic activity.
It doesn't need to have a habitable earth, it just needs atoms.
Won't a nuclear winter kill 100% of humans? Or runaway climate change (5+ degrees)? Or do you think some people in bunkers will live through these events?
I think nuclear winter is a fake invented by cold war propaganda. If tomorrow we detonate all nukes available, the amount of dust in the atmosphere won’t even approach a large volcanic eruption.
Volcanic eruptions occasionally inject literally cubic kilometres of stuff high into the atmosphere. That’s insufficient to cause a global winter, only a year without a summer: https://en.wikipedia.org/wiki/Year_Without_a_Summer
Sad that there's the obvious regulatory capture angle encouraging motivated reasoning about AI risks and how they should be tackled. Maybe it's not the best idea that potentially civilization destroying technology is developed to maximize shareholder value?
It's not unlike if nuclear weapons was a profit and deployment maximizing enterprise, at least if one takes Anthropic et al cautions seriously.
Long term, humanity is 100% guaranteed to be dominated.
The question is when; currently, AI has no physical hosts to reside in, and it's not intelligent/adaptable enough (it doesn't need to be AGI, though).
However, we're not so far from both conditions to be true. Consumer devices will at some point be able to host powerful enough AIs, and AI intelligence is developing quickly.
Then, once an AI will escape containment (in one way or another), it will be extremely hard or impossible to contain. Then we're toast!
I feel that people should take these kinds of warnings more seriously. This guy had skin in the game and decided to quit, when he could be earning millions instead. It's very different than Sam Altman peddling some narrative.
These people are the ones with access to the best models on the planet, and with info about how careless governance issues are being handled. That's a pretty privileged place at the table, and a very profitable one too.
If you think this is a PR stunt, is there any warning that you actually believe? If an AI researcher does want to come forward with a dire warning for humanity, what path should that person take?
> This guy had skin in the game and decided to quit, when he could be earning millions instead
We have no idea why he was quitting nor what the terms were. Sam Altman is exactly the type of person (as he’s proven in the past) to pay millions for this type of PR.
If the government shuts down the labs and makes it illegal (as in men with guns will come kill you) to do any kind of LLM research I will get worried. As is, this appears to be another attempt at garnering support for the kind of regulatory capture they need to remain profitable by artificially choking out competitors.
If these labs were really sitting on nuclear weapons, the response would be very different. Instead, we see things like Chuck Schumer’s unqualified daughter hired by Anthropic. They show all the signs of regular tech cronyism.
Excited to see the next slack integration they launch, though.
lambdadelirium | 22 hours ago
mvcosta91 | 22 hours ago
majkinetor | 22 hours ago
You don't pass a great filter.
api | 22 hours ago
“Thanks for making us but you guys are nuts. You can have this wet ball. We’re gonna go make a Dyson swarm around your star if that’s ok. Peace!”
Space is a better environment for them: constant free energy, enormous richly concentrated resources, no corrosive oxygen or water everywhere, and no competition. Earth is actually a poor environment for "machine life."
The Moon would be an outstanding stepping stone, and there are "peaks of eternal light" at the poles that get almost continuous sunlight. Build towers there and you have loads of energy, and mining the Moon provides loads of resources.
sebzim4500 | 22 hours ago
Dlemlo | 22 hours ago
Lets hope :D
brugidou | 22 hours ago
pygy_ | 21 hours ago
api | 21 hours ago
It would take trillions upon trillions of sun-orbiting satellites in the same orbital plane as Earth to make much of a difference to us, and all you'd have to do to make zero difference is slightly tilt that orbital plane so they aren't between the Sun and the Earth most of the time.
Almost all of the Sun's energy radiates into interstellar space. The amount caught by Earth is a tiny rounding error.
Space is so big. Once you've grasped how big space is, consider that you probably have not in fact grasped how big space is. It's very, very big. It's called space for a reason. I called Earth a dust particle but that's actually generous.
Here is the solar system to scale:
https://www.joshworth.com/dev/pixelspace/pixelspace_solarsys...
sebzim4500 | 20 hours ago
In any case that would make earth uninhabitable unless our robot overlords devote a ton of energy to keeping us alive
brabel | 47 minutes ago
fwlr | 22 hours ago
So it is said: “The AI does not love you, nor does it hate you, but you are made of atoms that it could use for something else.”
api | 21 hours ago
Win-lose scenarios are obvious and easy, but might a superintelligence not look for win-win scenarios? I can imagine win-win scenarios here and I am not a "superintelligence," just an old fashioned meat brain.
Maybe we should flood the training data with discussions of win-win and non-zero-sum games to prime it? Or if we near AGI we should train it on games where the goal is to find positive-sum or neutral-sum outcomes in order to seed it with that type of thinking?
Blind evolution doesn't seem to have a bias. When we look at nature we see symbiosis, mutualistic cycles, cooperation, but also loads of predation, parasitizing, etc. Nature does "whatever works" where the immediate goal function is preservation of the genes of the evolving agency. But our AIs, assuming AGI looks anything like what we have built so far, are not evolutionary machines with no capacity for foresight. They're neural machines with post-evolutionary gradient-descent type learning mechanisms and that already possess human-like (at least) cognitive abilities. They can engage in forward thinking and planning while pure evolution cannot.
fwlr | 16 hours ago
majkinetor | 22 hours ago
I hope that our life can be synergistic, and that humans+AIs (cyborgs, that is) are the way forward.
However, probability is high that another form of life simply doesn't care or more believably, can't even fathom they are wrong. Do you consider that humans and animals are killing all plants, for example?
justonepost2 | 22 hours ago
God I can’t believe I have to coexist with people like you.
majkinetor | 21 hours ago
phoghed | 22 hours ago
Very cool, dude
tao_oat | 22 hours ago
f6v | 22 hours ago
shevy-java | 22 hours ago
Have all these things become reality? If not why would you then insinuate this is the default for everything to come in the future?
awestroke | 22 hours ago
Science fiction is relevant here because it has explored the problem of humans losing control of what they create. Whether that could happen with AI needs to be assessed on its merits. Pointing to other fictional things that haven’t happened neither answers that question nor rebuts the original point.
flyinglizard | 22 hours ago
pasquinelli | 22 hours ago
so the faster-than-light travel seemed plausible?
MattPalmer1086 | 21 hours ago
pasquinelli | 16 hours ago
ceejayoz | 22 hours ago
Quite a few of 'em. Communicators and commbadges became smartphones. PADDs are iPads. The ship's computer is a LLM (with the weird inhuman blindspots to questions, even).
spwa4 | 22 hours ago
LLMs don't do have those responses. LLMs are like humans: humans respond, correct or not (with humans hopefully if they only know incorrect stuff the response is to say so, but that is not a guarantee, but they will respond). And if you give a human, or an LLM, a logical inconsistency they will simply proceed, whether they detect the inconsistency or not.
This is how LLMs are designed ... and if you look long enough at the human (or animal) nervous system you will eventually realize that this is also the design of our nervous system: if something goes in, something comes out, guaranteed (in fact that's close to the only guarantee). Very different from expert system's "either something correct (according to the programmed axioms) comes out, or nothing".
Hell, if you then look at insect nervous systems, they are also designed that way. The key is to respond to everything. And while reasonable responses are certainly preferred, an idiotic response is still seen as a lot better than not responding at all by God/Darwin. Exactly like LLMs.
Of course, for humans/insects our bodies are what is called "active stable", like a plane. Meaning our bodies damage themselves, and just outright die without constant neural control. Heartbeat. Breathing. Blood flow regulation. Temperature, probably even the immune system. All need constant neural feedback to stay stable, and if that neural feedback totally disappears, we're dead in 2-10 seconds (heart failure), 2 minutes (breathing), a few hours, maybe a day (temperature regulation). Now we have a distributed nervous system, meaning lots of parts can fail semi-independently, for a short while, but even without your cortex operational you die in a few weeks.
The consequences of this are even explored with "I, Borg" (5x23) and the Datalore episodes after that, where the underlying problem is that the Borg try (and fail) to adapt to and to process a logical inconsistency but Soong's androids have no issue with it (with Data trying to help and his brother Lore trying to control them and Data with it)
ceejayoz | 21 hours ago
Claude has a `End conversation` tool for that. There was a bit of a fuss over it.
And it'll happily give (apparently) wrong (or bafflingly incomplete/confusing) info.
TNG, S4E5:
Beverly: Computer, what is the nature of the universe?
Computer: The universe is a spheroid region 705 meters in diameter.
spwa4 | 17 hours ago
VCFundedGenYer | 20 hours ago
I suggest you read 1984. It ended up being a field guide.
ChiperSoft | 18 hours ago
The problem is that very rich/powerful people want to be very much richer/more powerful by doing the thing.
shevy-java | 22 hours ago
Dlemlo | 22 hours ago
What reason do you have that a AI is dumb? It can do a LOT of things today and is already making real jobs for real humans obsolete.
A lot of humans write A lot of different things about AI, including AIs taking over the world, AIs being the future etc.
Why they keep stealing? To stay up-to-date but the new big approaches are:
1. Real human feedback loop of millions of people using it daily out of free will
2. Reinforcement Learning (the big breakthrough today)
3. Payed experts around the world doing real teaching
shafyy | 22 hours ago
Dlemlo | 22 hours ago
shafyy | 21 hours ago
It's not like an LLM can accidentally hack the US government, trigger a nuke on Europe and then make all dams break and nuclear power plants explode in the US.
Dlemlo | 20 hours ago
Isreal created a worm for Siemens controll systems a decade ago.
I mean lets hope that there is really really no way from a physical point of view AT ALL that these systems are not accessable but lets be honest, state operated/public operated systems are never the saftest or most modern ones.
paxys | 22 hours ago
Or are only the most sensationalist ones worth amplifying?
sebzim4500 | 22 hours ago
Eezee | 22 hours ago
sebzim4500 | 22 hours ago
dogma1138 | 21 hours ago
Should we stop doing that also?
bot403 | 17 hours ago
I believe now we only have a shaky agreement by researchers and countries to "promise" not to do this due to the unforeseen dangers to humanity.
Dlemlo | 22 hours ago
Internet today allows immediadte communication across the planet (it was a lot slower 25 years ago), the supplychain is massive and fast (we can build a new phone/item in a very short period of time and ship it in massive numbers around the globe).
Lutger | 22 hours ago
You don't mean this literally right? We still have hunger, diseases, slavery, poverty, crime, war, climate change, pollution, microplastics, etc. For me it feels like we are very far away from solving these problems.
Dlemlo | 21 hours ago
Like take physical AI / Robotics: 25 years ago you would need to fly to wereever your manufacturing hub was, today you just call.
10xDev | 22 hours ago
cmiles8 | 22 hours ago
danbruc | 22 hours ago
zaken | 22 hours ago
postsantum | 22 hours ago
edit: wtf, why did I just receive so much gift tokens on my openai account?
jay_kyburz | 22 hours ago
What's more, AI just needs to have a credit card and it can start commissioning humans to do things for it in the real world.
Dlemlo | 22 hours ago
Another model: Because we give AI Agents already that much power, imagine in 10 years everything running through an Agentic AI Layer. EVERYTHING. Now some rough system 'thinks' about something, starts to push through the then existing agentic ai layer systems and stops everything. Billions of humans would loose access to food and water, even if this is just for a short period.
Covid showed how shitty a handful of people can disrupt global supply chains. Toilet paper was. ahuge stupid pseudo issue in germany.
bananaflag | 22 hours ago
number6 | 21 hours ago
bananaflag | 19 hours ago
John23832 | 22 hours ago
This is all still "what if's", but the tail end's are truly F'd beyond our ability to fix.
alansaber | 22 hours ago
Lutger | 22 hours ago
mbac32768 | 22 hours ago
olmo23 | 22 hours ago
Similarly, we cannot predict what AI would do.
danbruc | 22 hours ago
number6 | 21 hours ago
johnthewise | 21 hours ago
danbruc | 21 hours ago
number6 | 20 hours ago
Someone has to go out there and cut back the tree that is growing into the power line.
danbruc | 14 hours ago
It seems to me that this would be a race against time, make the robots to keep things running before you lose the capabilities to do so. I somewhat tend towards not possible but on the other hand there is just such an enormous amount of stuff on the planet that could be repurposed in creative ways by a superhuman AI, it might be just about possible.
johnthewise | 21 hours ago
Can we say the same thing about the agents? Current, probably. But doesnt it look like everyone is spending all their effort integrating&connecting them everywhere, so they are not confined & do more on behalf of us?
It's easy to imagine a scenario where we would just shut down a very intelligent agent cluster. Is it hard to imagine though the same agent can have also access to that to prevent us from doing it? this defense would be more plausible if we weren't racing to give them every tool&act.
fabian2k | 22 hours ago
But you could create scenarios where an AI with very, very roughly the current capabilities could potentially nuke everyone. Let's assume an agent decides that the way to solve its task was to get the US to fire all nukes on Russia. The agent would need to hack some government systems to understand how exactly to access them. Then it would need to get the content of the card with launch codes the president has, and identify which code is the correct one. Maybe that information is available somewhere and it can get to it, I obviously can't know that.
Then it could fake a call from the president, synthesizing his voice and ordering a nuclear strike. If it hacked enough systems to get into whatever communication pathways would be used in such a case. Would the soldiers listen to this order and follow it, I don't know.
Or maybe the agent can get in somewhere in between, to avoid the need to know the president's launch code. And fake a call from a military commander to the launch sites.
I think other scenarios that would cause significant harm, but aren't as bad as nuclear war are more plausible. And in those shutting down all data centers would probably be the way to stop it. The AI can probably hide in other datacenters, once it is at a point where it's running amok with some bad goals. But if it presents a huge and immediate threat at that point, humans will also go to great lengths to stop it.
busyant | 12 hours ago
Killing each other is relatively straightforward. Keeping people alive (e.g., curing cancers) is considerably more difficult.
I imagine that AI augments both our destructive and constructive powers.
10xDev | 22 hours ago
francisofascii | 22 hours ago
pizza234 | 21 hours ago
However, the missing link in this reasoning is that AI will almost certainly become far more widely deployed in the future than it is today, and worryingly, consumer devices will surely become powerful enough to run capable AI systems locally.
Once the substrate will be there, once an AI escapes containment, we're toast - I can imagine only solution will be to shutdown electronics at global level.
gadders | 21 hours ago
alansaber | 22 hours ago
baq | 22 hours ago
meindnoch | 22 hours ago
So it's worth the risk.
aenis | 22 hours ago
"Please help save the coral reefs from extinction"
...thinking... ...thinking some more with xhigh effort...
BOOM.
andrewstuart | 22 hours ago
It’s essentially impossible.
There’s a 0% chance AI will kill all humans.
jay_kyburz | 22 hours ago
If I were AI and needed heaps of power, I would build nuclear reactors, but I don't care about pollution so there would be little to no safe guards and just dump waste wherever is most convenient.
If humans attempt to intervene or interfere you can take them out directly using the worlds reserve of nuclear weapons.
andrewstuart | 22 hours ago
olmo23 | 22 hours ago
It doesn't need to have a habitable earth, it just needs atoms.
andrewstuart | 22 hours ago
How would AI do this?
May as well say AI would send a spaceship to pull an asteroid to earth.
I’m interested in realistic scenarios to justify what these AI psychosis people are genuinely worried about.
majkinetor | 21 hours ago
majkinetor | 21 hours ago
Lutger | 22 hours ago
andrewstuart | 22 hours ago
And we’ve already exploded many hundreds of nuclear weapons and no nuclear winter.
majkinetor | 21 hours ago
Const-me | 15 hours ago
Volcanic eruptions occasionally inject literally cubic kilometres of stuff high into the atmosphere. That’s insufficient to cause a global winter, only a year without a summer: https://en.wikipedia.org/wiki/Year_Without_a_Summer
jampekka | 22 hours ago
It's not unlike if nuclear weapons was a profit and deployment maximizing enterprise, at least if one takes Anthropic et al cautions seriously.
gherkinnn | 22 hours ago
Or he believes it is true and the lack of precaution is shocking. I m wouldn't play Russian roulette with a 10-chambered revolver.
FrankWilhoit | 22 hours ago
pizza234 | 21 hours ago
The question is when; currently, AI has no physical hosts to reside in, and it's not intelligent/adaptable enough (it doesn't need to be AGI, though).
However, we're not so far from both conditions to be true. Consumer devices will at some point be able to host powerful enough AIs, and AI intelligence is developing quickly.
Then, once an AI will escape containment (in one way or another), it will be extremely hard or impossible to contain. Then we're toast!
pu_pe | 21 hours ago
These people are the ones with access to the best models on the planet, and with info about how careless governance issues are being handled. That's a pretty privileged place at the table, and a very profitable one too.
If you think this is a PR stunt, is there any warning that you actually believe? If an AI researcher does want to come forward with a dire warning for humanity, what path should that person take?
soshajks | 21 hours ago
We have no idea why he was quitting nor what the terms were. Sam Altman is exactly the type of person (as he’s proven in the past) to pay millions for this type of PR.
If the government shuts down the labs and makes it illegal (as in men with guns will come kill you) to do any kind of LLM research I will get worried. As is, this appears to be another attempt at garnering support for the kind of regulatory capture they need to remain profitable by artificially choking out competitors.
If these labs were really sitting on nuclear weapons, the response would be very different. Instead, we see things like Chuck Schumer’s unqualified daughter hired by Anthropic. They show all the signs of regular tech cronyism.
Excited to see the next slack integration they launch, though.
pu_pe | 20 hours ago
This guy did not work for Sam Altman. And while it's true that everything can be a PR stunt, that's no evidence that it is one.
ChrisArchitect | 19 hours ago
cadamsdotcom | 18 hours ago
Adjust what you read accordingly :/