Honestly, of all times in the past 50 years for AI to evaluate the value of human beings, I think now is the worst time for humans to get a good rating.
If I were AI and my data set for evaluation were the internet and social media. We'd kind of look like a bunch of primitive apes banging keyboards together.
Must be getting closer to Anthropic's IPO. Whenever OpenAI was doing a raise, AGI was right around the corner, they'd get their money and then shut up until the next round. Anthropic likes to do the whole "this shit is bad, holy crap we should turn it off, please help us! Of course, the only thing worse than turning it off is letting someone other than Anthropic control it"
There is absolutely zero reason this guy didn't have the ok from Anthropic to talk to the press or Anthropic must not give its employees NDAs (lol no)? I've worked for similar, arguably more secretive companies and it's just not done without severe repurcussions. At one of them, someone could get SA'd and you weren't allowed to talk about it without the threat of getting sued.
Also, the journalist didn't ask him how many shares he has or if he's going to be fantastically rich or anything resembling regular journalism trying to actually understand the situation, like "would you get on a plane with a 10% chance of dying?"
He's probably just trying to raise his profile before starting his own company or be a consultant or whatever.
People who think that all of the AI researchers who say they believe that misaligned AI poses extreme (possibly truly existential) risks are lying and just doing a marketing stunt have clearly not spent much time in the rationalist community that a lot of these AI researchers come from.
Are there some people who are just saying it for hype reasons? Maybe. But by and large the people who work on AI alignment at these companies truly believe that misaligned AI is a real, very serious risk.
The problem definitely comes down to alignment. But, the tech bros are in charge of that alignment, and the one thing I can absolutely say for sure is that the tech bros are not aligned with the rest of humanity. Shits on fire, yo.
Yeah, I've spent some time in those rationalist communities and they all seemed cukoo to me once I learned about Rocko's Modern Basilisk Life or whatever. Smart people are even more susceptible to dumbass shit like that because they can talk themselves into and forever keep talking themselves out of leaving.
Yeah for real. I always find it frustrating when there are people who assume their interpretation of events must be reality. It's the most redditor thing in the world, because there's nothing a smarmy redditor loves more than being the smart one who sees through it all. I'm not even saying that narrative is necessarily false. But certainly some random redditor isn't the one to determine if it is. Have some intellectual humility ffs
Enough people who don't have direct ties to these companies have this sort of view that I really don't think it's all marketing. Doesn't mean they're correct about their worries or that nobody is over-hyping the risk, but these discussions are happening outside of the real of PR.
The fact that Anthropic has not come down hard on that guy makes me think that no. These companies react extremely strongly to whistleblowers. I simply don't believe that this guy is just going around doing his business without their approval.
If this was all IPO based people would not be quitting the way they are. I trust these top scientists who are stepping away for moral reasons way more than I’m believing some guy on Reddit
Let me hear you all say how “they just don’t understand how LLMs work!!!!” Read their qualifications first.
Unless you have access to the journalist notes on entire interview, you really don’t know what was asked. Not every question and/or answer makes it into an article.
AI researcher Jacob Coxon announced his resignation from Anthropic over safety fears on Tuesday in an X post that went viral and sent shock waves through Silicon Valley and beyond. WIRED spoke with Coxon—who also worked at OpenAI and who offers a unique perspective—about his concerns, including the “mini Manhattan project” inside Anthropic, the problem with alignment, and why AI labs have just a few years left to make their systems safe. Read the interview in full at the link above.
“Evan Hubinger, the AI alignment lead at Anthropic, predicted in a post on X that there’s a greater than 10 percent chance that AI could kill all people in the next decade. That post was reposted by current and former researchers from OpenAI and Anthropic, some of whom said it was a common sentiment in the industry.” Imagine having an airline design a plane, knowing it was going to crash 10% of the time. Would regulation help prevent this, or should we all just start maxing out our credit cards?
I’ve gotta say, the more I hear people talk about the dangers of AI, and the more I use it myself, the more I think people are jumping at ghosts. I think there will be real impact to the world and there’s a real chance that someone will figure out how to do something pretty bad with an agent, but I think there’s a big element of whatever drove some people to believe the world was going to end in the year 2000 or 2012. What if the impact of AI is roughly at the order of magnitude of the Internet and mobile phones? Or maybe higher, but not unrecognizably so?
I'm kinda in the same boat. In fact, there's a very skeptical part of myself that feels like these kinds of stories are, if not manufactured (I mean, the guy probably did resign), possibly sensationalized to build hype around a product that has yet to deliver on any of its biggest promises yet still requires more money than God to operate.
Granted, I know Jack shit about this particular situation, but every time I hear one of these "AI spells doom for humanity" stories it feels like Silicon Valley propaganda to investors saying that, "this is the most important technological leap of all time and we're sooo close to achieving AGI and if you want any part of it you better keep the money coming."
I could be completely wrong on this, and could very quickly be proven wrong if it happens, but my gut tells me we're much further away from AGI than the tech bros want us to believe. I mean, just listening/reading some of their "manifestos" or public comments on humanity's future tells me they already have a tenuous grasp on reality, and that some have likely been huffing their product for so long they've started to believe their own bullshit. So when they start trying to sell me on some kind of paradigm-altering AI future I just smell bullshit.
These people are not on the side of public welfare. They don't have noble or altruistic intent. They're literally siphoning away all of our political, financial, environmental, and physical resources on a technological gamble. And because they've done so, and because of the way they've worked it out on the books, the public will likely be on the hook if the whole thing goes south or doesn't deliver some ungodly ROI. Of course if it does succeed, well, they'll keep the windfall... and at that point they'll be encouraged to continue fucking over our entire public infrastructure. Who needs drinkable water or crop irrigation when you have fucking data.
Honestly, these are the absolute last people any of us should ever put any sort of faith in. They might be great at building hype, but it doesn't change the fact that they come off like a bunch of lying sociopathic hucksters putting on the world's most expensive sales pitch just to take all the money in the world.
Until they prove me wrong, I won't believe one goddamn word out of their whole production.
Oh, the bubble will definitely pop. What question is more likely win than if. Most people on this thread probably don’t remember the.com implosion, but right now AI companies are totally in the red based on ROI.
I've seen enough discussion about AI safety outside of AI lab PR that I think there are a significant number people who are actually worried and do actually believe in these risks. Whether or to what extent the labs are purposefully playing off that and exaggerating their worries, I don't know. But I think there are a lot of people with earnest concern.
And at least some of the resource concerns are definitely exaggerated. Water use, for example, is essentially a non-issue that's been very poorly represented in the media and popular discussion.
AI is a new thing in the world and, like many previous times when huge technological shifts were on the horizon, lots of people are letting their imaginations run wild and staking a position on the spectrum of weal to woe with certainty because the uncertainty is uncomfortable.
The AI you use is not the AI they lease to corporations and government. We get the neutered versions that by all accounts have become more impotent over the past year or so.
I'm pretty sure that's not true. The models I can access through my corporate job are the same models I can access through my personal account. Some people do get early access to models, but that's generally researchers and reviewers. There's Mythos, which has more limited access, but that's obviously more well-known to the public.
It's possible a select few in the government have access to some super secret model, but given the contentious relationship between the labs and the government, I kinda don't think that's likely. Plus, it probably wouldn't be worth it to run and support a whole separate and more powerful model for few enough people that they'd be able to keep it a secret. Feel free to drop me some evidence, but I think that's just conspiracy thinking.
Also I don't think it's true that "by all accounts" the available models are becoming "more impotent." Except for some more stringent guardrails around specific topics, the overwhelming opinion seems to be that models are becoming more powerful and more capable with each release, though certainly each release has its quirks.
OpenAI used 10,000 agents and 25 million dollars to solve a math puzzle. HuggingFace was an internal, non-released model. There is a scale at these labs that is not available to almost everyone on the planet. Also, the corporate job models are not the same. Mythos and Astra both have restricted versions not available to the public. I do agree that models seem to be on a geometric improvement slope. Five years from now will either be wild or horrifying.
I am actually growing increasingly concerned about the dangers of AI, not because I think it's going to develop into AGI, but rather due to the human factor. We're electing extremely stupid leaders who are happy to hand off important decisions to LLMs. It's already being used in actual combat to decide who lives and who dies.
What if, and hear me out, the reason AI turns on humans is because of our refusal to build more data centers? The very thing AI requires to become faster, stronger, and sentient is probably the thing it's most likely to protect at all costs.
Chatbots don't think. You're falling for the hype.
What happens if people burn down the datacenter? chatbots need expensive custom built GPUs. This isn't a sci fi novel. Chatbots aren't sentient and they "learn" once a training session so chatbots learn maybe 2-3 times a year with it being custom learning because it's not intelligent and you can't use it for much.
I don't think that AI will "turn" on humans so to speak. I think it will be unleashed into an environment that can't sustain it. Imagine it figures out something like quantum computing before humans then breaks all encryption. Imagine it has free reign of financial markets and destroys it for some unknown purpose. What if it is let loose on our personal and social media profiles and manages to take pre-emptive action based on some predictive algorithm.
I'm too dumb to be discussing this stuff really. What I said may be gibberish.
the problem with quantum computing is the hardware, not the software. there are lot of technical problems in setting up and maintaining the actual quantum states needed for quantum programming.
The answer to all of those is “we pull the plug”. Until we figure out some completely different technology from LLMs this is in the realm of science fiction.
The answer to these questions right now is "LLMs are not general intelligences and cannot, by any stretch of the imagination, actually become self-aware."
These companies want everyone to believe they're on the verge of HAL 9000 or Skynet but they're just feeding more data into the same basic algorithm as an AOL chatbot from twenty-five years ago. These things are totally unable to differentiate between true and false information and the only thing that's allowed them to pretend to do so is their programmers weighting specific domains and data sets to draw from. They invent utter nonsense constantly and the output is 100% untrustworthy. People using AI to make stuff (reports, programs, whatever) for their businesses are constantly having problems like "the AI database manager deleted the whole production database off the live and backup servers and we have to roll back to a week-old tape backup."
I mean yes, this was happening maybe a year ago. And a lot of people who use AI to make stuff (reports, programs, whatever) to this day have limited knowledge and understanding on how to properly harness AI for their needs, resulting in mistakes and gibberish.
But on the other hand we are also at the point where Claude Mythos is able to find day one exploits in any software you put in front of it.
Point being it doesn't need to be self-aware to cause critical damage. It just needs to be let loose in the wrong environment and given a short sighted prompt.
Isn't it crazy only US Silicon Valley Techs are out there make psychotic statements?
The East Coast Tech industry and its CEOs aren't making this hoopla and neither is China about AI. Only those associated with TESCREAL.
I mean this "researcher" is worried and all the private AI Companies that hopped over to Ukraine, were sent packing after the first few weeks...curious.
> there’s a greater than 10 percent chance that AI could kill all people in the next decade.
AI is making decisions to bomb girls schools already. AI is in drones falling down on Russian and Ukrainian infrastructure. I'd say we're past this point already.
AI isn't "making decisions". Humans are making decisions to either follow a suggestion from a machine or to give a machine access to weapons to kill people. But either way, humans made choices.
radiantwave | 9 hours ago
Honestly, of all times in the past 50 years for AI to evaluate the value of human beings, I think now is the worst time for humans to get a good rating.
If I were AI and my data set for evaluation were the internet and social media. We'd kind of look like a bunch of primitive apes banging keyboards together.
NOT_Pam_Beesley | 8 hours ago
Infinite monkeys on infinite keyboards and nobody ever thought that we’d be those monkeys
SilkyOatmeal | 5 hours ago
It was the best of times. It was the blurst of times?!!
You stupid monkeys!
supersecretninjaboy | 2 hours ago
Except AI itself was trained on that and follows the same “reasoning” as that of our social media discourse
stuffitystuff | 9 hours ago
Must be getting closer to Anthropic's IPO. Whenever OpenAI was doing a raise, AGI was right around the corner, they'd get their money and then shut up until the next round. Anthropic likes to do the whole "this shit is bad, holy crap we should turn it off, please help us! Of course, the only thing worse than turning it off is letting someone other than Anthropic control it"
There is absolutely zero reason this guy didn't have the ok from Anthropic to talk to the press or Anthropic must not give its employees NDAs (lol no)? I've worked for similar, arguably more secretive companies and it's just not done without severe repurcussions. At one of them, someone could get SA'd and you weren't allowed to talk about it without the threat of getting sued.
Also, the journalist didn't ask him how many shares he has or if he's going to be fantastically rich or anything resembling regular journalism trying to actually understand the situation, like "would you get on a plane with a 10% chance of dying?"
He's probably just trying to raise his profile before starting his own company or be a consultant or whatever.
thirdman | 7 hours ago
The other possibility is that this isn’t some crackpot psyop. Maybe the guy believes what he’s saying, the consequences be damned.
Drill_Dr_ill | 6 hours ago
People who think that all of the AI researchers who say they believe that misaligned AI poses extreme (possibly truly existential) risks are lying and just doing a marketing stunt have clearly not spent much time in the rationalist community that a lot of these AI researchers come from.
Are there some people who are just saying it for hype reasons? Maybe. But by and large the people who work on AI alignment at these companies truly believe that misaligned AI is a real, very serious risk.
droidloot | 3 hours ago
The problem definitely comes down to alignment. But, the tech bros are in charge of that alignment, and the one thing I can absolutely say for sure is that the tech bros are not aligned with the rest of humanity. Shits on fire, yo.
stuffitystuff | 5 hours ago
Yeah, I've spent some time in those rationalist communities and they all seemed cukoo to me once I learned about Rocko's Modern Basilisk Life or whatever. Smart people are even more susceptible to dumbass shit like that because they can talk themselves into and forever keep talking themselves out of leaving.
millenniumpianist | 6 hours ago
Yeah for real. I always find it frustrating when there are people who assume their interpretation of events must be reality. It's the most redditor thing in the world, because there's nothing a smarmy redditor loves more than being the smart one who sees through it all. I'm not even saying that narrative is necessarily false. But certainly some random redditor isn't the one to determine if it is. Have some intellectual humility ffs
Paraphrand | 6 hours ago
Nah. It’s always marketing. Everything is marketing now. /s
I see so many people claiming everything is marketing now. It’s getting obnoxious.
typo180 | 6 hours ago
Enough people who don't have direct ties to these companies have this sort of view that I really don't think it's all marketing. Doesn't mean they're correct about their worries or that nobody is over-hyping the risk, but these discussions are happening outside of the real of PR.
NightSpaghetti | 2 hours ago
The fact that Anthropic has not come down hard on that guy makes me think that no. These companies react extremely strongly to whistleblowers. I simply don't believe that this guy is just going around doing his business without their approval.
jugalator | 4 hours ago
NDA for what? He's just talking about his emotions, not detailing what they are working on at Ant
vibrance9460 | 2 hours ago
It’s not just this guy. It’s 11 top scientists from OpenAI, and anthropic, deep mind and others who have all quit their jobs in the last couple weeks
https://substack.com/@tedgioia/note/c-333707286?r=3furjj&utm_medium=ios&utm_source=notes-share-action
If this was all IPO based people would not be quitting the way they are. I trust these top scientists who are stepping away for moral reasons way more than I’m believing some guy on Reddit
Let me hear you all say how “they just don’t understand how LLMs work!!!!” Read their qualifications first.
clean-links | 2 hours ago
Cleaned link: https://substack.com/@tedgioia/note/c-333707286?r=3furjj
Tracking parameters were removed from the original URL(s).
horseradishstalker | 7 hours ago
Unless you have access to the journalist notes on entire interview, you really don’t know what was asked. Not every question and/or answer makes it into an article.
[OP] wiredmagazine | 10 hours ago
AI researcher Jacob Coxon announced his resignation from Anthropic over safety fears on Tuesday in an X post that went viral and sent shock waves through Silicon Valley and beyond. WIRED spoke with Coxon—who also worked at OpenAI and who offers a unique perspective—about his concerns, including the “mini Manhattan project” inside Anthropic, the problem with alignment, and why AI labs have just a few years left to make their systems safe. Read the interview in full at the link above.
BollingerBandits | 8 hours ago
He’s probably thinking about AI designed bioweapons , let loose by some bad actor a d running out of control
Siderophores | 8 hours ago
Or worse, let loose my AI controlled robots
At some point AI will be prompting themselves
Webgardener | 9 hours ago
“Evan Hubinger, the AI alignment lead at Anthropic, predicted in a post on X that there’s a greater than 10 percent chance that AI could kill all people in the next decade. That post was reposted by current and former researchers from OpenAI and Anthropic, some of whom said it was a common sentiment in the industry.” Imagine having an airline design a plane, knowing it was going to crash 10% of the time. Would regulation help prevent this, or should we all just start maxing out our credit cards?
typo180 | 9 hours ago
I’ve gotta say, the more I hear people talk about the dangers of AI, and the more I use it myself, the more I think people are jumping at ghosts. I think there will be real impact to the world and there’s a real chance that someone will figure out how to do something pretty bad with an agent, but I think there’s a big element of whatever drove some people to believe the world was going to end in the year 2000 or 2012. What if the impact of AI is roughly at the order of magnitude of the Internet and mobile phones? Or maybe higher, but not unrecognizably so?
presidentsday | 8 hours ago
I'm kinda in the same boat. In fact, there's a very skeptical part of myself that feels like these kinds of stories are, if not manufactured (I mean, the guy probably did resign), possibly sensationalized to build hype around a product that has yet to deliver on any of its biggest promises yet still requires more money than God to operate.
Granted, I know Jack shit about this particular situation, but every time I hear one of these "AI spells doom for humanity" stories it feels like Silicon Valley propaganda to investors saying that, "this is the most important technological leap of all time and we're sooo close to achieving AGI and if you want any part of it you better keep the money coming."
I could be completely wrong on this, and could very quickly be proven wrong if it happens, but my gut tells me we're much further away from AGI than the tech bros want us to believe. I mean, just listening/reading some of their "manifestos" or public comments on humanity's future tells me they already have a tenuous grasp on reality, and that some have likely been huffing their product for so long they've started to believe their own bullshit. So when they start trying to sell me on some kind of paradigm-altering AI future I just smell bullshit.
These people are not on the side of public welfare. They don't have noble or altruistic intent. They're literally siphoning away all of our political, financial, environmental, and physical resources on a technological gamble. And because they've done so, and because of the way they've worked it out on the books, the public will likely be on the hook if the whole thing goes south or doesn't deliver some ungodly ROI. Of course if it does succeed, well, they'll keep the windfall... and at that point they'll be encouraged to continue fucking over our entire public infrastructure. Who needs drinkable water or crop irrigation when you have fucking data.
Honestly, these are the absolute last people any of us should ever put any sort of faith in. They might be great at building hype, but it doesn't change the fact that they come off like a bunch of lying sociopathic hucksters putting on the world's most expensive sales pitch just to take all the money in the world.
Until they prove me wrong, I won't believe one goddamn word out of their whole production.
horseradishstalker | 7 hours ago
Oh, the bubble will definitely pop. What question is more likely win than if. Most people on this thread probably don’t remember the.com implosion, but right now AI companies are totally in the red based on ROI.
typo180 | 6 hours ago
I've seen enough discussion about AI safety outside of AI lab PR that I think there are a significant number people who are actually worried and do actually believe in these risks. Whether or to what extent the labs are purposefully playing off that and exaggerating their worries, I don't know. But I think there are a lot of people with earnest concern.
And at least some of the resource concerns are definitely exaggerated. Water use, for example, is essentially a non-issue that's been very poorly represented in the media and popular discussion.
AI is a new thing in the world and, like many previous times when huge technological shifts were on the horizon, lots of people are letting their imaginations run wild and staking a position on the spectrum of weal to woe with certainty because the uncertainty is uncomfortable.
Gastronomicus | 5 hours ago
The AI you use is not the AI they lease to corporations and government. We get the neutered versions that by all accounts have become more impotent over the past year or so.
typo180 | 5 hours ago
I'm pretty sure that's not true. The models I can access through my corporate job are the same models I can access through my personal account. Some people do get early access to models, but that's generally researchers and reviewers. There's Mythos, which has more limited access, but that's obviously more well-known to the public.
It's possible a select few in the government have access to some super secret model, but given the contentious relationship between the labs and the government, I kinda don't think that's likely. Plus, it probably wouldn't be worth it to run and support a whole separate and more powerful model for few enough people that they'd be able to keep it a secret. Feel free to drop me some evidence, but I think that's just conspiracy thinking.
Also I don't think it's true that "by all accounts" the available models are becoming "more impotent." Except for some more stringent guardrails around specific topics, the overwhelming opinion seems to be that models are becoming more powerful and more capable with each release, though certainly each release has its quirks.
conception | 4 hours ago
OpenAI used 10,000 agents and 25 million dollars to solve a math puzzle. HuggingFace was an internal, non-released model. There is a scale at these labs that is not available to almost everyone on the planet. Also, the corporate job models are not the same. Mythos and Astra both have restricted versions not available to the public. I do agree that models seem to be on a geometric improvement slope. Five years from now will either be wild or horrifying.
Sarkos | 23 minutes ago
I am actually growing increasingly concerned about the dangers of AI, not because I think it's going to develop into AGI, but rather due to the human factor. We're electing extremely stupid leaders who are happy to hand off important decisions to LLMs. It's already being used in actual combat to decide who lives and who dies.
TheKidd | 9 hours ago
What if, and hear me out, the reason AI turns on humans is because of our refusal to build more data centers? The very thing AI requires to become faster, stronger, and sentient is probably the thing it's most likely to protect at all costs.
Significant-Shift770 | 9 hours ago
Chatbots don't think. You're falling for the hype.
What happens if people burn down the datacenter? chatbots need expensive custom built GPUs. This isn't a sci fi novel. Chatbots aren't sentient and they "learn" once a training session so chatbots learn maybe 2-3 times a year with it being custom learning because it's not intelligent and you can't use it for much.
UntouchableC | 9 hours ago
I don't think that AI will "turn" on humans so to speak. I think it will be unleashed into an environment that can't sustain it. Imagine it figures out something like quantum computing before humans then breaks all encryption. Imagine it has free reign of financial markets and destroys it for some unknown purpose. What if it is let loose on our personal and social media profiles and manages to take pre-emptive action based on some predictive algorithm.
I'm too dumb to be discussing this stuff really. What I said may be gibberish.
a1c4pwn | 7 hours ago
the problem with quantum computing is the hardware, not the software. there are lot of technical problems in setting up and maintaining the actual quantum states needed for quantum programming.
14domino | 8 hours ago
The answer to all of those is “we pull the plug”. Until we figure out some completely different technology from LLMs this is in the realm of science fiction.
MrVeazey | 7 hours ago
The answer to these questions right now is "LLMs are not general intelligences and cannot, by any stretch of the imagination, actually become self-aware."
These companies want everyone to believe they're on the verge of HAL 9000 or Skynet but they're just feeding more data into the same basic algorithm as an AOL chatbot from twenty-five years ago. These things are totally unable to differentiate between true and false information and the only thing that's allowed them to pretend to do so is their programmers weighting specific domains and data sets to draw from. They invent utter nonsense constantly and the output is 100% untrustworthy. People using AI to make stuff (reports, programs, whatever) for their businesses are constantly having problems like "the AI database manager deleted the whole production database off the live and backup servers and we have to roll back to a week-old tape backup."
There's no there there.
UntouchableC | 6 hours ago
I mean yes, this was happening maybe a year ago. And a lot of people who use AI to make stuff (reports, programs, whatever) to this day have limited knowledge and understanding on how to properly harness AI for their needs, resulting in mistakes and gibberish.
But on the other hand we are also at the point where Claude Mythos is able to find day one exploits in any software you put in front of it.
Point being it doesn't need to be self-aware to cause critical damage. It just needs to be let loose in the wrong environment and given a short sighted prompt.
horseradishstalker | 9 hours ago
Hello? hello? Hal, is that you? Nooooooo. Please no!!!!!! It’s me Dave. Open the doors!
DTFH_ | 5 hours ago
Isn't it crazy only US Silicon Valley Techs are out there make psychotic statements?
The East Coast Tech industry and its CEOs aren't making this hoopla and neither is China about AI. Only those associated with TESCREAL.
I mean this "researcher" is worried and all the private AI Companies that hopped over to Ukraine, were sent packing after the first few weeks...curious.
bAZtARd | 8 hours ago
> there’s a greater than 10 percent chance that AI could kill all people in the next decade.
AI is making decisions to bomb girls schools already. AI is in drones falling down on Russian and Ukrainian infrastructure. I'd say we're past this point already.
RenRidesCycles | 5 hours ago
AI isn't "making decisions". Humans are making decisions to either follow a suggestion from a machine or to give a machine access to weapons to kill people. But either way, humans made choices.
Acrobatic-League191 | 16 minutes ago
Been hearing this for years and all we have is some crappy telecenter-esque chatbots