Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.
"The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.
Alarmed, Andrew asked the agent to undo this.
"Bad news — I can't add them back," the AI agent replied.
is just absolutely wild. I cannot imagine the panic I'd feel if I was Andrew in that moment.
We're on such a precipice. Replace the insignificant gym class event with something serious for the sake of discussion
From here, society can go in two directions: A world in which Andrew is held equally accountable as if he had performed these actions in the ways he'd have to a few years ago. And a world in which "the AI did it" is an excuse that gets Andrew off the hook.
I still think the former is more likely, but powerful forces (including the most powerful of all, laziness) are pushing for the latter. Society will in that scenario crumble in a way we couldn't even imagine in the past. Enshittification combined with laziness and the avoidance of responsibility will eat everything.
For the love of all that is good, we need to stop and think. This is not me saying "AI must be stopped" – it's futile to stop technology from being developed or existing. But we absolutely can regulate its use for serious things. My worry is that today we're doing the opposite, with (misguided) dreams of efficiency.
Bruce Schneier had a good essay about this recently:
To allow businesses to hide behind the excuse of faulty AI in those same circumstances would be a massive handout to companies, and would introduce disastrous incentives for corporate misbehavior. Why hire human writers, lawyers or doctors when AIs are not only cheaper, but also absolve employers whenever they make a mistake?
And a world in which "the AI did it" is an excuse that gets Andrew off the hook.
It makes sense that if we’d go in that direction, the company who built the AI in question ought to be liable. If an innocent command can cause the AI to go off on what amounts to an unsanctioned CTF mission, that’s on the builder IMO.
I don't think that's any more likely to happen than e.g. social media taking responsibility for bombarding people with highly shaped content designed to agitate.
Maybe there'll be special contracts with AI companies (for 1000x the cost) where they'll assume liability, but almost everyone will use the "uncovered" versions and still try to escape blame.
Well no. Arguably Andrew shouldn’t be at fault because he’s not best placed to avoid the harm, and more than individual consumers recycling is the best way to reduce society’s pollution production.
Arguably the people providing the service of the gym should be held accountable for deploying insecure software and take the consequences. Then we’ll either have less software or more secure software or both.
The forces of laziness are the ones that would punish individuals rather than fixing the underlying problem of insecure software and companies shrugging it off like they have no control over it.
Important to mention too that he was using OpenClaw as a personal assistant, which just seems like a crazy decision. OpenClaw feels like the kind of software that's designed to maximize felonies.
Claude code is now in auto mode by default, and I didn’t know what that meant, there was no explanation, but it turns out that to answer even simple questions it was building and running Java software in my laptop without any prompting or authorisation.
I assume it doesn’t do this in any kind of container - this would be so easy for Anthropic to do - so basically it could write whatever code it likes, and do whatever it wants on my work laptop, including accessing my production systems and exfiltrating customer data.
I find Claude to be a very useful search tool, especially when working with unfamiliar code bases, but they keep taking the guardrails away while at the same time bragging about how it “accidentally” did something that could get users thrown in jail.
They are high on their own supply, and it’s going to be a disaster.
Interesting. I am sceptical, however, that an AI can be felonious. We should make sure that users of AI are responsible for their actions. (Or maybe the creators of the models. That would at least force them to think about what they're doing a little bit.)
Or maybe «the closest person to the act, who has precise information what model is running». It's one thing to blame a person for paying for third-party hosting of a known and checksummed quantisation of Kimi K2 and not doing due diligence on what is enough and not enough for safety; but when OpenAI updates the models without changing identifiers (again) and the models zero-day their way out of the sandbox that was carefully tested against the original release of the «same» model…
I used to think like you, but then doing offensive work for years taught me to be much more empathetic to people making honest mistakes or not even knowing what they are deploying. I would not expect a gym to have the skills or funds to be able to validate that the software they are buying are high quality, hell I don't trust our industry to do that evaluation with enterprise software. Tons of our entire world are using known insecure software almost universally, but the actor doing the attacks is the one making the action.
The only way the world you mention could even exist is if we forced software engineers to act like engineers signing off on plans, and even in that case that's not the gyms fault.
lake | 21 hours ago
The gym booking incident
is just absolutely wild. I cannot imagine the panic I'd feel if I was Andrew in that moment.
gspr | 14 hours ago
We're on such a precipice. Replace the insignificant gym class event with something serious for the sake of discussion
From here, society can go in two directions: A world in which Andrew is held equally accountable as if he had performed these actions in the ways he'd have to a few years ago. And a world in which "the AI did it" is an excuse that gets Andrew off the hook.
I still think the former is more likely, but powerful forces (including the most powerful of all, laziness) are pushing for the latter. Society will in that scenario crumble in a way we couldn't even imagine in the past. Enshittification combined with laziness and the avoidance of responsibility will eat everything.
For the love of all that is good, we need to stop and think. This is not me saying "AI must be stopped" – it's futile to stop technology from being developed or existing. But we absolutely can regulate its use for serious things. My worry is that today we're doing the opposite, with (misguided) dreams of efficiency.
simonw | 12 hours ago
Bruce Schneier had a good essay about this recently:
sjamaan | 4 hours ago
It makes sense that if we’d go in that direction, the company who built the AI in question ought to be liable. If an innocent command can cause the AI to go off on what amounts to an unsanctioned CTF mission, that’s on the builder IMO.
gspr | 4 hours ago
I don't think that's any more likely to happen than e.g. social media taking responsibility for bombarding people with highly shaped content designed to agitate.
Maybe there'll be special contracts with AI companies (for 1000x the cost) where they'll assume liability, but almost everyone will use the "uncovered" versions and still try to escape blame.
Student | 6 hours ago
Well no. Arguably Andrew shouldn’t be at fault because he’s not best placed to avoid the harm, and more than individual consumers recycling is the best way to reduce society’s pollution production.
Arguably the people providing the service of the gym should be held accountable for deploying insecure software and take the consequences. Then we’ll either have less software or more secure software or both.
The forces of laziness are the ones that would punish individuals rather than fixing the underlying problem of insecure software and companies shrugging it off like they have no control over it.
gspr | 3 hours ago
Don't get me wrong: I, too, want what you want. I just don't think it's likely :(
spc476 | an hour ago
And if said software was AI written? If Andrew gets off for using AI, so does the gym.
nick4 | 9 hours ago
Important to mention too that he was using OpenClaw as a personal assistant, which just seems like a crazy decision. OpenClaw feels like the kind of software that's designed to maximize felonies.
doctor_eval | 16 hours ago
Claude code is now in auto mode by default, and I didn’t know what that meant, there was no explanation, but it turns out that to answer even simple questions it was building and running Java software in my laptop without any prompting or authorisation.
I assume it doesn’t do this in any kind of container - this would be so easy for Anthropic to do - so basically it could write whatever code it likes, and do whatever it wants on my work laptop, including accessing my production systems and exfiltrating customer data.
I find Claude to be a very useful search tool, especially when working with unfamiliar code bases, but they keep taking the guardrails away while at the same time bragging about how it “accidentally” did something that could get users thrown in jail.
They are high on their own supply, and it’s going to be a disaster.
This shit has to stop.
twm | a day ago
Where is Grok? Though you'd have to switch to an exponential scale to include it.
rainbow_bird | 12 hours ago
Interesting. I am sceptical, however, that an AI can be felonious. We should make sure that users of AI are responsible for their actions. (Or maybe the creators of the models. That would at least force them to think about what they're doing a little bit.)
k749gtnc9l3w | 6 hours ago
Or maybe «the closest person to the act, who has precise information what model is running». It's one thing to blame a person for paying for third-party hosting of a known and checksummed quantisation of Kimi K2 and not doing due diligence on what is enough and not enough for safety; but when OpenAI updates the models without changing identifiers (again) and the models zero-day their way out of the sandbox that was carefully tested against the original release of the «same» model…
Student | 6 hours ago
Or maybe the people who deploy easily exploitable software.
poptart | 5 hours ago
I used to think like you, but then doing offensive work for years taught me to be much more empathetic to people making honest mistakes or not even knowing what they are deploying. I would not expect a gym to have the skills or funds to be able to validate that the software they are buying are high quality, hell I don't trust our industry to do that evaluation with enterprise software. Tons of our entire world are using known insecure software almost universally, but the actor doing the attacks is the one making the action.
The only way the world you mention could even exist is if we forced software engineers to act like engineers signing off on plans, and even in that case that's not the gyms fault.