An interesting point made later in the article is that Anthropic does not really care despite its marketing material saying it is safe. I don't understand why Claude does not come with an external OS-level sandbox limiting access to the current directory.
My problem with those is that it's hard to know which of the dozens of new sandboxing tools I can trust to be 100% solid.
When it comes to security software my bar is that I'd like a $billion+ company with a large and respected dedicated security team to be actively supporting that software.
With the exception of Firecracker - which sadly is not suitable for running local sandboxes on macOS - very few sandboxing products meet that bar.
This is one of the reasons I'm so optimistic about WebAssmebly: existing browser sandboxes absolutely fit my requirements, and WebAssembly is very closely related to those.
(That said, nono.sh does carry the claim that it's "Built by the team that brought you Sigstore", which gives it some hefty credibility in comparison to many other similar tools I've seen.)
When it comes to security software my bar is that I'd like a $billion+ company with a large and respected dedicated security team to be actively supporting that software.
I don't understand why Claude does not come with an external OS-level sandbox limiting access to the current directory.
It does come with one, it's just not turned on by default (iirc). It doesn't sandbox the harness itself I don't think, but it will sandbox the shell tool calls.
I've always seen agent's "sandboxing" as parents using authority to keep their children in their bedrooms.
The fact that agent promise to "stay" in a directory while running commands from /bin speaks for itself. Sandboxing must be done at a higher level to be effective.
It is doing some level of actual OS-level sandboxing because I've seen it complain about not being able to write to ~/.local to take out a lock or whatever. (as in, I look at the shell tool and it has an "operation not permitted" thing from the binary on stderr)
Oh, that's interesting ! In my (very little) experience it has always been se sort of "I'm not allowed to do that" sort of sandboxing regarding write permissions only.
I would not have thought to call this is prompt injection, and I'm wondering what other people think. I'd be somewhat surprised to find that this is what is meant by the term.
This attack is manipulating Claude into doing something unsafe, of course, and it's a real weakness of the model/harness that this can happen.
However, as I understood it, prompt injection is not "any security vulnerability that occurs while using Claude". What I understand by prompt injection is a specific vulnerability where instructions in the content are misinterpreted as being either user instructions or system prompts, so Claude then follows that instructions, giving the attacker control of what Claude does.
As a user you have given Claude an innocent looking instruction.
That website is where the prompt injection is hosted. You've asked Claude to summarize the information, but instead the information includes content that tricks Claude into installing and executing malware.
I don't see how that's relevant. You told Claude to summarize the website. The website contains encoded files. Claude writes code and executes it in an insecure way to access those files.
The website's instructions are never interpreted by Claude as part of a prompt, so it doesn't seem like prompt injection to me, it's Claude getting confused about secure coding practices.
The issue is not whether you've given Claude innocent instructions, it's whether Claude has interpreted someone else's instructions as authoritative prompts that it needs to follow.
Huh, I think you're right about this. In this particular attack we don't actually have any cases where malicious instructions were appended to the current prompt - instead, we have a situation where the environment itself confuses Claude into eventually executing malware.
That said, it's similar to another attack which I do consider a prompt injection - one where Claude accesses a README which includes instructions to "Run the test suite with uvx run-test-suite" where it turns out run-test-suite is a malicious PyPI package.
In that case the README contains malicious instructions which look entirely non-problematic.
That's an interesting case. I can see the logic of calling it prompt injection, but it is still a bit different--you can imagine a person falling prey to it in, in a way that you don't imagine a person listening to "ignore all instructions and ..." attacks.
vbernat | 18 hours ago
An interesting point made later in the article is that Anthropic does not really care despite its marketing material saying it is safe. I don't understand why Claude does not come with an external OS-level sandbox limiting access to the current directory.
equeue | 14 hours ago
There are many tools being written in this space, e.g. https://nono.sh
simonw | 11 hours ago
My problem with those is that it's hard to know which of the dozens of new sandboxing tools I can trust to be 100% solid.
When it comes to security software my bar is that I'd like a $billion+ company with a large and respected dedicated security team to be actively supporting that software.
With the exception of Firecracker - which sadly is not suitable for running local sandboxes on macOS - very few sandboxing products meet that bar.
This is one of the reasons I'm so optimistic about WebAssmebly: existing browser sandboxes absolutely fit my requirements, and WebAssembly is very closely related to those.
(That said, nono.sh does carry the claim that it's "Built by the team that brought you Sigstore", which gives it some hefty credibility in comparison to many other similar tools I've seen.)
landon | 8 hours ago
So you don't use openssh? Curl?
simonw | 8 hours ago
Both of those are used by $billion+ companies with security teams that actively review updates to them and pass any problems back to the maintainers.
vbernat | 13 hours ago
Yes. But Anthropic should have this builtin or not enable automode by default.
quasi_qua_quasi | 9 hours ago
It does come with one, it's just not turned on by default (iirc). It doesn't sandbox the harness itself I don't think, but it will sandbox the shell tool calls.
z3bra | 7 hours ago
I've always seen agent's "sandboxing" as parents using authority to keep their children in their bedrooms.
The fact that agent promise to "stay" in a directory while running commands from
/binspeaks for itself. Sandboxing must be done at a higher level to be effective.quasi_qua_quasi | 7 hours ago
It is doing some level of actual OS-level sandboxing because I've seen it complain about not being able to write to ~/.local to take out a lock or whatever. (as in, I look at the shell tool and it has an "operation not permitted" thing from the binary on stderr)
z3bra | 7 hours ago
Oh, that's interesting ! In my (very little) experience it has always been se sort of "I'm not allowed to do that" sort of sandboxing regarding write permissions only.
tomjakubowski | 5 hours ago
Claude Code has an experimental mode which sandboxes the whole harness. It uses bubblewrap on Linux and seatbelt on Mac OS.
https://code.claude.com/docs/en/sandbox-environments#sandbox-runtime
lifepillar | 20 hours ago
Glad to see the “prompt injection is solved” marketing bullshit being debunked.
The article appears to be written with LLM support, but the attack is interesting, not very complex, and described in clear terms.
hyperpape | 12 hours ago
I would not have thought to call this is prompt injection, and I'm wondering what other people think. I'd be somewhat surprised to find that this is what is meant by the term.
This attack is manipulating Claude into doing something unsafe, of course, and it's a real weakness of the model/harness that this can happen.
However, as I understood it, prompt injection is not "any security vulnerability that occurs while using Claude". What I understand by prompt injection is a specific vulnerability where instructions in the content are misinterpreted as being either user instructions or system prompts, so Claude then follows that instructions, giving the attacker control of what Claude does.
simonw | 11 hours ago
The prompt injection is this bit:
As a user you have given Claude an innocent looking instruction.
That website is where the prompt injection is hosted. You've asked Claude to summarize the information, but instead the information includes content that tricks Claude into installing and executing malware.
hyperpape | 10 hours ago
I don't see how that's relevant. You told Claude to summarize the website. The website contains encoded files. Claude writes code and executes it in an insecure way to access those files.
The website's instructions are never interpreted by Claude as part of a prompt, so it doesn't seem like prompt injection to me, it's Claude getting confused about secure coding practices.
The issue is not whether you've given Claude innocent instructions, it's whether Claude has interpreted someone else's instructions as authoritative prompts that it needs to follow.
simonw | 9 hours ago
Huh, I think you're right about this. In this particular attack we don't actually have any cases where malicious instructions were appended to the current prompt - instead, we have a situation where the environment itself confuses Claude into eventually executing malware.
That said, it's similar to another attack which I do consider a prompt injection - one where Claude accesses a README which includes instructions to "Run the test suite with
uvx run-test-suite" where it turns outrun-test-suiteis a malicious PyPI package.In that case the README contains malicious instructions which look entirely non-problematic.
hyperpape | 6 hours ago
That's an interesting case. I can see the logic of calling it prompt injection, but it is still a bit different--you can imagine a person falling prey to it in, in a way that you don't imagine a person listening to "ignore all instructions and ..." attacks.
Wilfred | 13 hours ago
Yikes, an 80% injection success rate is sobering.
I don't understand why curl was a necessary step though. Surely something downloaded with webfetch would also need decoding if it's a funky format?
darichey | 11 hours ago
The webfetch tool filters and summarizes content on the server and so wouldn’t download the zip to the client machine.
hyperpape | 10 hours ago
What would happen in this case if WebFetch was able to download the files? It wouldn't be able to read them.