I think this is a very interesting post and worthy of discussion. For my part, I agree with it, but only partially. We are definitely at the time in which cheap / unaligned / locally-runnable AI models are excellent at finding vulnerabilities and crafting exploit scripts, and they're only going to get better.
That does suck in some regards, but I think it's 1) going to absolutely devastate the current OpenAI/Anthropic duopoly, and 2) going to / already is leading towards people actually taking security seriously. Like, the state of secure software has been SO bad for SO long... people were writing new projects in C/C++, even after Rust took off. That's crazy! There's really no excuse for not writing any new non-toy project in a memory-safe language, in my opinion.
I am particularly glad that people are starting to take supply-chain security seriously now, too.
One thing I wish this post covered is What do we do, as an individual? How do we prepare for what will inevitably be a deluge of cybersecurity attacks on all parts of the commons, over the next year or two? People in the Apple/Microsoft ecosystems already offload their security concerns to them, for better or for worse... but Linux is rather specific to the nerd using it. For my part, I'm rather concerned about the "long tail" of software -- software running on my computer that is too niche to have anybody fuzz it with an LLM (anybody with positive intentions, at least).
I'm personally going to resolve this by 1) running anything on my computer with network access in some kind of sandbox, 2) locking down the services exposed on my public server, and 3) switching everything over to NixOS (which puts services in secure containers by-default, and generally has stronger guarantees wrt. supply-chain security than everything else). Curious what others are doing.
I would like to see a whole lot more attention paid to sandboxing. It infuriates me that consumer operating systems don't ship with clearly documented, usable sandboxing features - the sandboxes they include today may as well have signs pasted on them saying "Beware of the Leopard".
I would like to be able to run a process in a way that prevents it from digging around in my home directory and exfiltrating anything it finds to someone else over the internet. This is way harder than it should be.
I remain convinced that the right answer for desktop OSes in the long term (albeit not necessarily one that's easy to get to from here) will have to involve capabilities. The identity-based model of processes that run "as you" with all your authority hasn't really fit the way we use computers for a long while now, but especially doesn't fit now.
UI design is definitely a challenge, yeah; capabilities have been more obscure for a while and haven't had as much UI work as more common architectures.
Though for what it's worth, things like the file/photo pickers in Android and iOS are capability-style powerboxes that people use every day. I think it's possible to expand on this, but it will take more design work.
The issue here is that, in general, for any group of aligned folks, everybody is on the security team; security for the group is always dependent on proper behavior from everybody in the group. All we can do is mitigate the inevitable compromise of individuals by attenuating the authorities delegated to them; that is, we can only protect folks who don't care about security by withholding most capabilities from them.
Forget easy to use, are there any actually hardened VM tools for anybody to use, even hard-to-setup ones? Like running a container in a hardened VM on linux. The best I found is firecracker and kata containers on Linux, but kata itself opens up a lot and I’ve been trying to harden that as well. The maintainers are really nice but it’s a lot of work. Are there alternatives though ?
Interesting... gVisor is entirely maintained by Google (they first released it in 2018) but I just saw that two weeks ago they started the process to donate it to CNCF.
Thanks a lot. I looked into it a bit. gVisor seems to have an issue where it's capable of exposing a lot of stuff from the host (socket passthrough, device passthrough, host filesystem sharing). Sure I could try to disable everything, but it's hard to know that everything is actually disabled. Kubernetes has this issue too, you can make Pods restricted to try to remove host access, but they can still get host filesystem access via PVC with host filesystem. It just feels like whack a mole.
I really like that Firecracker is heavily restricted by default. It just doesn't have any code to expose the host filesystem. You can't misconfigure enable host fs sharing. Most of the attack surface are things im adding to the VM myself, not just configured-disabled-features in the VM
To be clear, gVisor is very much intended to be fully secure in the default configuration. Features like using the host networking stack (rather than the default user space stack), device passthrough, etc are all disabled by default and must be explicitly enabled. It is a project goal to make things difficult to accidentally configure insecurely.
The only cases that come to mind where could be a bit more locked down by default are (a) network access could be completely disabled by default, (b) if used via docker, volume mounts specified on the docker command line (-v) will be respected without explicit gVisor level configuration.
I'd say Docker, but everyone warns against using Docker as a sandbox. I've not fully evaluated the new Docker Sandboxes thing yet, though it seems very focused on coding agents and I want to use it for other stuff, too.
Firecracker is a LONG way from being a user-friendly piece of software to install and use.
Apple Containers are nice - pretty much a clone of Docker with a slightly different CLI - but I don't know if they are recommended for sandboxing or not.
My problem with sandboxes is that what I really want is an easy-to-use sandbox from a company with a dedicated security team that works on that sandbox product, and who risk millions (ideally billions) of dollars if it leaks.
My problem with sandboxes is that what I really want is an easy-to-use sandbox from a company with a dedicated security team that works on that sandbox product, and who risk millions (ideally billions) of dollars if it leaks.
same, and this is what draws me to Firecracker, it being developed by Amazon for AWS Lambda. It has a very restricted minimal design. They're working on an official container integration so containers can run inside of it (and eg Kubernetes Pods) but it's not ready yet.
For now Im just using Firecracker with a patched version of Kata, having used Astra to delete a lot of Kata code to reduce the attack surface it introduces. Posted about that today. I needed to use an agent to set it up for my homelab but im pretty happy with it.
I wanted to offer that I've had pretty good luck with the (ahem) now deprecated Apple Sandbox (way different from Apple Containers, which is, itself, different from Library/Containers and Library/Group Containers)
As a nod to my sibling comment about chasing the never-ending least privilege, I also have to run it with this patch since they seem to have forgotten that /dev/null exists for some laughable reason
I'm fascinated by sandbox-exec because it's really good, almost completely undocumented, has been marked as deprecated for years, and is increasingly the secret sauce behind all sorts of other sandboxing tech, including that used by OpenAI and Anthropic.
I'm hoping Apple eventually notice how important it has become and decide to not deprecate it after all.
Arguably there is no such thing since the "easy" part is defining the rules and the "hard" part is knowing which rules to define, followed closely by alert-fatigue's sibling whereby one starts out with "deny /", "allow /some/dir" and ends up at "allow /" because it's just too incredibly painful to carve out the actual, no kidding, least privilege version in some iterative fashion as you chase transitive failures
I really love bubblewrap for this on Linux. It really does make it simple to run a single command with controlled permissions. You can share and unshare specific parts of the filesystem, as well as control network access with a simple wrapper script. Linux has some seriously powerful tools to do this, but they're very hard to use. Bubblewrap makes it pretty easy to use them. You can do similar stuff on mac with sandbox-exec, but it's a lot more unwieldy in my experience.
I believe the issue isn't sandboxing, but data-isolation between environments.
Apple gets this right - my terminal can't just my safari data, but it isn't customizable enough to work for arbitrary development or terminal environments. QubesOS is a linux based solution, it has very good isolation but is harder to work with and customize.
I haven’t read the article yet but I’ve had very similar thoughts:
we should assume that for the foreseeable future the rate of disclosure of critical vulnerabilities is going to be much higher, and we should be building systems with this in mind; and
I wonder if this is what finally causes software engineering to be regulated like other kinds of engineering. I’ve spent a year working on a system where I’m not even sure the people who wrote it knew how computers work.
I ran a security audit using GPT-6 Astra and Claude Fable 5.1 against my main open source project (Datasette) recently and then spent a full week fixing vulnerabilities that they found, many of which were extremely obscure, hence why nobody had spotted them before.
An obscure vulnerability is still a vulnerability.
Modern LLMs are ferociously good at finding holes in things. We have a very limited amount of time to shut down those holes before the Bad Guys figure out how vulnerable basically everything is already.
I think it's even worse than that, it's less "we don't even have a year" and more "we have negative six months". Black hats are absolutely going ham in the wild and just not making the news.
I’m in the middle of reading a book on the stuxnet attack and one thing that really stood out to me is that at least two of the zero days had been publicly published or disclosed for a year or more.
also known as "don't plug random hardware into your other random hardware"
I am thankful that I mostly remember to bring my own chargers with me, but just in case something goes awry I do actually carry a usb condom with me when I travel
USB protection doesn’t really matter when it only takes a single host to infect patched up to date systems or if it then adds itself to any healthy usb drive later inserted.
Also the centrifuges were air gapped so the usb technology aspect was choosen specifically, and not the standard autorun usb vulnerabilities.
I highly recommend the book if interested in learning more about it.
I've been spending quite a lot of time fixing llm-reported vulnerabilities this year, especially in the last 3 or 4 months. Hopefully at some point, chromium will just "not have bugs", and if introduced they get found quickly before they hit stable. It's a lot of work getting there though lol.
[OP] apropos | 13 hours ago
I think this is a very interesting post and worthy of discussion. For my part, I agree with it, but only partially. We are definitely at the time in which cheap / unaligned / locally-runnable AI models are excellent at finding vulnerabilities and crafting exploit scripts, and they're only going to get better.
That does suck in some regards, but I think it's 1) going to absolutely devastate the current OpenAI/Anthropic duopoly, and 2) going to / already is leading towards people actually taking security seriously. Like, the state of secure software has been SO bad for SO long... people were writing new projects in C/C++, even after Rust took off. That's crazy! There's really no excuse for not writing any new non-toy project in a memory-safe language, in my opinion.
I am particularly glad that people are starting to take supply-chain security seriously now, too.
One thing I wish this post covered is What do we do, as an individual? How do we prepare for what will inevitably be a deluge of cybersecurity attacks on all parts of the commons, over the next year or two? People in the Apple/Microsoft ecosystems already offload their security concerns to them, for better or for worse... but Linux is rather specific to the nerd using it. For my part, I'm rather concerned about the "long tail" of software -- software running on my computer that is too niche to have anybody fuzz it with an LLM (anybody with positive intentions, at least).
I'm personally going to resolve this by 1) running anything on my computer with network access in some kind of sandbox, 2) locking down the services exposed on my public server, and 3) switching everything over to NixOS (which puts services in secure containers by-default, and generally has stronger guarantees wrt. supply-chain security than everything else). Curious what others are doing.
simonw | 10 hours ago
I would like to see a whole lot more attention paid to sandboxing. It infuriates me that consumer operating systems don't ship with clearly documented, usable sandboxing features - the sandboxes they include today may as well have signs pasted on them saying "Beware of the Leopard".
I would like to be able to run a process in a way that prevents it from digging around in my home directory and exfiltrating anything it finds to someone else over the internet. This is way harder than it should be.
jfred | 8 hours ago
I remain convinced that the right answer for desktop OSes in the long term (albeit not necessarily one that's easy to get to from here) will have to involve capabilities. The identity-based model of processes that run "as you" with all your authority hasn't really fit the way we use computers for a long while now, but especially doesn't fit now.
simonw | 8 hours ago
If we can figure out how to wrap that in a UI that doesn't require half a degree in cybersecurity to use safely and effectively I'm all for it!
jfred | 7 hours ago
UI design is definitely a challenge, yeah; capabilities have been more obscure for a while and haven't had as much UI work as more common architectures.
Though for what it's worth, things like the file/photo pickers in Android and iOS are capability-style powerboxes that people use every day. I think it's possible to expand on this, but it will take more design work.
Corbin | 8 hours ago
It's worth considering the degree to which need-to-know policies, compartmentalization, and principles of least authority/access/privilege are already successfully used by various governments to organize groups of non-computer-specialists. In the USA, SCIFs are examples of physical partially-gapped powerboxed (tamed) sandboxes.
The issue here is that, in general, for any group of aligned folks, everybody is on the security team; security for the group is always dependent on proper behavior from everybody in the group. All we can do is mitigate the inevitable compromise of individuals by attenuating the authorities delegated to them; that is, we can only protect folks who don't care about security by withholding most capabilities from them.
srcreigh | 10 hours ago
Forget easy to use, are there any actually hardened VM tools for anybody to use, even hard-to-setup ones? Like running a container in a hardened VM on linux. The best I found is firecracker and kata containers on Linux, but kata itself opens up a lot and I’ve been trying to harden that as well. The maintainers are really nice but it’s a lot of work. Are there alternatives though ?
refi64 | 8 hours ago
You might find gVisor to be of interest.
simonw | 8 hours ago
Interesting... gVisor is entirely maintained by Google (they first released it in 2018) but I just saw that two weeks ago they started the process to donate it to CNCF.
srcreigh | 6 hours ago
Thanks a lot. I looked into it a bit. gVisor seems to have an issue where it's capable of exposing a lot of stuff from the host (socket passthrough, device passthrough, host filesystem sharing). Sure I could try to disable everything, but it's hard to know that everything is actually disabled. Kubernetes has this issue too, you can make Pods restricted to try to remove host access, but they can still get host filesystem access via PVC with host filesystem. It just feels like whack a mole.
I really like that Firecracker is heavily restricted by default. It just doesn't have any code to expose the host filesystem. You can't misconfigure enable host fs sharing. Most of the attack surface are things im adding to the VM myself, not just configured-disabled-features in the VM
prattmic | 3 hours ago
To be clear, gVisor is very much intended to be fully secure in the default configuration. Features like using the host networking stack (rather than the default user space stack), device passthrough, etc are all disabled by default and must be explicitly enabled. It is a project goal to make things difficult to accidentally configure insecurely.
The only cases that come to mind where could be a bit more locked down by default are (a) network access could be completely disabled by default, (b) if used via docker, volume mounts specified on the docker command line (-v) will be respected without explicit gVisor level configuration.
srcreigh | 3 hours ago
Yes, I was referring to running kubernetes pods in a gVisor runtime class (runsc), where hostPath mounts will also just work by default.
simonw | 8 hours ago
I'd say Docker, but everyone warns against using Docker as a sandbox. I've not fully evaluated the new Docker Sandboxes thing yet, though it seems very focused on coding agents and I want to use it for other stuff, too.
Firecracker is a LONG way from being a user-friendly piece of software to install and use.
Apple Containers are nice - pretty much a clone of Docker with a slightly different CLI - but I don't know if they are recommended for sandboxing or not.
My problem with sandboxes is that what I really want is an easy-to-use sandbox from a company with a dedicated security team that works on that sandbox product, and who risk millions (ideally billions) of dollars if it leaks.
srcreigh | 8 hours ago
same, and this is what draws me to Firecracker, it being developed by Amazon for AWS Lambda. It has a very restricted minimal design. They're working on an official container integration so containers can run inside of it (and eg Kubernetes Pods) but it's not ready yet.
For now Im just using Firecracker with a patched version of Kata, having used Astra to delete a lot of Kata code to reduce the attack surface it introduces. Posted about that today. I needed to use an agent to set it up for my homelab but im pretty happy with it.
mdaniel | 5 hours ago
I wanted to offer that I've had pretty good luck with the (ahem) now deprecated Apple Sandbox (way different from Apple Containers, which is, itself, different from
Library/ContainersandLibrary/Group Containers)I'm grateful to iTerm2's build process for turning me onto it: https://gitlab.com/gnachman/iterm2/-/blob/v3.7.2/deps.sb?ref_type=tags and recently Homebrew caught wind but I have no idea how long that's going to last given the deprecation: https://github.com/Homebrew/brew/blob/7.0.4/Library/Homebrew/extend/os/mac/sandbox.rb
As a nod to my sibling comment about chasing the never-ending least privilege, I also have to run it with this patch since they seem to have forgotten that
/dev/nullexists for some laughable reasonsimonw | 4 hours ago
I'm fascinated by
sandbox-execbecause it's really good, almost completely undocumented, has been marked as deprecated for years, and is increasingly the secret sauce behind all sorts of other sandboxing tech, including that used by OpenAI and Anthropic.I'm hoping Apple eventually notice how important it has become and decide to not deprecate it after all.
mdaniel | 5 hours ago
Arguably there is no such thing since the "easy" part is defining the rules and the "hard" part is knowing which rules to define, followed closely by alert-fatigue's sibling whereby one starts out with "deny /", "allow /some/dir" and ends up at "allow /" because it's just too incredibly painful to carve out the actual, no kidding, least privilege version in some iterative fashion as you chase transitive failures
see also: AWS IAM, et al
spc476 | 3 hours ago
Would SELinux work? Or is that too hard to even use effectively?
[OP] apropos | 8 hours ago
I just use a separate Linux user and Linux ACLs.
mdaniel | 5 hours ago
That's the security model used by Android, also (err, without the sudo part) - every app gets its own uid
spillybones | 5 hours ago
I really love bubblewrap for this on Linux. It really does make it simple to run a single command with controlled permissions. You can share and unshare specific parts of the filesystem, as well as control network access with a simple wrapper script. Linux has some seriously powerful tools to do this, but they're very hard to use. Bubblewrap makes it pretty easy to use them. You can do similar stuff on mac with sandbox-exec, but it's a lot more unwieldy in my experience.
decentstates | 5 hours ago
I believe the issue isn't sandboxing, but data-isolation between environments.
Apple gets this right - my terminal can't just my safari data, but it isn't customizable enough to work for arbitrary development or terminal environments. QubesOS is a linux based solution, it has very good isolation but is harder to work with and customize.
I have build a model solution for NixOS called nix-housing: https://decentstat.es/posts/shai-halud-nix-housing/
doctor_eval | 9 hours ago
I haven’t read the article yet but I’ve had very similar thoughts:
we should assume that for the foreseeable future the rate of disclosure of critical vulnerabilities is going to be much higher, and we should be building systems with this in mind; and
I wonder if this is what finally causes software engineering to be regulated like other kinds of engineering. I’ve spent a year working on a system where I’m not even sure the people who wrote it knew how computers work.
simonw | 10 hours ago
We don't even have a year.
I ran a security audit using GPT-6 Astra and Claude Fable 5.1 against my main open source project (Datasette) recently and then spent a full week fixing vulnerabilities that they found, many of which were extremely obscure, hence why nobody had spotted them before.
An obscure vulnerability is still a vulnerability.
Modern LLMs are ferociously good at finding holes in things. We have a very limited amount of time to shut down those holes before the Bad Guys figure out how vulnerable basically everything is already.
wareya | 2 hours ago
I think it's even worse than that, it's less "we don't even have a year" and more "we have negative six months". Black hats are absolutely going ham in the wild and just not making the news.
brad | 10 hours ago
I’m in the middle of reading a book on the stuxnet attack and one thing that really stood out to me is that at least two of the zero days had been publicly published or disclosed for a year or more.
mdaniel | 5 hours ago
also known as "don't plug random hardware into your other random hardware"
I am thankful that I mostly remember to bring my own chargers with me, but just in case something goes awry I do actually carry a usb condom with me when I travel
brad | 4 hours ago
USB protection doesn’t really matter when it only takes a single host to infect patched up to date systems or if it then adds itself to any healthy usb drive later inserted.
Also the centrifuges were air gapped so the usb technology aspect was choosen specifically, and not the standard autorun usb vulnerabilities.
I highly recommend the book if interested in learning more about it.
dmurph | 4 hours ago
I've been spending quite a lot of time fixing llm-reported vulnerabilities this year, especially in the last 3 or 4 months. Hopefully at some point, chromium will just "not have bugs", and if introduced they get found quickly before they hit stable. It's a lot of work getting there though lol.