How do people feel about the hardware isolation provided by microVMs versus strong software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up seeing a transition toward patchable software isolation, similar to the rise of software RAID that occurred a while back.
The general vibe I hear from security practitioners is that they (gvisor and Firecracker/Cloud-Hypervisor) are comparable. You generally do Firecracker if you can get away with it, because it's simpler and faster, but to do it well you generally need bare-metal hosts.
I think "hardware" vs. "software" is the wrong way to frame this. The big difference is between shared-kernel multitenant and guest kernels. gVisor (in some configurations) is like an 80% solution; a partially emulated guest kernel. I think the verdict is in on shared-kernel vs. not: don't share kernels between tenants.
The hardware isolation itself --- AMD-V, VT-X, &c --- has held up for almost 2 decades now. You never know, of course, but if you were going to place a bet it probably wouldn't be on hardware isolation failing before software did.
gVisor and Firecracker are nothing alike. gVisor is an application kernel written in Go. Application kernels will make your system much slower and can only solve a subset of security issues that would affect the kernel. They're like a kernel proxy of sorts.
Firecracker is a KVM virtual machine but made much more efficient. It's the most desirable of the two. Firecracker and KVM don't require bare metal, but in a virtualized environment they will require nested virtualization support. Nested virtualization is supported by just about every hyperscaler.
The reason KVM is preferable is because it provides strong isolation of the host kernel interface and independent emulation of host devices.
I've built systems out of both and they are quite similar, using different mechanisms as vectors towards the same outcome, which is to minimize shared kernel attack surface in untrusted workloads.
Firecracker doesn't require bare metal, but (1) you want it for performance and efficiency at scale and (2) you take a security hit for nesting.
Surely the fact that one is running a "a partially emulated guest kernel" as you put it vs a real linux kernel makes them sufficiently different, no?
I haven't used gVisor in a while, but could you mount an NFS or CIFS share on gvisor or can you manage nftables, seccomp, namespaces, etc? At least that didn't use to be possible. So depending on the solution you're trying to provide, there was a big difference.
In my experience, someone will eventually care, especially in enterprises or large orgs that don’t know what they are doing, then want to run dockerd or mount a share from some legacy system. Then the problem becomes do you lose that class of users or do you build 2 backends (A gVisor and a kvm/firecracker/etc) or just build a kvm one.
Google tried to do their Cloud Run gvisor only, then 2 years later switched to a hypervisor (though I don’t know if they use kvm or something else). That has been my experience also in trying to run an “emulated” Linux anything. Eventually some tough compat issue arrises and you wish you had just ran the real thing and figured all the issues with that instead. Tbf, gVisor offers much better density and much easier resource management etc. just the compat issues are tough unless you’re willing to (or can) say no to some scenarios.
> Who cares? The overwhelming majority of cloud customers just want to run straightforward Node.js apps.
The security professionals whose sentiments you were describing would surely care. I was answering as a software engineer that often works at this level of security. I would and do care.
Replit intrigues me. Other than Paul Graham talking it up a lot on Twitter - I've never once heard of anyone using it, seen any chatter about it in any or our internal slack forums. Every other AI company I get barraged with- no matter where I go, even antigravity/gemini carbon/argon/models/etc... - which nobody I know uses either - but at least I hear about them a lot.
I wonder what ecosystem replit exists in that seems to completely not overlap with any of the other AI technologies and companies....
I've been picking up bits of fractional CTO work over the last year or so and getting more involved with other founders... and I'm seeing plenty of people using it (and the other systems like it). They're just not generally people from a software engineering background, as you'd probably expect. So it's not a massive surprise that you're not crossing paths with them.
Generally it's entrepreneurial types who would have gone to an agency to get something built, or would have wanted to go to an agency but couldn't afford the costs involved. These platforms are far cheaper and quicker, and if we're really honest not necessarily worse quality than what they would have gotten from the vast majority of those agencies, especially the cheaper ones.
This is not a comment on whether this is a good thing or not, and I have all sorts of concerns, but it's undeniably something I'm seeing, and the nearshore/offshore agencies are certainly starting to feel the impact.
Replit used to be a great platform for sharing code in a way which made it easy for anyone to run or modify. It was great for students, and kids learning to code. But then, in 2024, they removed all social features, and reduced the maximum number of repls for free users from practically unlimited to 3. I stopped using it after this, and shortly after, they implemented a strict ratelimit for downloading your own repls.
I'm now taking a look at Replit for the first time in a while, and it looks like they completely redid their interface: Their whole web IDE appears to be gone or vastly simplified, and it seems like half of of everything I click on fails with error 429, after just a tiny bit of looking around. Their old IDE that functioned as a code editor, and gave you basic tools such as a file tree viewer, is no more. Replit turned an awesome product into a glorified LLM chat which is vastly dysfunctional for any sort of hand coding.
I have been helping a non-profit migrate off Replit. What an awful experience using the platform has been. Their LLMs produce very bad code compared to Claude Code or Codex with the exact same prompt. Ultimately you end up having to go back and forth to get to a better solution while they nickel and dime you on tokens. It feels questionable, like it is intentional to squeeze money out of you.
Having used lovable.dev as well, I must say Lovable is a much better product.
Replit is easily one of the worst products and platforms I've use. I wouldn't even recommend it to my enemies.
I did some contract work on a project that was produced with replit, extracting it to github and setting up its production and staging environments, auditing for issues, resolving some of them.
There was so much that wasn't done, it was basically an Airbnb-like service and it lacked any capacity to register users or hosts, pay for or disburse payments, create or manage listings, notifications, administration. They went from expecting to launch in a couple weeks to looking at my roadmap and quote of about 3 months more development work just for a launchable MVP, which wildly exceeded their budget or commitment to the project.
I don't know how much of this was Replit's fault but some of it certainly was, like despite being a monorepo the frontend had no connectivity with the backend, there was no barriers between user data and most endpoints were not secured, the server had zero hardening, some endpoints would take the user's id as a posted value.
I have seen it used in schools maybe 8 or 10 years ago. High school cs teachers were swearing by it because it made it easier than asking the school IT to install java or python on every computer a kid might use. They were paying for it too at some schools.
Replit's later attempted pivot to B2B or enterprise was odd. I didn't see them succeeding per se, but I had assumed they probably built a generic code execution backend, and probably figured they can resell it to some businesses doing patch processing or whatever have you. I have built many of these "execute this piece of (potentially malicious or untrusted) code for me" platforms in my career. They all end up looking more or less the same on the backend, but their characteristics are different depending on the particular consumer. I saw a billboard about a Replit + Plaid ad which was odd.
The university that I teach at used Replit a few years ago before migrating away.
I used it in our larger Programming Languages courses with 200-300 students, replit was the guaranteed alternative to offer if someone had a problem with say local installation of Clojure or Go or Rust.
At some point in 2023 or 2024 Replit became extremely restrictive in their Education offerings.
Looking at replit front page I guess they pivoted away from Education.
I can't even tell what Replit offers now as they don't even have a free plan anymore?
PS I moved all my courses to Github Codespaces as Plan B. It was a bit more work but heck even Prolog works fine there. I mean its just Ubuntu.
I’ve had multiple clients start a vibe coded app on replit. It’s insane what a normie can build nowadays. Of course they need my help turning it into a real boy, but I have to imagine we don’t hear about it because the people using it are just not in the tech circle at all.
My read on this is they re-invented the OTP supervision tree, process isolation, hibernation, and actor-to-actor message passing, but had to push the isolation boundary all the way down into Ring -1 (KVM/Cloud Hypervisor) and customized Linux tinyconfig kernels because their "actors" are actively trying to jailbreak the hypervisor.
No, that's pretty overwritten. The simple story is: like most neocloud providers, they were using Docker and/or Linux containers as a workload isolation primitive, those have never been safe, and they've now worked out how to scale actual KVM isolation to their users.
A thing I wish more people would do: ask prospective cloud hosting companies "is there a mode of this offering where my workloads might share a kernel with other tenants I don't trust, as would be the case with Docker?", and if the answer to that question is ever "yes", disqualify the provider.
WHAT? I had once went into an absolute rabbithole into a similar topic and not in my wildest dreams could I have imagined that Replit (or others) are just using normal docker for the workload.
I thought that the case of security was better within the industry and that the industry would be punished for bad security practices.
What were the engineers of Replit doing then? Even with a generous perspective, I fail to understand how they couldn't have pushed back against it just until now. This is unacceptable in many ways.
The engineers must have realized that the security that they had was mostly just all duct tape? (because if not then it's a bigger problem)
or is it just something about how startups operate or the incentive structures of the corporation that I am failing to understand?
What you must understand is business. Bad security practices haven't resulted in punishment that makes businesses take security seriously. At the end of the day, they may even be right about it. If it costs $10,000 to break down the door, but behind the door is only $100, why are you spending $10,000?
But doesn't the business realize that it's valuation is based on some level of trust including trust of data. Failure of properly securing that data and their customers hurts the brand in a more permanent manner.
Is the cost analysis of the situation treated it as a "worth it" situation? That, being fast and breaking things and bad security practices are a part of cost cutting and having more revenue faster, something without which a startup may fail.
Have they recognized some deeper secret where for the startup, it is much better to fail big than failing small? (especially so when in some sense most founders who went big and failed, are actually rewarded for it in their future more often than those who failed small)
I have no question in mind that for the executives, the situation is wildly different.
Though, I have a question to ask: What about the engineers working there?
Is it that management stresses them to not work on fixing it and if that is the case, then how are they able to rationalize it upon and their contributions to it (if the end result of it is that the company gets customers into such a system and ends up with their systems compromised because of following such bad practices?)
clhodapp | 20 hours ago
How do people feel about the hardware isolation provided by microVMs versus strong software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up seeing a transition toward patchable software isolation, similar to the rise of software RAID that occurred a while back.
tptacek | 18 hours ago
I think "hardware" vs. "software" is the wrong way to frame this. The big difference is between shared-kernel multitenant and guest kernels. gVisor (in some configurations) is like an 80% solution; a partially emulated guest kernel. I think the verdict is in on shared-kernel vs. not: don't share kernels between tenants.
The hardware isolation itself --- AMD-V, VT-X, &c --- has held up for almost 2 decades now. You never know, of course, but if you were going to place a bet it probably wouldn't be on hardware isolation failing before software did.
oooyay | 16 hours ago
gVisor and Firecracker are nothing alike. gVisor is an application kernel written in Go. Application kernels will make your system much slower and can only solve a subset of security issues that would affect the kernel. They're like a kernel proxy of sorts.
Firecracker is a KVM virtual machine but made much more efficient. It's the most desirable of the two. Firecracker and KVM don't require bare metal, but in a virtualized environment they will require nested virtualization support. Nested virtualization is supported by just about every hyperscaler.
The reason KVM is preferable is because it provides strong isolation of the host kernel interface and independent emulation of host devices.
tptacek | 14 hours ago
Firecracker doesn't require bare metal, but (1) you want it for performance and efficiency at scale and (2) you take a security hit for nesting.
eddythompson80 | 12 hours ago
I haven't used gVisor in a while, but could you mount an NFS or CIFS share on gvisor or can you manage nftables, seccomp, namespaces, etc? At least that didn't use to be possible. So depending on the solution you're trying to provide, there was a big difference.
tptacek | 12 hours ago
eddythompson80 | 12 hours ago
Google tried to do their Cloud Run gvisor only, then 2 years later switched to a hypervisor (though I don’t know if they use kvm or something else). That has been my experience also in trying to run an “emulated” Linux anything. Eventually some tough compat issue arrises and you wish you had just ran the real thing and figured all the issues with that instead. Tbf, gVisor offers much better density and much easier resource management etc. just the compat issues are tough unless you’re willing to (or can) say no to some scenarios.
tptacek | 11 hours ago
oooyay | 10 hours ago
The security professionals whose sentiments you were describing would surely care. I was answering as a software engineer that often works at this level of security. I would and do care.
stevekemp | 19 hours ago
I avoided reading the piece, just in case.
pearcoder | 19 hours ago
curiousgal | 18 hours ago
robby_w_g | 13 hours ago
ghshephard | 18 hours ago
I wonder what ecosystem replit exists in that seems to completely not overlap with any of the other AI technologies and companies....
robgough | 18 hours ago
Generally it's entrepreneurial types who would have gone to an agency to get something built, or would have wanted to go to an agency but couldn't afford the costs involved. These platforms are far cheaper and quicker, and if we're really honest not necessarily worse quality than what they would have gotten from the vast majority of those agencies, especially the cheaper ones.
This is not a comment on whether this is a good thing or not, and I have all sorts of concerns, but it's undeniably something I'm seeing, and the nearshore/offshore agencies are certainly starting to feel the impact.
mritchie712 | 16 hours ago
panphora | 12 hours ago
With the advent of vibe coding, pretty much everyone has dozens (or hundreds) of early-stage side projects now.
nafey | 17 hours ago
1e1a | 16 hours ago
doomspork | 16 hours ago
Having used lovable.dev as well, I must say Lovable is a much better product.
Replit is easily one of the worst products and platforms I've use. I wouldn't even recommend it to my enemies.
tmpz22 | 15 hours ago
1) they were inexperienced or just hesitant to use even managed infrastructure
2) they started with a small Replit footprint and kind of got stuck there out of some niche convenience
3) it started free* and was thought of as a cost cutting measure to use this temporarily VC subsidized service (we've all been there)
benoau | 15 hours ago
There was so much that wasn't done, it was basically an Airbnb-like service and it lacked any capacity to register users or hosts, pay for or disburse payments, create or manage listings, notifications, administration. They went from expecting to launch in a couple weeks to looking at my roadmap and quote of about 3 months more development work just for a launchable MVP, which wildly exceeded their budget or commitment to the project.
I don't know how much of this was Replit's fault but some of it certainly was, like despite being a monorepo the frontend had no connectivity with the backend, there was no barriers between user data and most endpoints were not secured, the server had zero hardening, some endpoints would take the user's id as a posted value.
eddythompson80 | 12 hours ago
Replit's later attempted pivot to B2B or enterprise was odd. I didn't see them succeeding per se, but I had assumed they probably built a generic code execution backend, and probably figured they can resell it to some businesses doing patch processing or whatever have you. I have built many of these "execute this piece of (potentially malicious or untrusted) code for me" platforms in my career. They all end up looking more or less the same on the backend, but their characteristics are different depending on the particular consumer. I saw a billboard about a Replit + Plaid ad which was odd.
bill10 | 9 hours ago
sireat | 4 hours ago
I used it in our larger Programming Languages courses with 200-300 students, replit was the guaranteed alternative to offer if someone had a problem with say local installation of Clojure or Go or Rust.
At some point in 2023 or 2024 Replit became extremely restrictive in their Education offerings.
Looking at replit front page I guess they pivoted away from Education.
I can't even tell what Replit offers now as they don't even have a free plan anymore?
PS I moved all my courses to Github Codespaces as Plan B. It was a bit more work but heck even Prolog works fine there. I mean its just Ubuntu.
jkahrs595 | 50 minutes ago
schainks | 18 hours ago
Is that about right?
tptacek | 14 hours ago
A thing I wish more people would do: ask prospective cloud hosting companies "is there a mode of this offering where my workloads might share a kernel with other tenants I don't trust, as would be the case with Docker?", and if the answer to that question is ever "yes", disqualify the provider.
Imustaskforhelp | 6 hours ago
I thought that the case of security was better within the industry and that the industry would be punished for bad security practices.
What were the engineers of Replit doing then? Even with a generous perspective, I fail to understand how they couldn't have pushed back against it just until now. This is unacceptable in many ways.
The engineers must have realized that the security that they had was mostly just all duct tape? (because if not then it's a bigger problem)
or is it just something about how startups operate or the incentive structures of the corporation that I am failing to understand?
solyin | 5 hours ago
Imustaskforhelp | 4 hours ago
Is the cost analysis of the situation treated it as a "worth it" situation? That, being fast and breaking things and bad security practices are a part of cost cutting and having more revenue faster, something without which a startup may fail.
Have they recognized some deeper secret where for the startup, it is much better to fail big than failing small? (especially so when in some sense most founders who went big and failed, are actually rewarded for it in their future more often than those who failed small)
I have no question in mind that for the executives, the situation is wildly different.
Though, I have a question to ask: What about the engineers working there?
Is it that management stresses them to not work on fixing it and if that is the case, then how are they able to rationalize it upon and their contributions to it (if the end result of it is that the company gets customers into such a system and ends up with their systems compromised because of following such bad practices?)
iberator | 17 hours ago
vivzkestrel | 11 hours ago