Hello, Reader!
I’m MJ Pooladkhay, and for the past couple of months I have been building a Virtio-FS driver for Redox OS and, more recently, getting Redox to run on Cloud Hypervisor. The driver is merged into the base repository, where most Redox drivers and system components live. The kernel, bootloader and driver changes that let Redox boot to a shell on Cloud Hypervisor are merged too (details below).
Both are fun pieces of work on their own, but they are really the first steps of a bigger effort with two directions:
The guest work is generously funded by NGI Zero Commons and NLnet as part of the Virtualized Redox grant.
Most Internet-facing software today runs in virtual machines, either in the cloud or on edge servers close to its users. Web servers, databases, API backends and microservices spend all day handling input from strangers, and that is where a memory-safe microkernel-based operating system has the most to offer:
This is the direction the project set in its development priorities: Redox as a secure host for web services, first as a private or edge server, and eventually as a multi-tenant cloud server where each tenant is isolated from others.
The easiest way I know to think about a hypervisor is as an operating system whose processes are operating systems. A process runs directly on the CPU until it needs the kernel, for a system call, a page fault or a timer tick, and then the kernel takes over for a moment. A guest OS runs directly on the CPU in the same way, until it does something that only the hypervisor is allowed to handle. That moment a “VM exit” happens where the control is transferred to the hypervisor (similar to how a system call transfers control to the OS kernel). On Linux, KVM in the host kernel handles it, either by itself or by passing it to the virtual machine monitor (VMM), an ordinary process such as Cloud Hypervisor or QEMU. I went through the theory behind this, and why x86 needed new hardware (VT-x and AMD-V) to make it work, in an earlier post on my personal blog.
From Redox’s side, a VM boots much like a laptop does. The virtual CPU starts in firmware, the firmware loads the bootloader from a disk, and the bootloader hands over to the kernel. It’s the same code that runs on real hardware, running natively, with the kernel in supervisor mode (ring 0 on x86) as usual.
Most of what Redox does never leaves the guest. System calls, context switches and page faults are handled by the Redox kernel itself, and the hypervisor doesn’t even see them.
Memory takes a few more steps than on bare metal, because there are now two kernels that each think they own it. Redox’s page tables turn a guest virtual address (GVA) into a guest physical address (GPA), the address Redox believes is in RAM. But guest RAM is really just a big chunk of memory the VMM allocated in its own address space, and the VMM tells KVM which part of it backs which range of guest RAM (a memslot). So every GPA is also a host virtual address (HVA) in the Cloud Hypervisor process, and the host kernel’s page tables turn that into a host physical address (HPA), the actual spot in RAM:
GVA --[Redox's page tables]--> GPA --[KVM memslots]--> HVA --[host page tables]--> HPA
Going through all of that on every memory access would be painfully slow, so KVM collapses the GPA to HPA part into a single step. It builds nested page tables that map guest physical addresses straight to host physical ones, and fills them in lazily as the guest touches memory. The CPU then walks Redox’s page tables and the nested ones together in hardware, and caches the result in the TLB. Redox manages its own page tables freely, and only touching guest memory that isn’t mapped yet causes an exit.
The HVA step has a nice side effect: since guest RAM is an ordinary mapping in the VMM, it can be shared with other host processes. That’s how virtiofsd reads Virtio-FS requests straight out of the guest’s memory, and why sharing a directory requires the guest’s memory to be shared.
Devices are where the hypervisor earns its keep, because every device the guest sees is made up. Some are emulations of real hardware, such as an 8259 interrupt controller, a programmable interval timer (PIT), a serial port or each CPU’s local APIC, and every register access to them is a VM exit that the hypervisor turns into behavior. Others are VirtIO devices, which don’t pretend to be hardware at all and talk to the host through queues in shared memory. Interrupts are made up too. The host keeps the real ones, and the guest only gets the ones the hypervisor injects on behalf of its virtual devices and timers. If the hypervisor doesn’t emulate a device, nobody raises its interrupt (keep that in mind for the ping story below).
In short, the CPU, its timestamp counter (TSC) and the memory are real, and everything else is virtual. The interesting question is which virtual devices you get, and that’s where cloud hypervisors and QEMU part ways.
So far, Redox has been run either on QEMU or on physical hardware. On both, it has leaned on devices that date back to the original IBM PC/AT: a pair of 8259 interrupt controllers, the PIT, a PS/2 keyboard controller, and so on. These are legacy even on real hardware. Modern machines, and OSes such as Linux, route interrupts through the I/O APIC and keep time with each CPU’s local APIC timer. Redox kept using the old devices mainly because it didn’t support the ACPI pieces needed to set up their replacements.
Hypervisors built for the cloud, such as Cloud Hypervisor and Firecracker, drop most of the legacy devices. Cloud Hypervisor gives a guest its CPUs (each with a local APIC), memory, an I/O APIC, a serial port, a real-time clock (RTC), a minimal ACPI platform and a handful of VirtIO devices. Not much else. Fewer emulated devices means less code in the hypervisor for a guest to poke at, and less for the guest OS to probe and set up at boot.
These hypervisors can also skip firmware entirely and jump straight into the kernel (PVH direct boot). Redox doesn’t do that yet and still boots through UEFI firmware.
I would say Redox is a very good fit for minimal VMs like these. A VM that runs a single service doesn’t need a USB stack, a sound daemon or a desktop, and since those are all ordinary userspace daemons on Redox, leaving them out is a matter of not putting them in the image.
What goes into the image, and what runs once it boots, is a short TOML file in the build system. Init services are small TOML files too, so an image can just as well boot straight into a web server, with no one logging in.
The server demo config is an example of this, with SSH, PHP and the nginx configuration.
Virtio-FS lets a VM use a directory that lives on the host. The virtio-fsd Redox daemon talks to the virtiofsd Linux daemon on the host using the FUSE protocol, carried over virtqueues. Each shared directory appears as its own scheme, /scheme/virtio-fs.<tag>, named after the tag the host gives it, so a share can be put in or left out of a process’s namespace like any other scheme. You can share more than one directory at a time.
Under the hood, the driver keeps its own cache of inodes and directory entries, and tells the host to forget the ones it hasn’t used in a while (FUSE_FORGET, driven by an LRU cache). The driver works on both QEMU and Cloud Hypervisor, and comes with a test suite.
Getting Redox to a working shell on Cloud Hypervisor was mostly a matter of finding every place where it assumed it was running natively on a PC, as well as a few other gotchas:
ExitBootServices (bootloader!61).virtio-blk was matched as a transitional VirtIO device, while Cloud Hypervisor only offers modern (non-transitional) ones, so the root disk never showed up (base!422).kvmclock, and rtcd sets the wall clock once at boot from the emulated CMOS RTC, after which it advances with the monotonic clock.QEMU’s q35 machine does have an 8259 interrupt controller and an HPET, so Redox keeps using them there exactly as before.
My favorite bug was this one: with networking up, ping -c 4 1.1.1.1 got a reply in about 15 ms and then… just sat there, until I interrupted it:
ping -c 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 40(68) bytes of data.
From 1.1.1.1 icmp_seq=0 time=15.304ms
Interrupted! Exiting gracefully.
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.00% packet loss
So the NIC, the route and the NAT on the host were all fine. Then where did the other three packets go?
They were never sent! ping sends the first echo right away, then asks the time scheme to wake it up one second later. In the kernel, that wakeup is posted by timeout::trigger, and the only caller of timeout::trigger was the PIT interrupt handler (on QEMU, the HPET drives the same interrupt line). Cloud Hypervisor has neither a PIT nor an HPET, so that handler never ran.
Reading the clock still worked, since kvmclock converts the CPU’s TSC to nanoseconds whenever someone asks, which is why the first reply had a sensible round-trip time. But nothing was ticking. Every ping -i, and every other wait on the time scheme, was waiting for an interrupt that was never going to come, and a nanosleep only woke up when something else happened to run the scheduler. The same handler also drives the scheduler’s preemption tick, so the system had quietly become cooperative: a process only gave up the CPU when it blocked or yielded.
The fix is to use a timer every x86 CPU already has: its local APIC timer. When there is no HPET, each CPU now arms its own local APIC timer in TSC-deadline mode, where you tell it the TSC value at which the next interrupt should fire. The tick period is converted from nanoseconds to TSC ticks using the same scale kvmclock already provides1. The handler then does the PIT handler’s old job, posting expired timeouts and driving the scheduler tick. With that in place, ping -c 4 sends all four packets, one second apart, and Redox is preemptive again!
Booting to a shell is a good start, but it’s not yet something you would deploy. Here’s a look at the remaining work, which is also covered by the current NLnet grant:
ps2d still tries to talk to a keyboard controller that isn’t there and logs an error on every boot. On QEMU, those devices stay in use.acpid already has a path to S5 (soft-off), but it writes the classic PM1 control register, and Cloud Hypervisor doesn’t power off on that. Cloud Hypervisor is a hardware-reduced ACPI platform, where shutdown is a write to the sleep control register and the power button is a Generic Event Device (GED) interrupt that nobody handles yet. The goal is for shutdown to work both from the host and from inside the guest.virtio-vsock for talking to the host without a network, virtio-balloon for returning memory to the host, and virtio-rng for drawing entropy from it. These join the existing virtio-{blk, net, fs, gpu} fleet.Once the list above is done, Redox should work like any other cloud image: you launch it, it picks up its hostname and SSH key on first boot, and you log in and run your services.
The same pieces also fit more specific uses, such as short-lived instances for CI and testing, and sandboxes for AI agents (more on that one below).
Whatever runs inside the VM, Redox can stack three layers of isolation around it:
To reach the host, a misbehaving tool would have to break through all three.
And for talking to the host:
virtio-vsock lands, whatever manages the VM on the host can talk to software inside it, with no network interface in the guest.A minimal image and Virtio-FS fit together nicely, because they split a system into the part that never changes and the part that has to survive.
The image holds the kernel, the drivers and the service, and nothing writes to it. Every instance boots from exactly the same bytes, an upgrade becomes just booting a new image, and a rollback is booting the old one. Redox can already run a live image entirely from memory, and the system could eventually live in InitFS, the in-memory filesystem Redox boots from, with no RedoxFS image at all.
Everything that has to outlive the VM, such as databases, uploads and logs, lives on the host and reaches the guest through Virtio-FS. Each share is just a scheme, so the service only gets the shares it needs, and its configuration can come in as a read-only share. Throw the VM away, boot a new image, point it at the same shares, and the data is still there.
The only catch is that Virtio-FS needs the virtiofsd daemon running on the host, next to the hypervisor. It’s the same mechanism Kata Containers uses to give its VMs their files. On machines you run yourself (a server, an edge box, a bare-metal cloud instance, or a cloud VM with nested virtualization), setting it up is easy. Regular public cloud VMs and Firecracker don’t offer Virtio-FS, so there the persistent part goes on an ordinary attached disk instead, which already works (Redox boots on Google Cloud, for example).
One potentially interesting use case is sandboxing AI agents, which needs every one of these pieces. Agents run code and tools that nobody has reviewed, and the usual answer is a throwaway VM for each task. On Redox, that VM starts from a small image or a snapshot. The agent’s tool runner gets its working folder and only the other schemes it actually needs, and tools compiled to WebAssembly add another layer on top. The agent’s workspace is a Virtio-FS share, so its changes land on the host as soon as it makes them, and the orchestrator talks to it over virtio-vsock. When the task is done, the VM is thrown away.
The second direction flips the picture. As a guest, Redox runs on top of Linux, and Linux’s drivers deal with the real hardware.
As a host, it would be the other way around. Redox runs on the hardware and uses its own drivers for the devices it supports, and a small Linux guest takes care of the ones it doesn’t (say, a network card with no Redox driver yet). The device is handed to the guest through PCI passthrough (VFIO on the Linux side), and Linux offers it back to Redox through VirtIO. Instead of all of Linux’s drivers or none of them, Redox gets to choose, and each borrowed driver can be replaced by a native one when it’s ready. This is the “hosted Linux for driver support” idea from the development priorities, and FreeBSD already does something similar with bhyve and wifibox, where a small Linux VM drives the Wi-Fi card.
Redox has explored hosting VMs before, with Revirt during RSoC 2022, and now we have a solid plan for it:
The kernel would gain AMD SVM support, exposed through a new kernel scheme. It would own only what a guest must never be able to change: SVM enablement, the VMCB (the structure that holds a guest’s state and decides what causes an exit), the nested page tables, the virtual local APIC and the switch between host and guest. Everything else would live in an ordinary Redox process, the VMM, which creates the VM, supplies its devices and handles its exits. It’s the same split as KVM and QEMU on Linux, but done the Redox way, through a scheme instead of ioctls.
The VMM would be a small program built on the rust-vmm crates (vm-memory, virtio-queue and linux-loader), not a port of Cloud Hypervisor or QEMU, and it wouldn’t clone /dev/kvm either. Its register and exit records would be KVM-compatible, though, so a Cloud Hypervisor or QEMU backend stays possible later.
The work would build up in steps:
Intel VT-x guests are future work, but VT-d will be implemented.
And what would tie both directions together? A Redox guest booting on a Redox host.
See how to run Redox on Cloud Hypervisor.
To share a directory from your host with a Redox VM over Virtio-FS, follow this guide.
If you want to help, or just follow along, join us on Matrix and take a look at the global issue tracker.