Shameless self-plug: I wrote a module for just that. Requires a fair bit of compilation, but enables fully-self contained initrd's with entire NixOS user-space in ~80 MiB (compressed with zstd). The work is not limited to self-contained initrds, the module can also just be used to debloat any NixOS (quite possibly also the install ISO)?
The size and quest for slimmer nixos closures & disk images is something that I've dabbled with since I started using NixOS in ~2021.
I did play around with a hobby project to build a minimal useful (for me) initrd back then. A few years later, earlier this year, I rewrote it with what I had meanwhile learned about NixOS: https://github.com/dep-sys/nix-dabei/blob/main/default.nix
The rewrite leans heavily on upstream tooling, for a minimal code base. Just about 400 sloc yield a ~60M (zstd-compressed; ~150M uncompressed) uki that can either be kexec'ed into (a bundle is included) or written to the ESP of a disk image. seeding ssh images and such is done via systemd credentials. That image includes nix (runable in the initrd), zfs, rsync, helix, etc to bootstrap/rescue "full systems".
It just uses the module system to include/exclude stuff (including all of stage2), but does (currently) build any custom packages or such so you should hit cache.nixos.org instead of rebuilding custom kernels or such.
It does lack documentation though, as the cultural tradition around NixOS requires. ;) I just recently took up work on a yet-unreleased installer based on nix-dabei that does ship more docs and some of them could probably be "backported".
Another dimension of "small nixos" that bugs me from time to time, is the fact that NixOS still goes through ~ALL nixos modules in nixpkgs to evaluate even the smallest closures which does have negative performance implications.
I also gave this my own shot in https://github.com/NixOS/nixpkgs/pull/507740 (and its sub-issues) but would describe it as currently stalled. For people who are really committed, the long discussion in https://github.com/NixOS/nixpkgs/pull/507676 might shed some light on pros & cons of my chosen approach and possible next steps (untangle nixos internals, to easy independent module evaluation).
I recommend this talk from nixcon where they shrunk nixos down as an alternative to yocto: https://youtu.be/AsXY61laNb8
It doesn't go into as much detail as I would have liked, but talking with Óli and Matthew about it at the con was incredible. Do anyone know if they have a writeup somewhere?
I'm glad you got it. Debloating Linux is always an interesting activity and learning why others were able to minimise he size so much can give more insight on the process and the components required.
It's like Linux from scratch, but in the opposite direction. Great post.
But there are also plenty of things you could do that require a deeper effort. For example, right now we're still bundling both systemdMinimal and systemd, which seems pretty silly.
Since 26.05, NixOS uses systemd in its initrd by default, since the variety of initrd usecases that a modern OS needs to support is large.
systemdMinimal is a systemd binary compiled with fewer flags/dependencies, which helps keep the initrd small(er).
But if a minimal ISO is the goal, perhaps they could both depend on the same binary.
The irony is that initrd needs a full systemd by now, if stuff such as tpm bound key enrollment needs to supported.
There's also quite bit of low hanging fruit. E.g. that a default 26.05 initrd does indeed include both systemd and systemdMininmal. But also that enabling zfs pulls systemd and util-Linux in the full variants into the closure. And NFS for an optional dependency
It should also be possible to reduce ssh footprint with this:
Yeah.. for the use-case that originally drove me to write the post I didn't really need ssh at all.
But this is probably the more sensible approach most of the time.
You can also import "${nixpkgs}/nixos/modules/profiles/minimal.nix". It contains some of your optimizations.
Yeah, I remember seeing it ages ago and getting a bit underwhelmed, so I didn't think to include it when I started the investigation. And once I did remember about it, I was already about halfway through and it felt like it would've been a bit dishonest to try to slot it in at the start where it ought to have gone.
It's so weird how many times you see perl pulled into the system these days. Small ISO - perl. Wanna compile anything serious from scratch - openssl -> perl.
You can run NixOS 26.05 and unstable without perl, while having a complete NixOS system by using that profile as far as I know. An earlier implementation did not support "switchable" systems, but only appliances. The current state should cover the complete stack (systemd-boot instead of grub, userborn, etc, overlay)
Something I wanted to do with NixOS was run it in a systemd-nspawn container. Declarative system description, running in a container, lighter than a VM, sounds great.
I somehow just couldn't get it working easily. If anyone has any links to do this I would appreciate.
If you're running NixOS on the host too then NixOS containers are basically this.
Personally I don't really love the current state of it though.. it simultaneously manages to be much heavier than OCI (since you're effectively running a slimmed-down second NixOS inside the container, systemd and getty and all), and still loses out on fine-grained reactivation (host-side activation/nixos-rebuild switch only knows how to tear down the container and boot up a new one if anything changes, it can't restart individual services like it can for the host itself).
wucke13 | 22 hours ago
Shameless self-plug: I wrote a module for just that. Requires a fair bit of compilation, but enables fully-self contained initrd's with entire NixOS user-space in ~80 MiB (compressed with zstd). The work is not limited to self-contained initrds, the module can also just be used to debloat any NixOS (quite possibly also the install ISO)?
https://github.com/wucke13/minimal-nixos/
phaer | 10 hours ago
The size and quest for slimmer nixos closures & disk images is something that I've dabbled with since I started using NixOS in ~2021.
I did play around with a hobby project to build a minimal useful (for me) initrd back then. A few years later, earlier this year, I rewrote it with what I had meanwhile learned about NixOS: https://github.com/dep-sys/nix-dabei/blob/main/default.nix
The rewrite leans heavily on upstream tooling, for a minimal code base. Just about 400 sloc yield a ~60M (zstd-compressed; ~150M uncompressed) uki that can either be kexec'ed into (a bundle is included) or written to the ESP of a disk image. seeding ssh images and such is done via systemd credentials. That image includes nix (runable in the initrd), zfs, rsync, helix, etc to bootstrap/rescue "full systems". It just uses the module system to include/exclude stuff (including all of stage2), but does (currently) build any custom packages or such so you should hit cache.nixos.org instead of rebuilding custom kernels or such.
It does lack documentation though, as the cultural tradition around NixOS requires. ;) I just recently took up work on a yet-unreleased installer based on nix-dabei that does ship more docs and some of them could probably be "backported".
Another dimension of "small nixos" that bugs me from time to time, is the fact that NixOS still goes through ~ALL nixos modules in nixpkgs to evaluate even the smallest closures which does have negative performance implications.
There's been a bunch of efforts of documenting and tackling this limitation. e.g. https://github.com/NixOS/nixpkgs/pull/148456 and https://github.com/NixOS/nixpkgs/pull/401751 come to mind, with much of the remaining historical context linked in those PRs.
I also gave this my own shot in https://github.com/NixOS/nixpkgs/pull/507740 (and its sub-issues) but would describe it as currently stalled. For people who are really committed, the long discussion in https://github.com/NixOS/nixpkgs/pull/507676 might shed some light on pros & cons of my chosen approach and possible next steps (untangle nixos internals, to easy independent module evaluation).
pbsds | 20 hours ago
I recommend this talk from nixcon where they shrunk nixos down as an alternative to yocto: https://youtu.be/AsXY61laNb8
It doesn't go into as much detail as I would have liked, but talking with Óli and Matthew about it at the con was incredible. Do anyone know if they have a writeup somewhere?
alemi | 23 hours ago
TinyCore Linux base system is 17 MB (Core).
If you want X with FLTK/FLWM, 23 MB (TinyCore).
If you want more WM and apps, 248 MB (CorePlus).
http://www.tinycorelinux.net/downloads.html
Diti | 23 hours ago
How is that relevant to declarative configuration and reproducible VMs?
[OP] natkr | 22 hours ago
Not super relevant for the immediate use case, but it is good as a benchmark for "huh, why can't we get closer to that, too?".
alemi | 12 hours ago
I'm glad you got it. Debloating Linux is always an interesting activity and learning why others were able to minimise he size so much can give more insight on the process and the components required.
It's like Linux from scratch, but in the opposite direction. Great post.
tuxes | 17 hours ago
Since 26.05, NixOS uses systemd in its initrd by default, since the variety of initrd usecases that a modern OS needs to support is large.
systemdMinimalis a systemd binary compiled with fewer flags/dependencies, which helps keep the initrd small(er).But if a minimal ISO is the goal, perhaps they could both depend on the same binary.
phaer | 5 hours ago
The irony is that initrd needs a full systemd by now, if stuff such as tpm bound key enrollment needs to supported.
There's also quite bit of low hanging fruit. E.g. that a default 26.05 initrd does indeed include both systemd and systemdMininmal. But also that enabling zfs pulls systemd and util-Linux in the full variants into the closure. And NFS for an optional dependency
vbernat | 23 hours ago
Good post. It's always a bit frustrating to have a small footprint with NixOS.
It should also be possible to reduce ssh footprint with this:
You can also import
"${nixpkgs}/nixos/modules/profiles/minimal.nix". It contains some of your optimizations.[OP] natkr | 23 hours ago
Yeah.. for the use-case that originally drove me to write the post I didn't really need ssh at all. But this is probably the more sensible approach most of the time.
Yeah, I remember seeing it ages ago and getting a bit underwhelmed, so I didn't think to include it when I started the investigation. And once I did remember about it, I was already about halfway through and it felt like it would've been a bit dishonest to try to slot it in at the start where it ought to have gone.
viraptor | 20 hours ago
It's so weird how many times you see perl pulled into the system these days. Small ISO - perl. Wanna compile anything serious from scratch - openssl -> perl.
tuxes | 18 hours ago
There is work to reduce and ratchet the amount of Perl in the base system: https://github.com/NixOS/nixpkgs/blob/master/nixos%2Fmodules%2Fprofiles%2Fperlless.nix
phaer | 10 hours ago
I think it's mostly done?
You can run NixOS 26.05 and unstable without perl, while having a complete NixOS system by using that profile as far as I know. An earlier implementation did not support "switchable" systems, but only appliances. The current state should cover the complete stack (systemd-boot instead of grub, userborn, etc, overlay)
ac | 18 hours ago
Without reading - I predict its some stupid perl script nobody has bothered to rewrite in C.
edit: yep
[OP] natkr | 10 hours ago
I mean, it's a part of it, but a pretty minor one.
sammko | 5 hours ago
Something I wanted to do with NixOS was run it in a systemd-nspawn container. Declarative system description, running in a container, lighter than a VM, sounds great. I somehow just couldn't get it working easily. If anyone has any links to do this I would appreciate.
[OP] natkr | 4 hours ago
If you're running NixOS on the host too then NixOS containers are basically this.
Personally I don't really love the current state of it though.. it simultaneously manages to be much heavier than OCI (since you're effectively running a slimmed-down second NixOS inside the container, systemd and getty and all), and still loses out on fine-grained reactivation (host-side activation/
nixos-rebuild switchonly knows how to tear down the container and boot up a new one if anything changes, it can't restart individual services like it can for the host itself).