I created a new override-utils package for simplifying Nixpkgs overrides/overlays, which is part of a broader project of mine to improve the usability of Nixpkgs. This post will focus more on the big picture but I'll also motivate the override-utils package.
Several years ago I wrote "The hard part of type-checking Nix", which was my take on the usability issues plaguing the Nix ecosystem. The relevant excerpt from that post is:
The fundamental problem that plagues all type-checking attempts for Nix is that nobody actually uses Nix the language at any significant scale. Instead, the community has adopted two sub-languages embedded within Nix for programming “in the large”:
Nixpkgs overlays
This is an embedded language that simulates object-oriented programming with inheritance / late binding / dynamic scope (depending on how you think about it)
NixOS modules
This is an embedded language that roughly emulates Terraform
Carefully note that these are not language features built into Nix; rather they are embedded domain-specific languages implemented within Nix. Consequently, a type-checker for “Nix the language” is not necessarily equipped to type-check these two sub-languages.
In other words, the problem with Nix isn't that Nix doesn't have types; it's that even if Nix had types they'd be just as impenetrable as current stack traces because Nix is operating at the wrong level of abstraction. We need to design better abstractions before we can build a type system that works well in inexpert hands.
To this end, I sat down and asked myself: "if I could build a programming language purpose-built for working with Nixpkgs, what would that ideal language look like?". I figured this would be a useful thought experiment, but also I have enough experience with implementing programming languages that I could perhaps build out a proof of concept if I thought it were compelling enough.
While designing this "Nixpkgs language" I realized that I struggle most with override functions and overlays so I started there. To illustrate what I mean, suppose that I needed to add the libvirt package as a native dependency to Haskell's libvirt-hs package1. I'd have to do something like this:
final: prev: {
haskellPackages = prev.haskellPackages (old: {
overrides = hfinal: hprev: {
libvirt-hs =
final.haskell.lib.overrideCabal
hprev.libvirt-hs
(old: {
libraryPkgconfigDepends =
(old.libraryPkgconfigDepends or []) ++ [
final.libvirt
];
});
};
});
}Gross. Moreover, it's an even bigger pain if I want to do this for a non-default GHC version:
final: prev: {
haskell = prev.haskell // {
packages = prev.haskell.packages // {
ghc98 = prev.haskell.packages.ghc98.override (old: {
overrides = hfinal: hprev: {
libvirt-hs =
final.haskell.lib.overrideCabal
hprev.libvirt-hs
(old: {
libraryPkgconfigDepends =
(old.libraryPkgconfigDepends or []) ++ [
final.libvirt
];
});
};
});
};
};
}I've used Nixpkgs long enough that I'm used to this sort of thing by now, but this is not the sort of user experience that I would confidently recommend to a coworker if they were on the fence about Nix. This poor user experience is (in my view) a big contributor to why Nix gets consistently sidelined as companies grow2.
So I asked myself: how would I have preferred to write that last example?
The answer I converged upon was something along these lines:
haskell.packages.ghc98.override.overrides =
libvirt-hs.overrideCabal.libraryPkgconfigDepends ++= [ libvirt ];There's plenty we can workshop there, but I still think something like that would be much less intimidating to a newcomer. Additionally, that syntax is much more autocomplete-friendly! A user could write:
haskell.packages.ghc98.override.<TAB>… and their editor could suggest all of the available overrides for completion, which is very doable because even without a type system you can query the list of available overrides3:
nix-repl> haskell.packages.ghc98.override.__functionArgs
{ all-cabal-hashes = false;
buildHaskellPackages = false;
compilerConfig = true;
configurationArm = true;
configurationCommon = true;
configurationDarwin = true;
configurationJS = true;
configurationNix = true;
configurationWindows = true;
ghc = false;
haskellLib = false;
initialPackages = true;
lib = false;
nonHackagePackages = true;
overrides = true;
packageSetConfig = true;
pkgs = false;
stdenv = false;
}Moreover, this simpler syntax also suggests a simpler type system. Instead of thinking in terms of override functions and overlays I believe we should just be thinking in terms of attribute paths and safe operations on those attribute paths. Structuring all overrides in that way would greatly simplify the type system (both the implementation and the user experience).
I have not yet built any such programming language. What I did do, though, is to release an override-utils package that approximates that idealized interface in pure Nix. I created this package as a starting point to prove out the idea before even thinking about embarking on a larger programming language project.
For example, using override-utils the above example would be written as:
final: override {
haskell.packages.ghc98.override.overrides = set (hfinal: override {
libvirt-hs.overrideCabal.libraryPkgconfigDepends =
append [ final.libvirt ];
});
}… which is already pretty similar to the original idealized interface I proposed:
haskell.packages.ghc98.override.overrides =
libvirt-hs.overrideCabal.libraryPkgconfigDepends ++= [ libvirt ];… but there are a few important differences that I want to dig into.
Something like attribute ++= list is not valid Nix, so override-utils works around that by instead writing attribute = append list.
On the other hand attribute = value is valid Nix, but is still not supported by override-utils and you have to instead do attribute = set value. Why? Because otherwise there's no way to distinguish between this:
{ foo = set { bar = 1; }; } # Replace `foo` with `{ bar = 1; }`… and this:
{ foo.bar = set 1; } # Replace `foo.bar` with `1`Those should not mean the same thing, and you can see why with this minimal example:
nix-repl> :print override { foo = set { bar = 1; }; } { foo.baz = 2; }
{ foo = { bar = 1; }; }
nix-repl> :print override { foo.bar = set 1; } { foo.baz = 2; }
{ foo = { bar = 1; baz = 2; }; }The former completely replaces the foo attribute (including the old baz sub-attribute), whereas the latter does not.
If you remove the set then both operations become { foo.bar = 1; } and we can no longer distinguish which one the user meant. If we were to implement the same thing in a separate domain-specific language (i.e. not Nix) then we could preserve that distinction and drop the need for set.
override-utils also requires you to bring the final package set into scope when creating an overlay so that you can refer to other packages within the package set. For example, this overlay:
final: prev: {
x = prev.x + 1;
y = final.x
}… corresponds to this override:
final: override {
x = add 1;
y = set final.x;
}In the idealized interface what I would prefer is that you can drop the final on the happy path and instead have naked references to other attributes when it's unambiguous. For example, I'd like to be able to write this:
x += 1;
y = x;However, you would still have the option of qualifying these references when there would be multiple final package sets in scope (e.g. final vs hfinal in our original example) so that you could disambiguate which one you meant.
To expand on that last point, let's revisit the original idealized example:
haskell.packages.ghc98.override.overrides =
libvirt-hs.overrideCabal.libraryPkgconfigDepends ++= [ libvirt ];I specifically chose this example as a starting point because in the absence of a qualifier I would like the libvirt reference in the idealized code to translate to hfinal.libvirt or final.libvirt in Nix (which would then further evaluate to final.libvirt since there is no hfinal.libvirt).
However, I'd still like to user to be able to opt out of implicit qualification and make things explicit like this:
final: haskell.packages.ghc98.override.overrides =
libvirt-hs.overrideCabal.libraryPkgconfigDepends ++=
[ final.libvirt ];The good news is that most other features I had in mind were totally doable in pure Nix, which is kind of cool!
In fact, the core implementation is actually very simple. If you drop support for error messages and default values, you can implement a "dollar store override-utils" in just a few lines of Nix code:
rec {
override = argument:
let
adapt = name: value:
let
default = old: old // {
"${name}" = override value old."${name}";
};
in
{ modify = value;
"*" = map (override value);
override = old: old.override (override value);
overrideAttrs = old: old.overrideAttrs (override value);
overrideDerivation =
old: old.overrideDerivation (override value);
overrideCabal =
old: haskell.lib.overrideCabal old (override value);
}."${name}" or default;
in
prev: lib.pipe prev (lib.mapAttrsToList adapt argument);
modify = f : { modify = f; };
set = value: modify (_: value);
add = value: modify (x: x + value);
subtract = value: modify (x: x - value);
append = suffix: modify (prefix: prefix ++ suffix);
prepend = prefix: modify (suffix: prefix ++ suffix);
# … add other utilities to taste …
}The real implementation is slightly more complete and featureful, but the above snippet should give you some idea of how things work under the hood. Those more familiar with functional programming lore might recognize that this is the Nix analog of semantic editor combinators (the predecessors to van Laarhoven lenses).
Or another way to think of it is that this is all equivalent to the Setter type from Haskell's lens package. Attributes in the attribute path are equivalent to Setters and chaining attributes is equivalent to composing Setters (and they both use ".").
Edit: I'm embarrassed to say that I had missed infuse.nix when researching prior art. A few people pointed this out and I wrote up a quick comparison between
override-utilsandinfuse.nixhere.
The closest existing project in this space is dream2nix, which (among other things) also exposes an improved interface to Nixpkgs (based on the original drv-parts project). Specifically, dream2nix exposes a NixOS module interface to Nixpkgs, where you can write code like this:
{ config, lib, ... }: {
pip.overrides.opencv-python = {
env.autoPatchelfIgnoreMissingDeps = true;
mkDerivation.buildInputs = [
pkgs.libglvnd
pkgs.glib
];
};
}… and for comparison, the equivalent overlay using override-utils would be:
final: override {
pythonPackageExtensions = append (pfinal: override {
opencv-python.overridePythonAttrs = {
env.autoPatchelfIgnoreMissingDeps = set true;
buildInputs = append [
final.libglvnd
final.glib
];
};
});
}Kind of similar, right? That's why I added an Appendix where I compare the two projects for people who were interested.
I think override-utils is a useful contribution to the Nix ecosystem in its own right, but I also think we can still do much better. In particular, in the conclusion of my post on The hard part of type-checking Nix I summarized several paths forward for the language:
Solution A: Don’t implement a type system for Nix
…
Solution B: Only type-check Nix “in the small”
…
Solution C: Type-check Nixpkgs overlays using a type system supporting row polymorphism
…
Solution D: Implement the two sub-languages in an external language
In other words, implement the Nixpkgs overlay system and NixOS module system in a separate language that is not Nix so that overlays and modules are supported by the language along with a type system that natively understands these features. Then you could compile this external language to ordinary Nix code that is compatible with the existing Nixpkgs overlay system or NixOS module system.
Solution E: Like Solution D, but upstream these features into the Nix language
…
Solution F: Like Solution D, but without Nix as an intermediate language
…
My best guess is that “Solution C” or “Solution D” are the two most promising approaches that strike the right balance between how difficult they are to implement and actually addressing what users want in a type-checker.
Nowadays my thinking that "Solution D" is the correct path forward. In other words, I think we need to build a domain-specific language for Nixpkgs (and NixOS) that compiles to Nix. This domain-specific language should not only be typed but also should operate at a higher abstraction level where abstractions like overrides/overlays and modules have built-in support from the language and type system.
override-utils is just a small step in that direction: I created this package to prove out the idea that the correct abstraction for overrides/overlays is composable Setters and safe operations on those Setters.
If you're interested in learning more about the override-utils package I recommend checking out the README which contains the full documentation for the package (it's not very complicated).
dream2nixThis section covers the biggest differences between override-utils and dream2nix.
As far as I can tell, dream2nix is an "all or nothing" interface, meaning that dream2nix replaces your project's Nix entrypoint entirely with its own interface (e.g. dream2nix.lib.importPackages or dream2nix.lib.evalModules). This means that you can't incrementally port over an existing Nix project to dream2nix one piece at a time: you need to design your entire project around dream2nix from the beginning or migrate everything over wholesale.
In contrast, override-utils generates overrides and overlays, meaning that you can use override-utils anywhere an override or overlay is valid and no additional ceremony is required.
Also, overlays are composable, which means that you can migrate things over to override-utils one piece at a time. For example, if you have a list of overlays:
import nixpkgs {
inherit system;
overlays = [
(import ./lib-overlay.nix)
(import ./rust-overlay.nix)
(import ./firefox-overlay.nix)
(import ./git-cinnabar-overlay.nix)
];
}Then any one of those individual overlays can be ported to use override-utils in isolation without any other changes to the surrounding code.
This is true not just for top-level overlays but also any nested overlay or override. For example, suppose you have an overlay like this:
final: prev: {
nix-serve-ng = prev.nix-serve-ng.overrideAttrs (old: {
patches = (old.patches or []) ++ [ ./lix-compat.patch ];
});
}Sure, you could replace the whole overlay with override:
final: override {
nix-serve-ng.overrideAttrs.patches = append [ ./lix-compat.patch ];
}… but you can also replace just the override function if you don't want to migrate the entire overlay:
final: prev: {
nix-serve-ng = prev.nix-serve-ng.overrideAttrs (override {
patches = append [ ./lix-compat.patch ];
});
}In other words, you can migrate as much or as little as you want because I designed override-utils to be as unobtrusive to adopt as possible.
The last section touches upon another difference: dream2nix is somewhat at odds with with the Nixpkgs overlay abstraction. Overlay support was added to Nixpkgs with the intention that overlays would serve as the grand unifying abstraction, and dream2nix is promoting NixOS modules as an alternative grand unifying abstraction instead.
override-utils instead aims to be more respectful of existing Nixpkgs design decisions and generates overlays (and override functions) instead of trying to replace or subsume them.
dream2nix benefits in a major way from using the NixOS module system because you can provide option declarations and they provide multiple benefits:
To expand upon the last point, if you define the following dream2nix option multiple times (across more than one module):
pip.overrides.opencv-python.mkDerivation.buildInputs =
[ pkgs.libglvnd ];
pip.overrides.opencv-python.mkDerivation.buildInputs =
[ pkgs.glib ];… then those multiple definitions are automatically merged into a single list because the declared type for buildInputs is a list and by default the list type appends multiple definitions together.
In override-utils the merge logic is specified at the point where the option is defined (e.g. using append, which also merges multiple definitions) because in override-utils there is no such thing as an option declaration or an option type.
Related to the last point, dream2nix only works if you wrap your desired Nixpkgs functionality in a NixOS module interface and there is a lot of functionality in Nixpkgs that you would need to wrap in this way in order to comprehensively support all of Nixpkgs.
override-utils requires no such up-front investment. All of Nixpkgs, today, works with override-utils and it only took ≈200 lines of Nix code (compared to ≈10,000 lines of dream2nix modules which still doesn't support anywhere near all of Nixpkgs).
dream2nix is doing much more than just provide a NixOS module interface to Nixpkgs; dream2nix is a more ambitious project that is attempting to unify all of the *2nix utilities under a single umbrella/framework. The NixOS module interface to Nixpkgs is just one component of that broader overarching vision to provide a cohesive interface for interoperating with every programming language from Nix.
On the other hand, I've been approaching this problem from a different standpoint, which is to think about Nixpkgs ergonomics from a programming languages perspective/background (e.g. syntax, types, and semantics). However, it's still fascinating that the two projects converged upon similar interfaces despite different overarching goals and implementations . That suggests some sort of convergent evolution for Nix ergonomics.
Note that libvirt is already a native dependency of libvirt-hs so this override is not necessary, but I've had to do overrides similar to this in the past to work around broken builds in the past. I just picked this as a contrived example that is still representative of real examples. ↩
It's so bad that one fintech company using Nix offered me $800K total compensation to tame their Python build. ↩
Thank you to whomever had the foresight to add support for that to the various .override* methods. ↩