Speeding up similar builds with OCI layers

The conceptual issue with the approach you suggest is that a Nix build is usually entirely unlike a docker build. The only commonality is that they run pre-determined programs inside of a sandbox.

Docker build is snapshot based. You take the entire filesystem state from the root and snapshot/store it with all its metadata and everything after each and every command execution.

In a “standard” Nix build OTOH, builds are typically ran in some temporary location that is not included in the output. Outputs are then explicitly installed into the output path (i.e. make install PREFIX=$out). All commands required to produce that output path are ran inside the same sandbox instance.

Storing the intermediary state as Nix drvs is theoretically possible but runs into practical limitations:

  1. There is no such concept as basing one step on the output of another as you cannot write to the other store paths. You’d have to copy the state from the previous output path (ro) to the new one (rw) for every command/phase ran.
  2. You cannot store arbitrary metadata in a nix store path. Mtimes, permissions etc. would all be stripped. This would already break make as it relies on mtimes to know which target have and haven’t been reached.
  3. You’d accumulate quite a few in-between derivations. A handful per actual build. They shouldn’t be in the runtime closure (though that is not guaranteed) but it still creates a lot of garbage to be cleaned up.

You’re not the first person to desire the ability to introspect failing builds. See i.e. Why is there no way to run `nix-shell` in a chroot and without the user's .bashrc? · Issue #903 · NixOS/nix · GitHub. I also recently read about plans to migrate the Nix sandbox to bubblewrap which would ease the implementation of such a feature somewhere but I can’t remember where.

Until then, manually running the phases inside of the package’s nix-shell gets you 90% of the way there. The only thing it can’t do is reproduce sandbox issues like undesired internet or filesystem access.

1 Like