đź”— distri: a Linux distribution to research fast package management

Links
1

Another distro that is trying to re-think packaging. I like that it’s using a squashfs image per store path. I suspect that unpacking NAR files onto the filesystem actually takes quite a long time for Nix.

2 Likes
2

It’s about trade-offs, I think. I actually like that nix doesn’t restrict you on filesystem choice and doesn’t require “mangling the mount namespace” when running packages (by which I mean containers, virtualization, etc.)

1 Like
3

It seems like distri is using FUSE to create an overlay on the /ro prefix (their /nix/store equivalent) and avoid mounting 1000s of filesystems. The trade-off is probably that installation is very fast but reading from the store is slower due to the FUSE<->kernel context-switching overhead.

Still, we could have both modes in Nix. Just like sandboxing has to be disabled inside of docker containers.

4 Likes
4

Some more interesting discussion over here: distri: a Linux distribution to research fast package management | Lobsters

5

It sounds like they could (should?) use the in-kernel OverlayFS for this, which gives native performance after files are opened.

6

It sounds like they could (should?) use the in-kernel OverlayFS for this, which gives native performance after files are opened.

I would guess that a FUSE union filesystem is easier to reconfigure on the fly without closing the files untouched by the change.

7

There is also this promising ExtFuse project. The idea is that some parts of the FUSE driver can be loaded in the kernel as eBPF extensions. Depending on what the FUSE driver does this can speed up quite considerably.

8

I looked into it a bit, turns out they did use OverlayFS originally but it has (had?) really long mount times when overlaying many directories.

9

Worth noting that Stapelberg was a user of Nix, and even contributed some fixes to Nixpkgs.

$ git log | grep -i stapelberg -B1 -A4
commit 9cbf8a06526385482e9732f9283fe73674756c02
Author: Michael Stapelberg <stapelberg@users.noreply.github.com>
Date:   Mon Nov 14 22:04:32 2016 -0800

    Fix buildMachines example: use lists, not string (#20361)
    
--
commit 87793207e806e837a3ae2958f3e770caec6f2277
Author: Michael Stapelberg <stapelberg@users.noreply.github.com>
Date:   Mon Sep 12 10:22:28 2016 +0200

    debug-info: use pkgs.lib.overrideDerivation
3 Likes
10

I think this discussion kind of misses the advantages of distri. The FUSE FS is cool, but NixOS may still struggle to be as fast as distri, even if it avoided extracting archives and used an overlay fs of its own.

An important feature of distri is enforced parallelism: packages cannot execute arbitrary pre and post-install hooks, so the package manager can do much more work concurrently.

11

I think you are somewhat mis-understanding. The pre- and post-install hooks he refers to are the kind used in typical Linux distributions, which are generally capable of reading from and writing to the entire filesystem. This is why they prevent concurrent package installation: you need a serialized installation order for the resulting system state to be at least somewhat predictable.

Nix build phases, on the other hand, are only capable of reading from their pre-defined inputs and writing to the package output path. Nix binary package substitutes can be (and are) downloaded and installed (almost) entirely concurrently as a result.

Nix source package builds do rely on having their full set of inputs available before the build can begin, of course, but they can (and do) still run concurrently with any package build that doesn’t depend upon them (directly or transitively).

3 Likes