How To Use "Super Colliding Nix Stores" or `local-overlay-store`?

Guides
1

Since there is Super Colliding Nix Stores we wanted to give this a try. My coworker and I were a bit confused by the various places of documentation.

Generally what does Super Colliding Nix Store do, when to use it?

As others have asked about this before

Here is a quick introduction after getting it to work:

  • super colliding nix stores is basically introducing the nix experimental feature local-overlay-store
  • this allows to have multiple nix-stores without collisions
    • a nix store consists out of /nix/store and /nix/var/nix/db/db.sqlite
  • local-overlay-store is a set of features
    • reading multiple nix db.sqlitefrom different locations
    • on a privileged system, setup mount points for you to “bootstrap” the system

Setting up the local-overlay-store is accessed through enabling the experimental feature, and then passing special arguments to the store = in your nix.conf (user or daemon).

An example of this is in your nix.conf:
store = local-overlay://?lower-store=%2Fhost-nix%2F%3Fread-only%3Dtrue&upper-layer=/host-nix/upper&check-mount=false

Which can be read as

  • use local-overlay://
  • pass the arguments
    • lower-store → required for the lower-store database access
    • upper-layer → not necessarily needed when you’re already mounting /nix/store in a different way
    • root → not needed when you’re mounting /nix/store in a different way
  • additionally there are “sub-arguments” (that’s why lower-store has URL encoding
    • read-only=true → otherwise nix will try to write to the lower store, error: remounting /host-nix/nix/store writable: Operation not permitted
      • this requires read-only-local-store experimental feature to be active
    • check-mount=false → some other nix oddity, errors with error: overlay filesystem ‘/nix/store’ mounted incorrectly otherwise

With that said, you can already go ahead and use the setup.

To replicate a minimal setup you can do the following (example is working with /scratch filesystem).

# 1. Create the directory structure
sudo mkdir -p /scratch/tmp/nix-overlay/upper /scratch/tmp/nix-overlay/work /scratch/tmp/nix-overlay/merged /scratch/tmp/nix-overlay/conf


# 2. Prepare a `nix.conf` that can be re-used across docker runs
echo "experimental-features = nix-command flakes local-overlay-store read-only-local-store
store = local-overlay://?lower-store=%2Fhost-nix%2F%3Fread-only%3Dtrue&check-mount=false
sandbox = false
build-users-group =
" | sudo tee /scratch/tmp/nix-overlay/conf/nix.conf

# 3. Mount the OverlayFS on the HOST
sudo mount -t overlay overlay -o lowerdir=/nix/store,upperdir=/scratch/tmp/nix-overlay/upper,workdir=/scratch/tmp/nix-overlay/work /scratch/tmp/nix-overlay/merged

After this preparation work you can start interacting:

docker run --rm -it --name nix-overlay-test \
              -v /scratch/tmp/nix-overlay/merged:/nix/store:rw \
              -v /nix:/host-nix/nix:ro \
              -v /scratch/tmp/nix-overlay/conf/nix.conf:/root/.config/nix/nix.conf:ro \
              -v /run/current-system:/run/current-system:ro \
              -v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/static:/etc/static:ro \
              -e PATH="$PATH" \
              -e NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt \
              -e USER=root -e HOME=/root \
              cfgarden/empty:latest sh -l

Inside there you can run

nix shell nixpkgs#cowsay and it will retrieve data and put it onto the merged overlay.

Q&A:

  • Q: Running nix-collect-garbage -d will do what?
    • A: This will create an overlayfs notice (and nix sqlite) that these files don’t exist anylonger → can be rebuild, they still exists on the lower-store but become invisible for the merged view
  • Q: How can I validate it is working?
    • A: If you check inside the container the size of ls -lh /nix/var/nix/db/db.sqlite it should be much smaller than a hosts one. For me I have a container one that is 3.1MB and the host one with 400MB
  • Q: What can happen now?
    • A: The “lower-dir” nix store can receive updates, for example if you run nix shell nixpkgs#audacity on your host and later in the already started container as well it will show up instantly.
  • Q: What happens if I delete /nix/store paths on the host system?
    • A: overlayfs is not ready for this task, this will break the overlayfs with funny errors

Generally, this can allow you building secure nix build pipelines that are performant but still follow certain guarantees. Main issue is that if you’re using NFS for the lower-dir, the files are not cachable with just using overlayfs.

1 Like