How to use a local directory as a nix binary cache?

Help
1

Many CI providers give you a directory whose contents are retained across builds and you can use that as a cache. Everything that is stored elsewhere is lost. This means that any artefacts that are created during a nix-build that are placed in the nix store ( /nix/store ) are lost. I’m trying to figure out how to convince nix to prefer that other directory over the global /nix/store. However the documentation is a bit lacking.

What I’ve tried so far:

  • Add file:///the/path to substituters and then nix copy --to that path. However I discovered that nix only creates some metadata files in that directory and copies the actual derivation into /nix/store . That’s not what I want.
  • Use local?root=/the/path instead of the file:// url (btw, this syntax is not documented anywhere, I only found it in a single github issue!). That made nix copy the whole derivation to that folder, but I couldn’t figure out how to convince nix-build to actually consult that store during build.

I’ve also tried to make /nix a symlink to a folder inside the cache directory, but nix errors out if /nix is a symlink.

Nix Installation Guide - NixOS Wiki lists two more options: option 1 is to use nix 2.0 --store option, but that assumes that nix (nix-shell) is already installed globally (I think, please correct me if I’m wrong). And option 2 is to use PRoot which I couldn’t get to work inside a docker image.

4 Likes
2

The first approach should work. The key is to use file:/// in the nix copy command too though — did you maybe not do that? If you just use the bare path it will be used as a chroot store rather than a flat-file binary cache.

1 Like
3

Problem with the explicit nix copy command is that, well, it’s explicit. Realistically I can only copy the end result, but I don’t know which intermediary derivations were required during the build, so I can’t copy them. Ideally nix would automatically store all derivations it had to create during the build process in that other store (for example compilers etc).

In my particular case I can install nix globally (into /nix) just fine. So I’d like to instruct nix:

  1. If a derivation is available in /nix/store → nothing to do, use it.
  2. If the derivation is available in the local binary cache → copy that to /nix/store and use it.
  3. If a derivation is not available and you have to download or build it → store the result in the local binary cache and continue with step 2.
4

Have you tried to use the --store /cachedir option yet?

This argument can be passed to all the nix command line tools to change where the /nix/store is being stored. It does some unshare+mount magic so it can still download things from the binary cache. You might want to symlink /nix to /cachedir/nix if running anything outside of the nix sandbox.

5

You can use nix-store -qR on the .drv file(s) to determine the build-time closure and copy the outputs of all of those to the cache. You could also use --all to just copy everything in the local store into the cache.

6

This could run into concurrency issues if you ever have more than one nix accessing the store at a time. Garbage collection would also probably not be safe. I’m not sure of all the details though, so this could also be worth a try.

7

Making /nix a symlink doesn’t work (I mentioned already that I tried it)

I also just tried nix-shell --store $HOME/nix-store -p hello and that doesn’t work, fails with the following error:

error: executing shell '/nix/store/jj8g9104z7rkk718s80js3nddr7vbg8p-bash-interactive-4.4-p23/bin/bash': No such file or directory

Interestingly though, I thought --store option only changes where /nix/store is, but it seems to affect the whole /nix folder. After running the above mentioned nix-shell command I was left with the following folder structure:

$HOME/nix-store/
└── nix
    ├── store
    │   ├── 000k6hdph1fqnnv4ab85bndnxj80d181-libiconv-osx-10.11.6.drv
    │   └── ...
    └── var
        └── nix
            ├── db
            │   └── ...
            ├── profiles
            └── temproots

I.e. it not only moved the store there but the whole /nix directory (including var, db, profiles etc).

Setting NIX_STORE_DIR resulted in a different error:

NIX_STORE_DIR=$HOME/nix-store nix-shell -p hello
error: path '/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh' is not in the Nix store
8

--store allows specifying an alternate store, which can be any type of store (like for nix copy’s --to option and binary caches) — so you could use --store file:///some-flat-file-binary-cache as well. Note however that not all stores support all operations. nix-shell for instance does not support any stores other than the local one (optionally via the daemon). You can use nix run with some alternate stores however.

Using a plain path (i.e. no URI scheme) for this means a chroot store, i.e. the whole of /nix is lives in that path (as you have observed). The symlink is useful for running software that lives in your chroot store without involving nix, but nix does not support a symlinked /nix (as you have also observed).

NIX_STORE_DIR is another thing, and is very likely not what you want. It changes the “logical” location of the nix store — so even within the build environment everything lives in a different path than /nix/store. This breaks all compatibility with the official binary cache, so you have to build everything yourself.

1 Like
9

Do I have to somehow initialize the store?

$ nix-build --store file://$HOME/nix-store something.nix
error: requested operation is not supported by store 'file://$HOME/nix-store'

I tried to initialize it by nix copy’ing something to it but that didn’t help.

Btw, the --store option is not documented in the nix-build man page.

10

Flat-file binary caches (file:// stores) do not support building, so what you were trying to do won’t work. Rather, you’d have to build stuff in the local store or a chroot store, then copy the results from there into the flat file binary cache.

11

I guess my only real option then is to somehow make proot work within the container so I can transparently map /nix to the cache.

12

No, using a chroot store should work too, barring concurrency issues and lack of gc safety. nix-build --store $HOME something.nix

13

… but only on linux? I have macOS and this is the result I get:

$ nix-build --store $HOME something.nix
[… snip]
**error:** building using a diverted store is not supported on this platform
14

Oh yeah, chroot stores won’t work on macOS. Neither will proot. Having a flat-file binary cache in the shared dir and copying to/from that will be your only option there.

15

One last question: is it possible to set --store ?? globally via nix.conf or some environment variable? I’d like that to be completely transparent to end-users, they shouldn’t have to know that they have to set that option when executing nix/nix-build. If that’s not possible I’ll have to provide wrappers around those executables that automatically set --store to the correct value, something like this in a shell environment

nix() {
  nix --store $CACHE_DIR "$@"
}
nix-build() {
  nix-build --store $CACHE_DIR "$@"
}
16

Yes. In /etc/nix/nix.conf or $HOME/.config/nix/nix.conf, put a line store = /foo/bar.

2 Likes