Caching NixOS Files for Offline Builds and Bandwidth Reduction

Help
1

There seems to be more than one way to ‘cache’ files to reduce the bandwidth consumed by doing a nixos-install or nixos-rebuild. I have now tried two methods with varying levels of success.

My most recent attempt followed this guide from the old unofficial wiki ( my search error :man_shrugging: ) using NixOS 24.05.

I got to the point where I can successfully add a package (lnav) to my local binary-cache-server

[fsbof@binary-cache-server:/var]$ sudo nix-build '<nixpkgs>' -A pkgs.lnav
this path will be fetched (2.69 MiB download, 9.85 MiB unpacked):
  /nix/store/y91bmbdc0zw84fdv5xsghxvy5h7dzcis-lnav-0.11.2
copying path '/nix/store/y91bmbdc0zw84fdv5xsghxvy5h7dzcis-lnav-0.11.2' from 'https://cache.nixos.org'...
/nix/store/y91bmbdc0zw84fdv5xsghxvy5h7dzcis-lnav-0.11.2

…and then add the file to configuration.nix on my client (testclient4), perform a rebuild and see that it appears to ‘copy’ the binary from the local binary-cache-server, even though this client can’t access the Internet at all (for test purposes).

<snip>
building Nix...
building the system configuration...
these 14 derivations will be built:
<snip>
  /nix/store/r0zd078qq9rfzkby133gfja2yyfyg1gm-nixos-system-testclient4-30Jun24-1516-lnav_with_no_internet.drv
this path will be fetched (0.00 MiB download, 9.85 MiB unpacked):
  /nix/store/y91bmbdc0zw84fdv5xsghxvy5h7dzcis-lnav-0.11.2
building '/nix/store/js4ddnhv0ykw7zi2kx1s9v7frazkqswi-boot.json.drv'...
copying path '/nix/store/y91bmbdc0zw84fdv5xsghxvy5h7dzcis-lnav-0.11.2' from 'http://binary-cache-server.local'...
building '/nix/store/4c0j85j9xjxxiszsy99khwz0if8l22cq-issue.drv'.
<snip>

So far, so good - looks like it does what it should.

So I tried a new install, using the same base-configuration.nix that I used to build both the binary-cache-server and testclient4 before I started following the wiki. My theory being with a similar configuration and being built just days apart, I should see a substantial reduction in bandwidth from the usual circa 1GB.

nixos-install --option substituters \"http://binary-cache-server.local\"

This is where it failed with many errors like this;

warning: ignoring substitute for '/nix/store/ckk16ns9yyy5lk0i1l4xmgghqw055yc2-dbus-1.14.10-dev' from 'http://binary-cache-server.local', as it's not signed by any of the keys in 'trusted-public-keys'

I presume this error is because the files in /nix/store haven’t been signed as they were installed before I did the setup of nginx and nix-serve.

Question 1 : Is there a way to sign all the packages that already exist in /nix/store or to force them to refresh and get signed? ( Of course if this is not the issue, I’m sure I will get schooled appropriately :wink: )

Question 2: The one thing that is also kind of unclear here, if a package is not on the cache, is there a method to get the cache to automatically go and get it, or do I have to manually install/copy it each time? (ie. if I add a package to testclient4, do I need to manually add it to binary-cache-server before I do my rebuild or can I just do the rebuild and binary-cache-server will sort it out!)

I also tried doing this nixos-install --option substituters \"http://binary-cache-server.local https://cache.nixos.org/\" as a test, expecting it to fallback, which it did, but it added nearly 50% to the install time. My Internet access is not great and can be variable …there is far too much copper and not enough fibre it’s diet… so that could have been the issue to and the download was still around 1GB.

Question 3: I guess my last question, is… is the right way to go? The other method I tried was following @Solene’s guide but this was a while ago and my NixOS knowledge has improved considerably since then. Should I go back to that or do something else? :man_shrugging:

TIA

1 Like
2 
nix sign-paths --all -k /etc/nix/signing-key.pem

You need to add it before hand to the cache. You can build the config on the cache (or any other machine and push it to the cache) and then just pull it on the client.
For the attic binary cache you can run it in a daemon mode and it will monitor your store and push new paths to the store automatically.
There are some ways to build automatically on the remote server, I found this to be a good resource when I recently had to build a cache: Running Remote Builds - nixbuild.net documentation

Depends on what you intend to do.
Solene’s approach is ony a cache for the official cache.
It won’t help you if you build custom projects, e.g. I use poetry2nix and it often has to compile packages from scratch. With my own binary cache I only have to do is on one machine.

3 Likes
3

This appeared to work or at least it gave no error but trying to resolve the deprecated alias led me on a merry path that stopped with flake.nix errors and I don’t use flakes. I did find this link but it didn’t help me much.

In terms of the build it was still reporting that many files were not signed as before, so I guess it didn’t quite work after all.

[fsbof@binary-cache-server:/var]$ sudo nix sign-paths --extra-experimental-features nix-command --all -k /var/cache-priv-key.pem
warning: 'sign-paths' is a deprecated alias for 'store sign'

[fsbof@binary-cache-server:/var]$ sudo nix store sign --extra-experimental-features nix-command --key-file /var/cache-priv-key.pem /nix/store
path '/nix/store' does not contain a 'flake.nix', searching up
error: could not find a flake.nix file

Thanks will look at the other answers.

4

I think the flake error happens because you provide it a path.
So this should be enough:

sudo nix store sign --extra-experimental-features nix-command --key-file /var/cache-priv-key.pem --all
1 Like
5

IIRC there isn’t much output when it is successful.

1 Like
6

I think I sussed it. Thank you :smiley:

Using `store sign` instead of `sign-paths`

sudo nix store sign --extra-experimental-features nix-command --all --key-file /var/cache-priv-key.pem also gave no errors or warnings. I think before I dropped the --all. :man_facepalming:

Checking if files are signed

I checked to see if a file that was showing in the log was or wasn’t signed. It was signed.

warning: ignoring substitute for '/nix/store/g5a8ba31a1ghw8656ybh3qw2vk324j90-glibc-locales-2.39-52' from 'http://binary-cache-server.local', as it's not signed by any of the keys in 'trusted-public-keys'

[fsbof@binary-cache-server:/nix/store]$ curl http://binary-cache-server.local/g5a8ba31a1ghw8656ybh3qw2vk324j90.narinfo
StorePath: /nix/store/g5a8ba31a1ghw8656ybh3qw2vk324j90-glibc-locales-2.39-52
URL: nar/g5a8ba31a1ghw8656ybh3qw2vk324j90.nar
Compression: none
NarHash: sha256:198b2bw1607dl58kw217ka90w6anc0d96q3si5rc3iz6z8fypx0c
NarSize: 3067664
References: g5a8ba31a1ghw8656ybh3qw2vk324j90-glibc-locales-2.39-52
Deriver: 27flvk0dww5s3dbqk6bbsx7zipyfk936-glibc-locales-2.39-52.drv
Sig: binary-cache-server.local-1:cfBq0CK1r7DHKJiJ8Yl2iXdBPjs741mt1bDbAqKNu6++QSO2dVjw3hj2kYyL2ytC0yz8WsOYIsPINn/6Ep/tCA==

The Sig: line at the end says it is signed :white_check_mark:

Doing a `nixos-install` with substituters and trusted-public-keys

So I figured I wasn’t giving it the trusted-public-keys. They were in the config I was attempting to build but what I needed to do was pass them on the command line.

nixos-install --option substituters "http://binary-cache-server.local" --option trusted-public-keys "binary-cache-server.local-1:i40stmkBufmdWjmgMn14wAnlvgCscP+LD/KtmVxLWWc="

Install worked with no internet downloads. Seemed to take about 1-2 minutes longer than a normal install which is odd but that is probably something to do with my setup.

Thanks @Nebucatnetzer. @fricklerhandwerk is it worth adding a couple of lines to the official wiki or the old unofficial wiki showing people how to both sign the existing files in their /nix/store and also how to add the two command line options to nixos-install to do offline builds. IMHO this was the missing part of the puzzle.

7

Snap - Just saw this :+1

8

It should be fine if it is in the global config like this: nixos/modules/misc/common/default.nix at 4b5ed0dd29d9572a3bbf06e232e580d93d235299 · Nebucatnetzer/nixos · GitHub

9

Well, my configuration.nix contains the trusted-public-keys and substituters entries as per your git file, but they don’t seem to be working.

Currently in order to do a nixos-install, a nixos-rebuild or a nix-build I need to add the two options to the command line and then they work fine. I’ve clearly missed something!

Update 1 - checking what the following config produces; 
  nix = {
    settings = {
      substituters = [
        "http://binary-cache-server.local"
      ];
      trusted-public-keys = [
        "binary-cache-server.local-1:i40stmkBufmdWjmgMn14wAnlvgCscP+LD/KtmVxLWWc=" # this came from cat /var/cache-pub-key.pem
      ];
    };
  };

$ nix --extra-experimental-features nix-command show-config substituters
http://binary-cache-server.local https://cache.nixos.org/

$ nix --extra-experimental-features nix-command show-config trusted-public-keys
cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= binary-cache-server.local-1:i40stmkBufmdWjmgMn14wAnlvgCscP+LD/KtmVxLWWc=

I can see that the my options have been appended to the defaults rather than overriding them. Perhaps it is a network issue as I have the machine offline for testing and to make sure it is doing what I expect.

Update 2 - timeout / order issue

Definitely a network / timeout issue. It tries to use https://cache.nixos.org/ first and it takes over 15 minutes to timeout, but then it falls back to the local binary-cache and works. So just trying to workout how to remove https://cache.nixos.org/ or at least reorder them.

Update 3 - use lib.mkForce to control the order

lib.MkForce did exactly what I wanted and allowed me to specify what I wanted and over-ride the defaults. :white_check_mark:

{ config, lib, pkgs, ... }:
{
  nix = {
    settings = {
      substituters = lib.mkForce [
        "http://binary-cache-server.local"
      ];
      trusted-public-keys = [
        "binary-cache-server.local-1:i40stmkBufmdWjmgMn14wAnlvgCscP+LD/KtmVxLWWc=" # this came from cat /var/cache-pub-key.pem
      ];
    };
  };
}

$ nix --extra-experimental-features nix-command show-config substituters
http://binary-cache-server.local

$ nix --extra-experimental-features nix-command show-config trusted-public-keys
cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= binary-cache-server.local-1:i40stmkBufmdWjmgMn14wAnlvgCscP+LD/KtmVxLWWc=

Offline install is now working as expected.

Update 4 - Works well, but not for everything

Been testing quite a bit, many packages work just by adding them to the binary-cache-server but some, like fish seem to still require access to the Internet to download tarballs even when they are installed and working on the binary-cache-server. It’s a little hard to tell which packages will and won’t work just by looking at search and I guess this is a fairly normal state of play.

10

I added two sections to the official wiki and removed the reference to extra-substituters which doesn’t seem to have been available since 22.05.

1 Like
11

Yes the main cache gets added everytime. An easier way to configure the order is to adjust the priority of the cache. Maybe the order in the list doesn’t even matter when the caches are sorted by priority anyway.

IIRC the main cache has a priority of 41 so if you set it lower your cache will be queried first.

For this I don’t know the answer, maybe it requires the source in order to calculate the hash but then doesn’t build it because sees the output hash is already present.
Maybe you can push the source to the cache?

1 Like
12

Thanks I’ll also try “priority” as per your example config tomorrow. :+1:

13

I haven’t quite got to testing priority yet, but after about 30Gb of builds today, my log shows that I have still used less than 500Mb of Internet traffic. Awesome :hugs: :hugs: :hugs: so chuffed!

1 Like