Nix store cache in local network

Help
1

I was wondering what would be the best way to keep and serve a nix store cache in my local network to be able to more quickly update local machines which all run on a GitHub hosted flake nixcfg/flake.nix at 9d15eecf1c45cd599e7bdc5459a6c4be8b4ad4d1 · pbek/nixcfg · GitHub.

The cache would have the highest priority, so derivations will not be downloaded from the internet if they are available locally.

I had several ideas:

  1. I tried gathering all derivations of the current generation and pushing them on an attic server in the current network (running in docker on my Synology NAS) with nixcfg/push-all-to-attic.sh at 9d15eecf1c45cd599e7bdc5459a6c4be8b4ad4d1 · pbek/nixcfg · GitHub
    • For some reason that process is terribly slow, often only kB/s on my GBit network (attic even has a Postgres DB). :frowning_face:
  2. Building a docker image of the flake on GitHub with the help of nixos/nix:latest, docker pull it to my NAS and then serve /nix/store in the local network somehow.
    • I wasn’t able to build the flake in docker yet, because there is no nixos-rebuild and I was able to achieve it with nix itself. :frowning_face:
  3. rsync avh --delete /nix/store via SSH to my NAS from the first machine that downloaded the new derivations from the internet and serving it from the NAS somehow.
  4. Leaving one NixOS machine running and serving the nix store via Serving a Nix store via SSH - Nix Reference Manual
    • electricity is expensive nowadays, I already have two NAS and several Raspberry Pis running :sweat_smile:

Any ideas or advice?

3 Likes
2

I have a machine on my LAN running GitHub - edolstra/nix-serve: A standalone Nix binary cache server. It also runs a bunch of other services, and pulls around 10-15W when not doing anything heavy.

I do all my building on there and all other machines use it as a substituter. Works great!

The only thing to watch out for is negative caching. If another machine tries to build something and finds it not in the local cache, it won’t query the local cache again for some time (even if you build the artifact there). You can reset the lookup cache by running

rm $HOME/.cache/nix/binary-cache-v*.sqlite

2 Likes
3
1 Like
4

I have a Hydra instance set up on a server on my network configured with:

store_uri = file:///var/lib/nixcache?secret-key=/var/lib/hydra/nix-key&compression=zstd

And an nginx server that serves that /var/lib/nixcache directory. Works out great as a local cache. Downloads are fast and zstd decompression is nice to have.

That said, I have been eying attic. It seems really nice. Just needs hydra integration maybe?

2 Likes
5

You can simply use Nginx on a server as a reverse proxy that will keep the files

I wrote a guide here Solene'% : How to make a local NixOS cache server

You could also use peerix to exchange story entry between machines on your local network, but it’s unmaintained, and sometimes doesn’t work well Solene'% : Local peer to peer binary cache with NixOS and Peerix

3 Likes
6

Yeah, thank you. Some day I will also need to put a NixOS box up in my LAN to always run :sweat_smile:
Thank you for the info about negative caching, I didn’t know that!

1 Like
7

Hah, that is nice too! Looks like it also needs NixOS, or can it run in a Docker container?

1 Like
8

Thank you! Does Hyrda also run in a Docker container?
Attic is great, you can have multiple caches. Set them public or not public. Does deduplication on block level over all caches. Has a garbage collector. AND I have it running in a Docker container!

1 Like
9

I’ve never tried to use hydra in a docker container. NixOS already has a good module for it so I just use it: NixOS Search

In fairness, I do use it in a NixOS style container, but that’s just for file system isolation purposes.

1 Like
10

Thanks a lot for your blog posts! A proxy server that does the caching! Why didn’t I come up with that? :laughing: You are keeping stuff for 60d, right? I wonder if I can directly build a Docker image from your config, or else I’d need to back-port it to be used with the conventional Nginx Docker image.

1 Like
11

Thank you, one of these days I’ll simply put up a local NixOS server too, would make some things a lot easier. But Attic also did manage to build the atticd for a Docker image in attic/build.yml at b1fb790b5f2afaaa1b2f7f18979b8318abe604bb · zhaofengli/attic · GitHub.

1 Like
12

I rewrote the nix-nginx config as nginx config in nixcfg/nginx.conf at b8ed727902e99becfb4fb51277b6081cd853fa33 · pbek/nixcfg · GitHub and replaced https://cache.nixos.org/ with my local cache server, which runs on docker in my LAN.

Works great!

Of course nothing is ever found in the /var/public-nix-cache (/var/nix/public-nix-cache in my config), but after the 1st time a file is accessed it comes from cache.

@Solene, what purpose had /var/public-nix-cache for you? Did you put something there?

2 Likes
13

The relevant part of my docker-compose.yml looks like this:

  nix-cache:
    image: ghcr.io/pbek/nixcfg-nix-cache-nginx:latest
    restart: always
    ports:
      - "8282:80"
    volumes:
      - /volume1/nix-cache:/var/nix
1 Like
14

I don’t remember exactly, I found that from another idea of caching, this file shouldn’t be cached for long IIRC

1 Like