Nix-serve on Ubuntu for my whole LAN?

Help
1

I apologize if this question isn’t purely Nixotic, but I’m hoping someone can help with this.

I’m learning NixOS using virtual machines on my laptop, but I’m quickly tiring of downloading the same packages over and over again, so I’d like to set up nix-serve on my LAN server. I followed the instructions as best I could on the Binary cache page, but the docs on that page seemed to assume that nix-serve was being run on a NixOS box. I don’t (yet) have that luxury, so I’m trying to set up a systemd.service(5) unit that will let me run the service on my LAN that any machine or VM can point to, which will serve cached Nix packages from the LAN server and go out to the Internet to get them from cache.nixos.org if needed (for the laptop or VM).

Here’s the setup:

  • The LAN server, server.zone.domain.tld, is currently running Ubuntu 24.04 LTS. Since it’s hosting several services I use regularly, I can’t “nuke and pave” it with NixOS.
  • server has Nix installed with its store at /nix/store, but isn’t (yet) using Nix for anything on the server itself (so its store doesn’t have a lot in it right now). At the moment, I just want to have a common binary cache (up to a few TB) that any machine or VM on the LAN can access, while I’m learning.
  • server is using Traefik as a reverse proxy to direct requests to services it’s hosting, such as this binary cache, and has a valid Let’s Encrypt SSL certificate for https://nix-cache.server.zone.domain.tld.

Here’s my Systemd service unit:

[Unit]
Description=Nix cache
Wants=network-online.target
After=network-online.target zfs-mount.service
Requires=zfs-mount.service

[Service]
Type=simple
Environment="LC_ALL=C"
Environment="NIX_SECRET_KEY_FILE=/var/local/apps/nix-cache/config/cache-pub-key.pem"
ExecStart=/nix/var/nix/profiles/default/bin/nix run github:edolstra/nix-serve -- --listen 0.0.0.0:5000 --store-dir /nix/store
User=root
Group=root
RemainAfterExit=no
Restart=on-failure

[Install]
WantedBy=multi-user.target

Here’s the Traefik dynamic config file for the service:

http:
  routers:
    nix-cache-router:
      rule: Host(`nix-cache.server.zone.domain.tld`)
      entryPoints:
        - websecure # host machine listens on port 443
      service: nix-cache-service
      tls:
        certResolver: production
        options: default@file

  services:
    nix-cache-service:
      loadBalancer:
        servers:
          - port: 5000 # Nix-serve listening on port 5000
        passHostHeader: true # preserve incoming Host header

Now, when I run nix store info --store https://nix-cache.server.zone.domain.tld, I get back Store URL: https://nix-cache.server.zone.domain.tld. When I curl "https://nix-cache.server.zone.domain.tld/nix-cache-info", I get:

StoreDir: /nix/store
WantMassQuery: 1
Priority: 10

Seems OK, right? It’s definitely seeing a Nix store there. However, when I launch a VM on my laptop, I can ping nix-cache.server.zone.domain.tld with no problem, but when I add this to its /etc/nixos/configuration.nix…

nix = {
  substituters = [
    "https://nix-cache.server.zone.domain.tld/"
    "http://server.zone.domain.tld:5000/"
  ];
  trusted-public-keys = [
    "server:A019<snip>553v="
  ];
};

…I get a bunch of these errors (even though the VM can successfully ping the server):

warning: error: unable to download 'https://nix-cache.server.zone.domain.tld/rxrhw4f5bgari4wl4pirnr41q17baa8g.narinfo': HTTP error 500
response body:
error fetching the narinfo from the store: error fetching the narinfo record: no such table: narinfos; retrying in 338 ms

Here are my questions:

  1. Anyone know what’s causing these 500 errors?
  2. Can a binary cache set up on the (x86_64) LAN server work for an (aarch64) machine/VM on the LAN?
  3. Does nix-serve’s option --store-dir need to point at the LAN server’s Nix store (i.e., /nix/store), or should it point at another folder for the cache (i.e., /var/local/apps/nix-cache/store)? Will pointing it at /nix/store “pollute” the LAN server’s store, should I use it to install something on the server later?
  4. How can I make server the primary machine to fetch Nix packages that aren’t yet installed on the LAN, instead of the VM going out to the main cache? I.e., if a VM on my laptop goes to install zsh, it asks the nix-cache on the LAN server, and if the cache has the package, it returns it; if it doesn’t, the LAN server pulls it in from the Internet, and then returns it to the VM?
  5. What command can I add to my unit file under ExecStop= to safely stop nix-serve (e.g., when the machine reboots)? (Also ExecRestart=)
  6. Is there any way nix-serve could be run as a Docker/LXC container, for people who aren’t yet ready to make the switch to NixOS?

Thanks again, everyone. This distribution is really interesting!