How to serve a binary cache separate from host's nix store?

Help
1

I’ve been hoping to explore option for hosting a local binary cache to reduce my network bandwidth usage.

I have a nas whose root is on a small ZFS-mirrored pair of NVME drives (500G), with the main storage being a reasonably large set of HDDs also on ZFS (~20T).

I’m hoping to be able to use the HDDs to host a binary cache for a number of machines on my LAN (aarch64-darwin, aarch64-linux, x86_64-linux); this way I don’t run out of drive space on the NAS’s relatively small root.

However, I’d really like to keep the NAS’s root on the NVMEs for faster startup times and local operations.

Is there a “right” or best way for me to do this? I’m wondering about the local-overlay-store option, for example.

2

I think the fastest way for you to get to your goal is to mount the nix store in a container or a vm in /nix/store and run a naive binary cache implementation like nix-serve while having a reverse proxy in the immediate outside layer to expose this with the port and listen you want.

Ultimately the binary cache (http) protocol is just like any other http protocol so you can modify any of the servers like nix-serve to suit your need.

3

Interesting, thanks for your thoughts. So basically just using a container or VM to emulate facilitate having a second nix-daemon and store on the same machine?

I had planned to serve over ssh with sshServe and a shared key stored in agenix – I suppose I would just have the container / VM listen for SSH on a separate port and configure that in the clients’ ssh_config.

4

Looks like ncps may be another option: GitHub - kalbasit/ncps: Nix binary cache proxy service -- with local caching and signing.

Downsides: requires unauthenticated PUT to add to the cache, apparently nix fails if cache is not available (not an ncps problem)

2 Likes
5

Maybe harmonia works for you?
I haven’t tried it but from the README it might be possible to put the store in another directory?

# Allow to override the store path advertised in /nix-cache-info
# virtual_nix_store = "/nix/store"
# Allow to serve the nix store from a different physical location
# Default: empty
# Example: if you use `nix copy --store /guest` to populate a store than configure:
# real_nix_store = "/guest/nix/store"