Local Binary Cache - Questions

Help
8

Never mind… I’ve solved the problem mentioned in my post above, and have arrived to a solution that I like, though it was not what I had originally wanted.

The solution is to use task spooler within the post-build-hook script, which prevents the nix copy commands from blocking a nixos-rebuild:

#!/bin/sh
set -eu
set -f
export IFS=' '

CACHE_HOST="host.nix-cache.cyn.internal"

if [[ -n "''${OUT_PATHS:-}" ]]; then
   export TS_MAXFINISHED=1000
   export TS_SLOTS=10

   echo "Uploading $OUT_PATHS"
   printf "%s" "$OUT_PATHS" \
   | xargs ts nix copy --to "ssh://tk@$CACHE_HOST"
fi

I found the solution in this thread.

I’ve decided against using the remote builder for all of my homelab hosts, however - I don’t think that’s necessary, and while my build host can handle it, the VM I’m using for the nix cache can not.

I didn’t realize until some testing, but populating the nix cache with receiving nix copy from a fresh nixos-rebuild on a host makes the nix-daemon go absolutely wild - is extremely expensive and taxing on the CPU.

I was not even able to populate the cache initially because my VM had a pitiful 2CPUs and it seemed frozen. I tried harmonia’s new rust-based nix daemon which sped up the process dramatically and made it possible, but I still ended up assigning two additional CPUs for a total of 4 to make it faster.

I think this is fine as it is for now, here’s the relevant parts of my nix config:

I’m still curious, if there would be better strategies of caching for a homelab - the idea was to have the nix-cache server to be this anemic weak VM with a bunch of storage. It’s possible that some of the issues I’m having are harmonia-related, but I’m suspecting that I’d have similar problems with the other binary cache solutions.