I’m doing research aiming to deploy a build server. One of the things I’m not getting is the ability to have a private and self-hosted binary cache.
The idea is that after a push to the main flake, the build server would:
Pull the new commit
for each system, check architecture
a. build locally if arch match
b. either virtualize other architectures or send to a secondary build server with matching arch
check packages dependencies and build anything not available in the official binary cache
upload built packages to a custom binary cache
deploy/push updates to all machines
I couldn’t find any solid documentation on self-hosting a binary cache. I’m aware of places like cachix, but I prefer self-hosting my own data
At the moment I just use an SSH based cache but that can be a bit tricky when you do garbage collection because the hosts store is the actual cache.
harmonia looks like it could solve this problem.
I wasn’t aware of either of those. I’ll have to dig and explore.
And yeah, I can imagine using a bare SSH cache would come with significant drawbacks. I’m not even sure how it would handle the multi-arch thing since it’ll have to host packages for x86_64-linux, aarch64-linux and aarch64-darwin
Hi, I have a build server that periodically rebuilds multiple NixOS hosts, basically to have auto upgrades with some staging/production environments.
This server use its local nix store as cache for itself and its target hosts, for each target host I’m doing a nixos-rebuild build --fast --flake /path/to/repo#hostname which generates a result file symlinked in /nix/var/nix/gcroots/auto/, so the build is kept in cache when doing nix-collect-garbage on the build server.
This build step offers another advantage, I can then check if the target host will need to reboot or not, so I will run nixos-rebuild boot|switch ... --target-host accordingly. To check if a reboot will be needed I compare derivations like this:
I do something similar. I use one of my servers as a build server (using nixos-rebuild) but do the evaluation locally on my laptop. When I’m ‘done’ I will probably do the same as you.
One thing I can say is that cross-building like this works fine (at least for raspberry pi) using boot.binfmt.emulatedSystems = ["aarch64-linux"];
That’s not a problem at all.
My cache server is a Raspberry Pi and I push configs from two x86 systems to it.
Building on the Pi for the x86 machines obviously doesn’t make sense but caching works fine.
I’d think the built-in ssh/object store cache would be the most ideal for your use case. You don’t need multi-tenancy or any of the other fancy features the alternate implementations add, so you might as well go with the one that is battle-tested.
You’d just need to spend like 20 minutes thinking about the details of cache expiry, which you need to do anyway, and reject the temptation of nix-collect-garbage -d.
Most of the complexity of your setup stems from the push-based deployment workflow, which is 100% on the CI design side, not the cache. But even all that is not very complex, nix has all these arch thoughts built-in, you just need to configure the ordering of the host’s binary caches, pre-seed them, and use the target as the build host for nixos-rebuild so that it pulls data from the cache and not the “real” build host.
I’m currently doing research, and I’m still torn between a binary cache (most probably attic[1]) and the built-in store of the build server.
One workflow question: What’s easier, doing something like nix copy ... from the various aarch build servers[2] or giving the builders push acls on a binary cache like attic?
I feel like building the environment using built-in nix tooling would be like creating my own binary cache system from scratch, with the drawback of more managed complexity and time, but the result would be a more robust and battle-tested setup
Since the main major issue with attic seems to be storage related and goes away as soon as you use an S3-compatible storage backend, It looks very tempting considering that I already have a MinIO server. ↩︎
cross-building for aarch64 on x86_64 never worked reliably for me ↩︎
To be honest, nix-collect-garbage is the only way I know for nix store cleanup. Any general docs that would help with what I’m trying to do? I feel like I’m venturing into an area where clear docs are scarce.
I found attic much more user friendly to use than the built in tools.
It even has nice helpers like monitoring your store and automatically upload it to the cache so you don’t have to think about it.
I would love to use it if it wasn’t broken for me.
