Is there any way to set extra substituters and have Nix just ignore them if they’re offline? I’ve tried adding them via nix.binaryCache, but if the first one(s) it tries are offline, it spends forever retrying and eventually aborts the command entirely after finally timing out.
I’m trying to improve performance for the multiple NixOS machines I have on my home network by having them share binary caches, but Nix handling offline substituters this way makes things much worse, not better.
I think the way to do it would be to override the nix config and/or remove the substituer from nix.conf entirely. You can set nix config directly in the NIX_CONFIG variable on recent enough versions of unstable nix for example.
On a related note, I think nix#5324 could help to alleviate this in the future by allowing you to query more substituters in parallel. Combined with nix#5109 it should be enough to solve your problem by simply returning on the first success.
I don’t fully understand those PRs, but looks like nix#5109 doesn’t solve the problem on its own so I guess still I need to wait for nix#5324?
$ nix --version
nix (Nix) 2.5pre20211007_844dd90
$ nixos-rebuild switch --option extra-binary-caches 'http://192.168.0.123:11111 http://192.168.0.124:11111'
warning: error: unable to download 'http://192.168.0.123:11111/xa2vkwj77s9p55ykxwib3cm1qbd5v94b.narinfo': Couldn't connect to server (7); retrying in 288 ms
... (many more messages like the above)
error: unable to download 'http://192.168.0.123:11111/nar/8yx0hfgs4hgck2i4aqhp8bvr7zq54whs.nar': Couldn't connect to server (7)
I’ve worked around it for now by using a custom bash function that first pings the hosts with a very quick timeout and then adds the argument for just the binary caches that are currently online, but surely there’s got to be a better way.
There’s also Bug: The build fails if a build machine/cache is offline · Issue #3514 · NixOS/nix · GitHub, but it’s tricky to fix it.
Was the intent to bulid from source if cache is down or should that be an error?
For binary caches like cache.nixos.org it’s more likely the latter, but for other caches it might be the former.
Ah, #3514 sounds like exactly the same problem!
My intent would be for Nix to try the caches in order until it either 1) finds one that’s online or 2) falls back on cache.nixos.org.
I don’t want an error if any of my local caches are offline unless they’re all offline, including cache.nixos.org. I also don’t want it to spend a long time trying to reach the caches because then commands will always take a long time unless the first cache listed is online, which defeats the purpose of having them.
My primary use case is having ~5 machines running more or less the same NixOS config and not wanting each one to have to re-download ~20GB of packages when I do a nixos-rebuild if there’s already a machine with them downloaded.
Im running into the same issue, ive got caches on my local network not available outside.
For me, I would not like it to fail, even if all the local caches are offline.
But I would like it to fail if cache.nixos.org is offline.
FWIW, it fails a lot quicker if you use hostnames instead of ips.
so thats nice, and works well for remote builders, which fails quickly and gets out of the way if they are offline.