I can ping my remote builder’s nix store but attempting to use it for building does not succeed.
What could be wrong and how I fix this such that I can build using my remote builder for building?
While I’m here, is it possible to make Hydra use an ssh config?
Logs:
$ nix store ping --store ssh://builder --extra-experimental-features nix-command && echo ok
Store URL: ssh://builder
ok
$ nix-build -j0 --builders 'ssh://builder' shell.nix
these 3 derivations will be built:
/nix/store/nq75c1k8n1q3363cspg7m3vv8i7jj9fp-stdenv-linux.drv
/nix/store/1xlw65h582fqplj8d8n80n2jylz0qrrq-sudo-1.9.12.drv
/nix/store/sqxqj84fkhg5k4q65amg0icakhkxp3lh-nix-shell.drv
error: unable to start any build; either increase '--max-jobs' or enable remote builds.
https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html
SSH Config (via jump1 to a VM running on jump2):
host builder
user builder
IdentityFile ~/.ssh/my_key
ProxyJump jump1,jump2
Hostname 127.0.0.1
Port 9999
When you run nix store ping, Nix will setup an ssh connection to the store with the user that is executing nix store ping. But when you are running a build the connection will be made with the user running nix-daemon (usually root). So you need to make sure your ssh config is usable by root too. The easiest way to verify this is probably by running sudo ssh builder.
Also, adding the option -v (or possibly -vv) to the nix-build command will cause Nix to print out more information about the ssh connection initiation.
I made my ssh config available to root (nix-daemon does indeed run as root on my laptop) and can $ sudo ssh builder succesfully.
However, attempting to use the remote builder still fails (same error message as previously),
-vvv reveals the following, which I am not sure how to interpret:
starting build hook '/nix/store/pnsp9jf5grg4aann9zm1ca49ijwdmmm0-nix-2.11.0/bin/nix __build-remote'
got 0 remote builders
hook reply is 'decline-permanently'
killing process 55109
I tried both with /root/.ssh/config and programs.ssh.extraConfig (as this gist suggests) but with no luck.
Based on the error message my guess is that the execution gets here but I do not know why getMachines() is empty.
Are the derivations you are trying to build using another system than your local machine? If they are, and your builder machine supports that system, you need to tell Nix about it like this:
--builders "ssh://builder aarch64-linux"
I just used aarch64-linux as an example here.