Way to build nixos on x86_64 machine and serve to aarch64 over local network?

So the question is kinda long and has few conditions. Here’s the breakdown:

What I have:

  1. A regular x86_64 PC
  2. An aarch64 Raspberry Pi 4B
  3. A local network connecting the two

What I want:

  • Build nixos on PC, because even with emulated build it is faster
  • Deploy ‘built’ system and all generated binaries to RPi
  • Do so using ‘only’ local network, to reduce latency and bandwidth.

What is working so far:

  • Building aarch64 packages on PC (NixOS on ARM - NixOS Wiki)
  • The command to do so from PC (nixos-rebuild boot --flake .#pi -v --target-host root@pi --build-host localhost)
  • SSH acces between two machines (authorizedKeys on Pi)

What is not working:
As soon as all the packages are built, I get a prompt for password on remote user, but actual package transmission fails with below error:

building '/nix/store/vfhblsqxb9w3na1s2wig3aw4r5cqr8al-nixos-system-pi-22.05.20220415.5181d59.drv'...
Running nix-copy-closure with these NIX_SSHOPTS:  -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.88ylG4/ssh-%n -o ControlPersist=60
$ nix-copy-closure --to user@pi /nix/store/m4dxpqfxfcxd0j6gqw3nbcdck1fdyz1d-nixos-system-pi-22.05.20220415.5181d59
(user@pi) Password: 
Could not open a connection to your authentication agent.
copying 16 paths...
copying path '/nix/store/1v8g8b4f6az62l4zx8adf03dryaj452n-nixos-manpages_fish-completions' to 'ssh://user@pi'...
error: cannot add path '/nix/store/1v8g8b4f6az62l4zx8adf03dryaj452n-nixos-manpages_fish-completions' because it lacks a valid signature

I tried deplyoing via root@pi, but getting same error. I believe I need to setup binary cache and generate signatures for it (here, here, here, here and here). But my n00b status is becoming apparent and I need some help if fitting the pieces together.

I explicitly do NOT want to share my cache over the internet, just the local network (192.168.x.x), I have setup hosts file on both PC and PI, and it is working at least for SSH. But the last piece is proving difficult for me.

Any help appreciated.

1 Like

The default setting in nix.conf is that require-sigs = true. So every store path you want to add to your /nix/store and you didn’t build yourself needs to be signed. That means the raspberry will refuse any path that isn’t from cache.nixos.org, because it knows of no other keys by default.
I’d recommend you read the manpage for nix.conf, in particular the entries about require-sigs, secret-key-files, and trusted-substituters.
There’s also the nix store sign command that you can use to sign paths after you already built them, but it’s generally best practice to have secret-key-files on your build machine and add the public key to the trusted-substituters of any machine that you might want to copy to.

3 Likes

Thanks for pointing in the right direction @manveru!

I was able to get it working with secret-key-files on PC and trusted-public-keys on Pi.

I specifically did not want to have PC as trusted-substituters for Pi because I intend to always push my deployments from PC to Pi, and substitutes are (per my knowledge) sources to pull from. This way, if I avoid running nix-serve from my PC and exposing port 80/whathaveyou while still benefiting from its raw compute.

Overall, quite happy how this turned out. Things are clean, and apart from imperatively deployed signing keys, everything is in single declarative config repo. Only remaining wrinkle is having to deploy via root@pi, but I’m hoping to trusted-users will be do the trick (ran out of time for now :slight_smile: ).

For any future viewers, this is the final command that worked:

cd /etc/nixos && nixos-rebuild boot --flake .#pi -v --target-host root@pi --build-host localhost

trusted-users did not work, so now I’m kinda unsure how to take this forward. I was hoping to keep the whole thing to bare minimum and get it working with nixos-rebuild but perhaps I’ll have to make use of nixops or something like it after all.

Just found your post while searching for something different.
The command that I’m using is the following:

nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host $fqdn --flake ".#$host"

I reckon you’re missing --use-remote-sudo