Setting up remote builder cross-architecture

1

I have a Raspberry Pi (so aarch64) and a laptop (x86_64), both running NixOS. I want the Pi to use the laptop as a remote builder, as it is much more powerful. Currently, however, when I try and run nixos-rebuild switch --flake .* --max-jobs 0, I get the error:

Failed to find a machine for remote build!
derivation: …
required (system, features): (aarch64-linux, [])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux, aarch64-linux], 1, [nixos-test], [])

I presume I need to set something else up to allow the x86_64 machine to build for the Pi? Currently the config for the Pi contains

  nix.buildMachines = [ {
    hostName = "kernighan";
    systems = ["x86_64-linux" "aarch64-linux"];
    protocol = "ssh-ng";
    maxJobs = 1;
    speedFactor = 2;
    supportedFeatures = [ "nixos-test" ];
  } ];
  nix.distributedBuilds = true;
  nix.extraOptions = ''
    builders-use-substitutes = true
  '';
2

Immediately after posting this I discovered this command, which I thought had things working perfectly—things began building on the other system.

nixos-rebuild switch --flake .* --max-jobs 0 --builders "ssh://ritchie aarch64-linux"

However, I then get the new error:

error: build of '/nix/store/hh8fd1jx3m92dv0izi37yxsc05fzrjmx-homeassistant-2023.8.2.drv' on 'ssh://ritchie' failed: error: a 'aarch64-linux' with features {} is required to build '/nix/store/hh8fd1jx3m92dv0izi37yxsc05fzrjmx-homeassistant-2023.8.2.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
error: builder for '/nix/store/hh8fd1jx3m92dv0izi37yxsc05fzrjmx-homeassistant-2023.8.2.drv' failed with exit code 1;
3

Add this to your builder config:

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

That allows your builder to run aarch64 binaries transparently with QEMU emulation. Emulation is quite slow.

For better performance, you want cross-compilation, where the compiler is running on the builder’s native architecture, but producing a binary for the target architecture. Nix has good support for cross-compilation. But a cross-compiled derivation is different than a native or emulated one, so it does not cleanly substitute. For example, if you cross-compile vim but your system wants a natively built vim, then your system will ignore the cross-compiled version in your nix store and try to build it natively (or via a remote builder with emulation).

The practical implication is that you either need to pick and choose which packages to cross-compile, or you need to cross-compile your entire system (which results in a lot of cache misses).