Running nixos-rebuild across platforms

Help
1

I’m using a Flake to define a NixOS system for a Raspberry Pi. By setting:

  nixpkgs.buildPlatform = "x86_64-linux";
  nixpkgs.hostPlatform = "aarch64-linux";

I’m able to cross-compile NixOS on my x86 box, i.e. I can generate an SD card image like this:

nix build .#nixosConfigurations.sandy.config.system.build.sdImage

Which boots up fine on the Pi.

Now the issue is that nixos-rebuild seems to want to run aarch64 binaries on my build host:

❯❯  nixos-rebuild switch --flake .#sandy --target-host 192.168.0.81 --use-remote-sudo
/nix/store/svk2dgsb1j87z6sqb3c5257d6n0l4vf8-nixos-rebuild/bin/nixos-rebuild: line 443: /nix/store/7hwrcf22m39i3mx7qxwbjjidnz2in777-coreutils-aarch64-unknown-linux-gnu-9.6/bin/mktemp: cannot execute binary file: Exec format error

I have a feeling there’s a way to work around this by just getting my x86 host to support running Arm binaries with binfmt_misc hackery, but I don’t really wanna do that if I can avoid it. (My x86 host system is not NixOS, plus this is inefficient and generally unsatisfying).

Also, I tried setting --build-host to point to the Pi as well, this is also not really a solution but I wanted to see what would happen. Turns out I still got the Exec format error which is a bit surprising…

Does anyone know what I’m missing here?

1 Like
2

I run into a similar error with with my Raspberry
.../bin/bash: Exec format error
with a config that built fine 2 weeks ago

More similar problems multiple packages: crash on aarch64-linux · Issue #399133 · NixOS/nixpkgs · GitHub

3

Left a comment on the issue multiple packages: crash on aarch64-linux · Issue #399133 · NixOS/nixpkgs · GitHub.

I’ve encountered the same error on an Intel laptop, might be due to some systemd hardening options present in the nixos modules I’ve used - grafana and postgresql are pretty popular, so they should have hardening options enabled. I’ve created my own module for qbittorrent-nox by blindly copying some of the options from other services. Initially it worked fine, but then suddenly started crashing with the “exec error”.

Unfortunately my pi ended up dying due to a broken sd card, so I didn’t get the opportunity to investigate this further.

1 Like
4

I’m getting this error too. Cross compiled qbittorrent-nox on my x86 machine to run it on the raspberry pi.

Did you find a solution?

5

In the end I did indeed just set up binfmt_misc on the host. I strongly suspect the actual issue is a bug where some aarch64 logic “leaks” into the x86 host and nobody has noticed because most NixOS devs also have binfmt_misc set up.

FWIW I also now discovered that at least for my use case, it’s actually preferable to just do everything via binfmt_misc. That is: I set my host up to be able to build aarch64 packages that way and then I just abandoned the build platform stuff and set system = "aarch64-linux". I thought this would be extremely slow but in practice it’s a lot faster because it means the public build caches work! Obviously this only works for me because my usecase doesn’t require any nontrivial software for the Arm box that isn’t cached in nixpkgs.

1 Like
6

Thank you bjackman, this made it work.

This on the build machine in configuration.nix:

  # Required to build aarch64 packages
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

And this in flake.nix for the raspberry pi:

system = "aarch64-linux";
7

Oh yeah - and I should highlight the reason I changed to this is because the “binfmt_misc hackery” I referred to in the OP turns out to be extremely easy to set up for aarch64, even on other distros (e.g. I think on Ubuntu it was a single apt install away).

8

I really discourage using binfmt if you don’t actually have to use it. And you shouldn’t need to just to nixos-rebuild --target-host to a different platform. On unstable, it seems to me like it automatically determines if the reexec of the newer nixos-rebuild works and falls back to not doing that, so you don’t have to do anything. On older versions (probably including 25.05), you would need to pass --fast to prevent it from reexec’ing.

1 Like