Hi group,
I want to build aarch64 packages using a remote builder.
I have a non-aarch64 machine baseline configured as a remote builder, meaning I can just run ssh builder nix-store --version.
The machine does not have NixOS installed but nix and nix-daemon running.
I am stuck with where to add boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; equivalent as I have no configuration.nix. Do I have to set up a qemu vm with NixOS inside ?
Checkout the config section of the binfmt.nix module in NixOS. It’ll give you an idea of how to implement the same functionality on a non-NixOS system.
Hi @ElvishJerricco, thanks for giving a good hint here, as far as I understand it, the remote builder running on arch linux I would have to install the binfmt packages provided in arch linux’ aur ?
Or do you mean implementing the config boot.* attribute in like home-manager, cause I’m using home-manager on that remote builder ?
Still lost somehow.
Edit: Found this as a further reference. If there is an alternative approach to try I’d be happy to contribute to it, especially the binfmt.nix based one for nix-only (but non-NixOS) setups.
Following NixOS on a Raspberry Pi: creating a custom SD image with OpenSSH out of the box | Roberto Frenna’s description I installed the needed binaries on my arch linux box and trying with the minimal sd-image.nix given in Robert’s article triggered a build using
nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage --option system aarch64-linux --option sandbox false -I nixos-config=sd-image.nix
but got another error: /run/current-system/systemd/bin/systemctl is called in udev rules but is not executable or does not exist.
Regarding my last comment describing just a special case where the remote isn’t necessarily a physically different one but running inside qemu.
In most other cases installing / configuring QEMU binfmt wrapper on the other hence remote machine should just work, see this comment on github as well. Bc we are talking about non-NixOS remotes you don’t even need the configuration.nix, qemu.nix and overlay changes mentioned there.
If you’re trying to use GitHub actions and Cachix to build aarch64 derivations, you can do this with
uses: docker/setup-qemu-action@v1
or similarly, run the following installation on ubuntu:
sudo apt-get update -q -y && sudo apt-get install -q -y qemu-system-aarch64 qemu-efi binfmt-support qemu-user-static
Then do your build with
nix-build --option system aarch64-linux --extra-platforms aarch64-linux
It does work for me but to be precise you’ll need Ubuntu 20.04+ (old qemu version are failing because of missing syscall) and in order for your builder process to access quemu you must set --option sandbox false
Just had the same problem (Ubuntu 22.04 on one PC, 24.04 on another), but I didn’t wish to disable the sandbox (that causes other problems for me):
Instead I did something similar as NixOS does for itself:
First the required packages:
sudo apt install binfmt-support qemu-user-static
The qemu-system-XXX and qemu-efi packages mentioned above are not necessary, those are intended for a full VM. Also note that the static version of this package is necessary and you should use that instead of the regular qemu-user (without -static) package as the non-static package is a nightmare to get working inside the sandbox.
Then add to /etc/nix/nix.conf the extra-platforms option as mentioned above. But in addition to that be sure to extend extra-sandbox-paths with /usr/libexec/qemu-binfmt and the absolute path of the symlink for every architecture you care about in that directory. This can be obtained with e.g.:
$ realpath /usr/libexec/qemu-binfmt/arm-binfmt-P
/usr/bin/qemu-arm-static
For me this leads to this snippet in /etc/nix/nix.conf:
extra-platforms = aarch64-linux armv7l-linux i686-linux
extra-sandbox-paths = /usr/libexec/qemu-binfmt /usr/bin/qemu-aarch64-static /usr/bin/qemu-aarch64-static /usr/bin/qemu-arm-static
Explanation: every single file mentioned directly and indirectly from the interpreter line in /proc/sys/fs/binfmt_misc/${your_binfmt_architecture} needs to be explicitly added to the sandbox’ permitted paths. This includes the full set of symlinks and final executable but also every used shared library (which is why static linked executables are preferred).