So I rented an aarch64 system on Hetzner and nixos-infected it. I’d like to use my (fairly beefy, lots of space) x86_64 desktop computer to build a NixOS system and copy it across. I think I’ve made some progress setting it up (thanks to resources like this, this, and this). But it still isn’t really working.
My flake looks roughly like this (the whole thing is here):
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
};
outputs = { nixpkgs, ... }:
{
nixosConfigurations.test-nginx = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
{
nixpkgs.crossSystem.system = "aarch64-linux";
# The rest of the configuration...
}
}
}
The full flake also includes a bare-minimum nginx configuration, including SSL certificate generation.
First, a couple of annoyances (that I could work around, but I’d be happy to hear of any better solutions):
grub failed to cross-compile (specifically, /nix/store/hga9z0z3w5fnan247sm1d24ckh9w3gcj-perl5.38.0-Test-utf8-1.02-aarch64-unknown-linux-gnu.drv). Worked around it by using systemd-boot instead.
nixos-rebuild --flake .#test-nginx switch --target-host root@neeman.me failed trying to execute an aarch64 binary on my local machine. Worked around it using nixos-rebuild build and then nix-copy-closure to copy the result, followed by ssh’ing into the remote machine and manually running switch-to-configuration
The good news is that this process delivers a working system. I can ssh to it and do usual bash-y things. I can reboot it and it comes back up. The weird thing that I’m stuck on is that my cross-compiled nginx’s https doesn’t work: curling it gives “curl: (56) HTTP/2 stream 1 was reset”. Http does work, though.
I built almost the exact same system (changing system = x86_64-linux to system = aarch64-linux and removing the nixpkgs.crossSystem line) on aarch64 without cross-compilation and nginx worked just fine. On both the locally-built and cross-compiled system I tried running the locally-build and cross-compiled nginx binaries by running them directly from the nix store. It doesn’t matter where the underlying system was built: the cross-compiled nginx binary fails to serve https.
Does anyone have any advice on what to look at next? Any other tips for cross-compiling nixos systems?
For starters, try to enable all nginx debug logging you can find and then compare the native/cross version output. And minimize your config (what about plain http?)
If that doesn’t help, strace/ltrace and see if you can spot a difference.
Seems like this might yet be another subtle cross bug. Consider filing it on the GitHub issue tracker as well.
At the risk of stating the obvious: you realize you don’t have to cross compile right? Although cross fixes are useful in any case.
Thanks for the tip! Googling some of the debug logs sent me here, where it seems like cross-compiling nginx is just not supposed to work.
Could you elaborate? If you’re mean the boot.binfmt.emulatedSystems route, I was hoping to avoid that because eventually this NixOS config will involve compiling some custom software; that’s pretty slow under qemu…
The weird thing that I’m stuck on is that my cross-compiled nginx’s https doesn’t work
if you have the store path to the broken nginx, run nix path-info -r /nix/store/...-nginx-aarch64-unknown-linux-gnu-.../. this gives the runtime dependencies of nginx, and if it’s cross compiled right they should all have that -aarch64-x-y-z suffix. it’s a common packaging problem that a program might need, say, python available at run time but also build time, and the version it sees at build time makes it into the runtime.
If you’re mean the boot.binfmt.emulatedSystems route, I was hoping to avoid that because eventually this NixOS config will involve compiling some custom software; that’s pretty slow under qemu…
by the way, you can use that cross-compiled config but also enable boot.binfmt.emulatedSystems. i don’t recommend that as a long-term thing because it ends up sloppy quick, but for that grub failure (Test-utf8 failing to cross compile), it should fix it (Test-utf8’s build process involves running code that was compiled for the host system – binfmt will handle that transparently, invoking only that one binary in qemu).
I run NixOS on a few embedded systems. Here are two examples that may help you:
The first example is my router, which runs on a Raspberry Pi CM4 board. This is an aarch64 platform. For this device, I’m not running many custom builds, so I just set system = aarch64-linux and build it on my x86_64 desktop. I added aarch64-linux to boot.binfmt.emulatedSystems, too. I can remote deploy with nixos-rebuild switch --target-host <hostname>. System builds are generally snappy because most packages are available from the cache, even for aarch64.
One notable exception is a kernel patch I added to fix the LEDs on one of the ethernet ports. (Super important.) I set up the kernel to cross-compile, but left the rest of the system as native aarch64. You can generally do this by pulling packages from nixpkgs.legacyPackages.<build-system>.pkgsCross.<cross-system>. Natively compiled and cross-compiled packages can coexist in the same system.
(Later, I refactored that patch so that I only have to build one small kernel module and can reuse the cached kernel build. I couldn’t figure out how to cross-compile the kernel module… but no matter, it’s fast enough with QEMU.)
The second example is an old NAS, which is an armv7l system. There aren’t cached builds for armv7l, so the whole thing is cross-compiled. I added armv7l-linux to both nixpkgs.crossSystem and my desktop’s emulatedSystems and that’s that, in theory. In practice, the build often breaks. I track the stable branch because nixos-unstable regularly breaks cross-compiled systems.
Sometimes, packages just won’t cross-compile and it’s not easy to figure out a fix:
I ran into the same GRUB issue you had, and similarly worked around it by not using GRUB.
ZFS was working fine on my NAS, but one day the nix build started failing because of an aggressive new compatibility check.
In short: Cross-compiling generally works, but you should expect to make compromises to get things working. Maybe you natively compile some packages with QEMU, or maybe you do without. I found aarch64 is significantly easier to use (because there are cached native builds) than armv7l.
Most of those (all except gzip and linux-headers) are coming from the bash, and I checked that /nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/bash is indeed an x86-64 binary. So is that a packaging bug in nginx?
I didn’t dig into it too much further, because the boot.binfmt.emulatedSystems made everything work. (And fortunately for me, all the arm systems I plan to do this for are aarch64.) I’ll deal with the per-package cross-compilation if I need it later, but this will get me started for now. Thanks for the detailed examples, @Majiir !