Hybrid QEMU chroot + cross-compilation

Since converting all my home devices to NixOS, I now have a plethora of different architectures to manage: aarch64, x86_64, and ppc64le. I would like to compile all my packages on the beefiest server, despite being a different architecture than some of the targets.

There are two main ways of building a package for a different architecture:

  1. QEMU emulated “native” compilation
  2. Cross-compilation

Both of these have their drawbacks. QEMU compilation almost always works but can be quite slow and inefficient (albeit still faster than a Raspberry Pi). Cross-compilation is fast but can get hairy. Complex packages such as Firefox are likely to fail to cross-build.

There is a hybrid approach I have successfully used with Debian packages which I would like to try and get working with Nix. Essentially you take the first approach (QEMU full emulation), but selectively replace executables in that chroot with their native/cross-compilation counterpart. Typically replacing GCC alone is enough for a significant speed boost, but replacing bash, perl, etc can also be helpful. Here’s a post from the qemu mailing list describing this strategy.

I don’t expect to have a breakthrough with this strategy anytime soon, but I wonder if anyone has already attempted this with Nix?

2 Likes

that’s reasonably easy to achieve with overlays that selectively replace native packages with their cross-compiled variants. here’s a writeup of someone who did that to make kernel compiles on an arm device less painful: nixos on underpowered devices. the same thing could be done for almost any package

1 Like

Fantastic! That’s not quite the hybrid approach I was going for, but it’s definitely a good solution. I will try the inverse of that blog post: cross-compile everything, and fall back to native compilation for broken packages.

that should be just as easy by switching around a few attributes and package sets. do keep an eye out though, some packages simply refuse to cross-compile for the oddest reasons that can sometimes even look like the toolchain being broken instead of the package itself. :confused:

While playing around I noticed an odd thing: using the cross-compilation infrastructure to target the native architecture does not seem to work. Maybe this is by design, although I feel it ought to work.

gnu64 build host → raspberryPi target :heavy_check_mark:
gnu64 build host → powernv target :heavy_check_mark:
gnu64 build host → gnu64 target :x:

powernv build host → raspberryPi target :heavy_check_mark:
powernv build host → gnu64 target :heavy_check_mark:
powernv build host → powernv target :x:

In both cases the error message is virtually identical: while building bash it complains the “C compiler cannot create executables”.

chad@talos $ nix-build '.' -A pkgsCross.powernv.nix
Beginning configuration for bash-4.4-release for powerpc64le-unknown-linux-gnu

checking for powerpc64le-unknown-linux-gnu-gcc... powerpc64le-unknown-linux-gnu-gcc
checking whether the C compiler works... no
configure: error: in `/build/bash-4.4':
configure: error: C compiler cannot create executables
See `config.log' for more details