Hybrid QEMU chroot + cross-compilation

Since converting all my home devices to NixOS, I now have a plethora of different architectures to manage: aarch64, x86_64, and ppc64le. I would like to compile all my packages on the beefiest server, despite being a different architecture than some of the targets.

There are two main ways of building a package for a different architecture:

  1. QEMU emulated “native” compilation
  2. Cross-compilation

Both of these have their drawbacks. QEMU compilation almost always works but can be quite slow and inefficient (albeit still faster than a Raspberry Pi). Cross-compilation is fast but can get hairy. Complex packages such as Firefox are likely to fail to cross-build.

There is a hybrid approach I have successfully used with Debian packages which I would like to try and get working with Nix. Essentially you take the first approach (QEMU full emulation), but selectively replace executables in that chroot with their native/cross-compilation counterpart. Typically replacing GCC alone is enough for a significant speed boost, but replacing bash, perl, etc can also be helpful. Here’s a post from the qemu mailing list describing this strategy.

I don’t expect to have a breakthrough with this strategy anytime soon, but I wonder if anyone has already attempted this with Nix?

2 Likes

that’s reasonably easy to achieve with overlays that selectively replace native packages with their cross-compiled variants. here’s a writeup of someone who did that to make kernel compiles on an arm device less painful: nixos on underpowered devices. the same thing could be done for almost any package

1 Like

Fantastic! That’s not quite the hybrid approach I was going for, but it’s definitely a good solution. I will try the inverse of that blog post: cross-compile everything, and fall back to native compilation for broken packages.

that should be just as easy by switching around a few attributes and package sets. do keep an eye out though, some packages simply refuse to cross-compile for the oddest reasons that can sometimes even look like the toolchain being broken instead of the package itself. :confused:

Hosted by Flying Circus.