How do I cross-compile a flake?

I wrote a flake that has both x86_64-linux and aarch64-linux versions of the same package.

Now I would like to cross-compile that package for aarch64-linux on an x86_64-linux machine.

Is there a good way to do that?

None of the flags/options I saw in the wiki or elsewhere online seem to apply. Is there already a (convenient) way do do this, or is that functionality just not available yet?

For context this is the flake I’m working on.

4 Likes

I can’t say I’ve tried it, but would an option like registering QEMU as a binfmt wrapper like on NixOS on ARM - NixOS Wiki work?

2 Likes

Oh wow. Looks like it’s building something now. Thanks!

I will update this topic with the results.

2 Likes

Following your suggestion I added

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

to the configuration of my x86_64-linux build system.

As soon as I did that I was able to build my flake using

nix build .#defaultPackage.aarch64-linux

That worked like a charm! :partying_face: :exploding_head: I even copied over the closure to verify that the result runs on the actual aarch64-linux target, and it does. :grin:

Just as a reference for others, before adding that option I only got the following error message:

nix build .#defaultPackage.aarch64-linux
warning: Git tree '/home/mschwaig/flakes/tensorflow-lite-hello-world' is dirty
error: --- Error ----------------------------------------------------- nix
a 'aarch64-linux' with features {} is required to build '/nix/store/my1h7yqcplmpihygkhq8s4i4syfcazpv-builder.pl.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
(use '--show-trace' to show detailed location information)
2 Likes

This is emulated native compilation, not cross compilation. I don’t think proper cross compilation is part of the flakes design, yet?

1 Like

@mschwaig what did you do with the ./result exactly? After it was built. I mean, how did you copy it to the target machine?

Thanks.

You can do cross compilation with flakes:

$ uname -a
Linux jon-desktop 5.4.142 #1-NixOS SMP Wed Aug 18 06:57:05 UTC 2021 x86_64 GNU/Linux
$ nix build nixpkgs#legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.hello
$ file ./result/bin/hello
./result/bin/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/zh8wdxlvbaycfnnd3c9hzq07h5x7blaw-glibc-aarch64-unknown-linux-gnu-2.33-47/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, not stripped

Using boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; is still probably preferred, as the target system will likely expect to use aarch64 packages that were built on aarch64 (cross compilation produces different .drv’s). I’m not really sure how to “consume” cross compiled packages without referencing the store path directly.

You can use a nix copy command to copy the result to the target machine like this:

nix copy --to ssh://user@hostname .#defaultPackage.aarch64-linux

You can also use that command to copy to a binary cache and have the target get to them that way.

1 Like

I am not convinced being able to do

nix build nixpkgs#legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.hello

means that you can do ‘real’ cross-compilation with Flakes, as an alternative to compiling under emulation. This seems helpful if you want to cross-compile a package from nixpkgs with the new nix command.

As far as I can see this only works with nxpkgs specifically and I don’t know what it would take to have any other Flake to offer an interface for cross-compilation like that. I have not read about anything like that yet.

apply an overlay adding your package, then any of the pkgsX cross-compilation overlays will also work.

2 Likes

@jonringer do you have an example of doing that? I am struggling to figure out how to get it to work based on my limited nix (and flake) experience, and I can’t seem to find any documentation showing how to do it with a flake.