Emulated cross-platform toolchain challenges with Rust (aarch64-linux and x86_64-linux)

Help
1

Hello everyone,

I’ve been trying to configure a nix project to produce binaries for Darwin (arm64, x86_64) and Linux (arm64, x86_64) systems, so people can easily download them from GitHub releases. I’m trying to use Nix as the build process this time, to have a reproducible build locally as well as on GitHub Actions.

Instead of going through the cross-compilation route, with logic to use either ${system}.packages or ${system}.pkgsCross.${targetSystem}, I’m attempting to use an emulated cross-platform toolchain with qemu-user over binfmt_misc on the Kernel.

Challenges with emulated toolchain/compilation

For most of the different computer environments I had access, qemu was able to produce the binaries for both the host and the emulated platform, except for a few scenarios that I’ll need more help, as I’m not as versed on the NixOS internal implementations.

  1. NixOS boot.binfmt.emulatedSystem vs cc derivation symbol errors

For most of the execution process, setting boot.binfmt.emulatedSystem = ["aarch64-linux"] on a NixOS VM allows me to execute aarch64-linux binaries and derivations.

nix shell nixpgs#hello --system aarch64-linux
nix shell nixpgs#hello --system x86_64-linux

But when trying to compile a Rust project, the cc linker (emulated) fails.

/build/rustcdhlbOQ/symbols.o: file not recognized: file format not recognized

In several other situations, when the emulation is done by a qemu-user-static binary on a non-NixOS host (eg: Docker, LXC), the same project compiles flawlessly and produce the cross-architecture binary as desired. I’ve tried to read more the cc derivation which is really complex with a lot of patching being done, as well as changing between sandbox = true|false, llvm|gcc.stdenv, etc.

I can’t figure out how to avoid the cc derivation script to remove the /tmp path to investigate what it produces also.

  1. craneLib.cargoTest vs craneLib.cargoNextest

Running craneLib.cargoTest works flawlessly on the emulated toolchain (with a non-Nixos host…), but calling cargo nextest run from a shell or using the craneLib.cargoNextest derivation fails with an error that seems to be related to linking symbol loading issues.

 Caused by:
>   for `project`, command `/build/source/target/release/deps/project-f15c34dc6dacf9a1 --list --format terse` exited with code 1
> --- stdout:
> Error while loading __double-spawn: No such file or directory

The resulting intermediate binaries are produced with the desired target architecture, are fully linked to their corresponding glibc architecture, and work when executed directly, but fails when called as a fork/exec from the emulated cargo-nextest process.

This is not a crane specific issue as I can also trigger the error from a nix shell nixpkgs#cargo-nextest. This might be specific issues with cargo-nextest, but I’m curious if there is any lessons on use of patchelf, LD_LIBRARY_PATH or any other magic that can help patch the call, so the project can continue using cargo nextest.

Questions

I would appreciate any help figuring out how to produce this cross-architecture emulated recipe for project distribution. I’ve put together a quick troubleshooting repo if anyone has spare time to help figure out these arcane linking issues.

Overall, I have a few questions:

  • Is using emulated toolchains usual on the Nix ecosystem? Or is it preferred to do cross-compilation?
  • How can I prevent the Nix cc script from removing intermediate/failed files so I can investigate what it’s doing?
  • What could cause the NixOS qemu emulation error, while Non-NixOS emulation works fine? Any tips on how to start this branch of investigation?

Thanks for reading it so far and for any insights you can contribute.