Cross-building for aarch64 with Nix on a non-NixOS (Fedora 39) machine won't use the cache

Help
1

Hello.

I am experiencing an issue(?) with cross-architecture building on a non-NixOS machine (Fedora 39 with Nix installed). It appears that the binary cache for aarch64 is not being utilized, leading to the need for a complete rebuild.

I’ve created flake:

{
  description = "NixOS Raspberry Pi configuration flake";
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-23.05";
  outputs = { self, nixpkgs }@inputs: {
    nixosConfigurations = {
      rpi =
        let
          system = "aarch64-linux";
          config = config;
          pkgs = import nixpkgs {
            inherit system;
            config.allowUnfree = true;
          };
          lib = pkgs.lib;
        in
        inputs.nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
            {
              nixpkgs.buildPlatform.system = "x86_64-linux";
              nixpkgs.hostPlatform.system = "aarch64-linux";
            }
            (import ./raspberry-pi-zero-2.nix { inherit config lib pkgs; })
            (import ./stage1/configuration.nix { inherit inputs lib pkgs system; })
          ];
        };
        rpiImage = self.nixosConfigurations.rpi.config.system.build.sdImage;
      
... not important ...

and I run it like:
nix build --extra-experimental-features nix-command --extra-experimental-features flakes --impure --update-input nixpkgs --commit-lock-file .#nixosConfigurations.rpiImage

On NixOS machine, some aarch64 packages were downloaded from binary cache.
However, on my Fedora 39 machine, with Nix installed,
it seems like no binary cache were used and ~630 packages need rebuild, which takes too much time.

Did I miss something in my flake or cmdline, for non NixOS machines?

Maybe, cache is not yet filled?
Part of flake.lock:

        "lastModified": 1699596684,
        "narHash": "sha256-XSXP8zjBZJBVvpNb2WmY0eW8O2ce+sVyj1T0/iBRIvg=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "da4024d0ead5d7820f6bd15147d3fe2a0c0cec73",
        "type": "github"
2

Cross-built packages are different from their native-built counterparts. They may be functionally identical, but Nix has no way to know that. It just knows how to build them. Since the native-built aarch64 packages in the cache are built differently than these cross-built packages, Nix won’t consider them for substitution of cross-built packages. The cache does include some of the absolute basics for cross-building packages, like the compiler, but most of it isn’t cached and will have to be built locally.

1 Like
3

Thank you.

That also explains, why nix-rebuild to remote machine, repopulate whole system after SD image were build and used.

So my another question would be:

Is there way, how to enable emulated build on Non-NixOS machine?
By emulated I mean something similar to:
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

For example, I have installed (on Fedora machine):

qemu-user-static-aarch64-8.1.1-1.fc39.x86_64
qemu-user-8.1.1-1.fc39.x86_64
qemu-user-binfmt-8.1.1-1.fc39.x86_64
qemu-system-aarch64-core-8.1.1-1.fc39.x86_64
qemu-system-aarch64-8.1.1-1.fc39.x86_64

and I’m able to run aarch64 binaries on x86_64 machine.

However:
nix build --impure --update-input nixpkgs --commit-lock-file .#nixosConfigurations.rpiNative.config.system.build.sdImage

ends with:

error: a 'aarch64-linux' with features {} is required to build '/nix/store/26r17f8h2ll22n9hrxllbmaava9aq4yy-config.txt.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

where rpiNative is same as rpi, but only aarch64 as a system is used (no cross compiling).
This works great on NixOS with emulation enabled.

So it seems that Nix needs to know, somehow, that this kind of emulation is supported on machine. I’m not calling for support of my Fedora 39, but I’m interested how Nix itself is able to detect support of emulation outside of NixOS world and provide great build environment.

4

If you’ve already got binfmt configured, then you just need some extra lines in nix.conf. Here’s what NixOS generated for me w.r.t. that:

extra-platforms = aarch64-linux i686-linux
extra-sandbox-paths = /run/binfmt /nix/store/rkwxakv5hfzxh513kaa7mlypf402q9v8-qemu-aarch64-binfmt-P-x86_64-unknown-linux-musl

extra-platforms tells it that we can build for those platforms, and extra-sandbox-paths is necessary for the emulator to be usable inside the linux sandbox (since sandboxing should function roughly the same on other linux distributions as it does on nixos, meaning you’d need this)

1 Like
5

Wow. Thank you very much again.

Ok. I ignored nix.conf for some reasons :(.

I just added (Fedora 39 specific, and not right if I want to add another platforms):

extra-platforms = aarch64-linux
extra-sandbox-paths = /usr/bin/qemu-aarch64-static

into /etc/nix/nix.conf and I’m able to build image for rpi zero 2w much faster (no rebuild of whole system).

Solved :). Thank you.