Cross compiling and the nix cache

Help
1

Hey all,

I’m pretty sure I am missing something here but I can’t find what. I am trying to build an aarch64-linux image (Raspberry Pi 3B+) on my x86_64-linux machine. In the image’s nix config I set

nixpkgs.crossSystem = lib.systems.examples.aarch64-multiplatform // { system = "aarch64-linux"; };

and in my flake.nix I have

     nixosConfigurations.raspi = nixpkgs.nixosSystem {
        system = "aarch64-linux";
        modules = [
          ...
          ./hardware/raspberrypi.nix # <- this sets crossSystem like above
          ...
          ({ ... }: {
            nixpkgs.localSystem.config = "x86_64-unknown-linux-gnu";
            nixpkgs.localSystem.system = "x86_64-linux";
          })
        ];

When I build this on my local machine, I have to compile a lot of packages myself. However, when I build this on an AWS graviton instance that is aarch64 native (without specifying localSystem and crossSystem), everything is fetched from the nix cache.

What am I missing for my local machine to fetch the packages from the nix cache? It’s the same flake, so the same nixpkgs revision, which means that everything should be in the cache.

2

Cross compiling does have different inputs than native compilation. So the results hash is different, so it won’t be found in the cache.

You might be able to use binfmt to do native arm compilation on the x86.

1 Like
3
  • In nixosSystem ={…, set system to “x86_64-linux” - yep, looks strange, dunno why it has to be that way, but it works,
  • I don’t set system next to crossSystem,
  • and I don’t set any localSystem.

That’s how I did it recently. Maybe, the first change I’ve mentioned will be sufficient.

Please, stop recommending binfmt. That’s horrible slooow.
If there’s something that doesn’t cross-compile, we should try to fix it.

(Sorry for the short crappy formated msg. Typing on the phone is awful.)

4

What I’d like to add here is that this can’t work because the local system is
part of the Nix derivation (observable in the drv files). So native and
cross compiled targets can’t have the same hash.

Cross compiling and native compiling are really different processes, so
it absolutely makes sense that they are not available behind the same
cache key. E.g. it is quite common for packages in Nixpkgs to fail cross
compilation, while they compile just fine natively for the same target.

5

I understand that cross compiling is different from native. What I don’t understand is why I need to cross compile anything at all if everything I need is packaged and cached for my target architecture and nixpkgs revision.

What I’d like to add here is that this can’t work because the local system is
part of the Nix derivation (observable in the drv files). So native and
cross compiled targets can’t have the same hash.

Can you elaborate why this is the case?

Thanks @bartsch for the suggestion, but unfortunately that doesn’t work either.

6

Lets assume a “simple” case. You want to build a hello-world go program.

Natively the hello-world was compiled using arm go compiler, and gets cached by hydra. Only the hello world program needs to get substituted from cache.

With CC though a x86 Go compiler was used to create the hello world, which changes its hash, as the hash depends on the hash of its inputs. Hydra only builds natively, not CC. Thats why the cross compiled hello-world is not in the cache.

7

Thanks @Nobbz, I hadn’t considered that! Consider this question answered, thanks all! :slight_smile:

8

@Hjdskes – it looks like you had some success here. I’m struggling through a similar project – are you able to share your final (working) flake?

9

Hi @n8henrie, as explained in this thread cross compiling cannot inherently use the same Nix cache. In the end, I gave up and switched to an X86 machine I was already using for other purposes.

10

Right, I just wanted an example of a flake that was working for cross compiling an RPi3 image – but I think I figured it out.

11

Hi @n8henrie, would you be so kind as to share how you “figured it out”?

12

Sure – you’ll find a great deal of details in my thread history: Profile - n8henrie - NixOS Discourse

In particular you might be interested in: