Recommended style to cross-compile flake nixosSystems

Help
1

I’m currently tinkering with cross-compiling NixOS from an AMD machine (“x86_64-linux”) to Raspberry Pi 5 (“aarch64-linux”) and Starfive VisionFive 2 (“riscv-linux”), and would like to know about the latest best practice to recommend in guides. I found several similar styles, but i’m not sure which one is the latest and best:

 mySystem = nixpkgs.lib.nixosSystem {
        modules = [
          {
            nixpkgs.buildPlatform = "x86_64-linux";
            nixpkgs.hostPlatform = "riscv64-linux";
          }
        ];
      };

like it seems to be recommended in the official NixOS manual cross-compile section (but unclear as it mentions other styles as well) and in Jörg Thalheims blog post.

Also recommended by @waffle8946 in Recommended style to cross-compile flake nixosSystems - #16 by waffle8946

 mySystem = nixpkgs.lib.nixosSystem {
        modules = [
          {
            nixpkgs.localSystem.config = "x86_64-unknown-linux-gnu";
            nixpkgs.crossSystem.config = "riscv64-unknown-linux-gnu";
          }
        ];
      };

that seems to be recommended in the official nix.dev cross-compile guide (although that style is shown only for pkgsCross usage)

 mySystem = nixpkgs.lib.nixosSystem {
        modules = [
          {
            nixpkgs.localSystem.system = "x86_64-linux";
            nixpkgs.crossSystem.system = "riscv64-linux";
          }
        ];
      };

that seems to be used in multiple blogs like the fantastic recently published handbook guide by Ryan Yin.

 mySystem = nixpkgs.lib.nixosSystem {
       system = "x86_64-linux"
        modules = [
          {
            nixpkgs.crossSystem.system = "riscv64-linux";
          }
        ];
      };

I have seen this style many times in blog posts.

 mySystem = nixpkgs.lib.nixosSystem {
       system = "x86_64-linux"
        modules = [
          {
            nixpkgs.crossSystem.config = "riscv64-unknown-linux-gnu";
            nixpkgs.crossSystem.system = "riscv64-linux";
          }
        ];
      };

I have sometimes also have seem both system and config having been defined.

So which one is the latest, recommended way? (Or what are the trade-offs?). I could create a PR against the official documentation as well if you can point me to the right place.

6 Likes
2

Shameless bump as i stumbled upon this part-hilarious blog post https://ianthehenry.com/posts/how-to-learn-nix/cross-compilation/ about the state of the Nix manual on cross-compiling and it describes the same questions and confusion i have. I would like to improve the documentation and start with this low-hanging fruit here.

6 Likes
3

What did you decide on is ultimately the best currently?

1 Like
4

I am also confused why we would encode the build platform.

That means if I wanted to do a nixos-rebuild switch on my other machine (i.e. Raspberry Pi) it would now fail?

5

Okay I could do:

        {
          # This is the architecture we build from (pkgs.system from above)
          nixpkgs.buildPlatform = builtins.currentSystem;
          # pkgsCross.<yourtarget>.system
          nixpkgs.hostPlatform = "aarch64-linux";
        }

but you have to pass --impure but i’m not sure otherwise how to make a nixosConfiguration that I can either build on my laptop or on my raspberry pi. I feel like I’m missin gsomething.

1 Like
6

I enabled this on my notebook to build the config for the Raspberry Pis:

I have two different “configs” however for x86 and aarch64, don’t know if there is a better way but it’s what I’ve been using for a while now.

7

That’s flake-specific. With flakes, you’re supposed to create additional outputs that have different system configurations.

8

I haven’t decided. I’m still waiting for some Nix experts to give recommendations. I think i remember all variants working, but it’s confusing for non-experts as every guide is slightly different.

9

That’s because, as with many things in Nix, there isn’t one proven best way of doing things. If you ask 4 experts, you’ll get 4 opinions on how things should be done.

You’ll have to think for yourself here.

10

I couldn’t get it to work. Something was failing.
I resorted back to binfmt sadly.

I also had to disable check phase for a few packages that were expensive to do in emulation.

11

@malteneuss
I’d like to throw in two more ways to declare cross compilation:

mySystem = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux"
  modules = [
    {
      nixpkgs.buildPlatform = "x86_64-linux";
      nixpkgs.hostPlatform = nixpkgs.lib.systems.examples.armv7l-hf-multiplatform;
    }
  ];
};

mySystem = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.armv7l-hf-multiplatform.nixos {
  imports = [
    ./my-module.nix
  ];
};

Thank you for sharing :slight_smile:

I wish there was a way to search for specific hydra built images and inspect the sources for all those artifacts. I am hoping this could help me overcome my current armv7l cross compilation issues.

2 Likes
12

Is there any kind of consensus now?

13

Not that i’m aware of. It would be great though for someone with more experience to explain the motivation between those different designs.

14

Don’t take my word as gospel since it’s been a while since I touched this topic.
Here is an implicit way to cross compile with flakes (don’t ask me why this works).
The only argument I can give for this method is that it looks clean.

As a blob of general knowledge I can add that cross-compiled derivations have a slightly different output because the compiler does slightly different things depending on instruction set. This is the reason cross-compiled derivations are impure?! and means when the target system wants to rebuild it will rebuild everything.
Instruction level emulation (binfmt) of the native compiler avoids this, at the cost of massively reduced speed.

15

These lines do nothing in your code, and this is not cross-compiling. Probably you’re using emulated compilation. If you want to cross compile, one way would be to do this:

diff --git a/flake.nix b/flake.nix
index e0956fe..45b2dc6 100755
--- a/flake.nix
+++ b/flake.nix
@@ -4,16 +4,10 @@
   };
   outputs = { nixpkgs, ... }:
   let
-    system = "x86_64-linux";
-    pkgs = import nixpkgs {
-      inherit system;
-    };
-    lib = nixpkgs.lib;
+   lib = nixpkgs.lib;
   in rec
   {
     nixosConfigurations.solaraspi = lib.nixosSystem {
-      system = "aarch64-linux";
-
       modules = [
         {
           imports = [
@@ -22,6 +16,10 @@
             ./configuration.nix
           ];
 
+          nixpkgs = {
+            hostPlatform = "aarch64-linux";
+            buildPlatform = "x86_64-linux";
+          };
           boot.kernelParams = lib.mkOverride 0 [ "console=ttyS1,115200" "console=tty1" ]; # Enable serial console on pins 8,10
           nix.extraOptions = ''experimental-features = nix-command flakes'';
         }
@@ -30,4 +28,4 @@
 
   solaraspi-image = nixosConfigurations.solaraspi.config.system.build.sdImage;
   };
-}
\ No newline at end of file
+}

See https://search.nixos.org/options?channel=24.11&show=nixpkgs.buildPlatform&from=0&size=50&sort=relevance&type=packages&query=buildplatform

Other cross-compilation methods mentioned above are somewhat outdated.

1 Like
16

@endimi @malteneuss I suggest sticking with method 1, as passing system to lib.nixosSystem is deprecated.

3 Likes
17

You’re right. However, the flake does produce a working image without enabling emulation in the host config. What is the default behavior here?

18

Not possible unless you built it already.