I’m currently tinkering with cross-compiling NixOS from an AMD machine (“x86_64-linux”) to Raspberry Pi 5 (“aarch64-linux”) and Starfive VisionFive 2 (“riscv-linux”), and would like to know about the latest best practice to recommend in guides. I found several similar styles, but i’m not sure which one is the latest and best:
I have sometimes also have seem both system and config having been defined.
So which one is the latest, recommended way? (Or what are the trade-offs?). I could create a PR against the official documentation as well if you can point me to the right place.
Shameless bump as i stumbled upon this part-hilarious blog post https://ianthehenry.com/posts/how-to-learn-nix/cross-compilation/ about the state of the Nix manual on cross-compiling and it describes the same questions and confusion i have. I would like to improve the documentation and start with this low-hanging fruit here.
{
# This is the architecture we build from (pkgs.system from above)
nixpkgs.buildPlatform = builtins.currentSystem;
# pkgsCross.<yourtarget>.system
nixpkgs.hostPlatform = "aarch64-linux";
}
but you have to pass --impure but i’m not sure otherwise how to make a nixosConfiguration that I can either build on my laptop or on my raspberry pi. I feel like I’m missin gsomething.
I haven’t decided. I’m still waiting for some Nix experts to give recommendations. I think i remember all variants working, but it’s confusing for non-experts as every guide is slightly different.
That’s because, as with many things in Nix, there isn’t one proven best way of doing things. If you ask 4 experts, you’ll get 4 opinions on how things should be done.
I wish there was a way to search for specific hydra built images and inspect the sources for all those artifacts. I am hoping this could help me overcome my current armv7l cross compilation issues.
Don’t take my word as gospel since it’s been a while since I touched this topic. Here is an implicit way to cross compile with flakes (don’t ask me why this works).
The only argument I can give for this method is that it looks clean.
As a blob of general knowledge I can add that cross-compiled derivations have a slightly different output because the compiler does slightly different things depending on instruction set. This is the reason cross-compiled derivations are impure?! and means when the target system wants to rebuild it will rebuild everything.
Instruction level emulation (binfmt) of the native compiler avoids this, at the cost of massively reduced speed.
These lines do nothing in your code, and this is not cross-compiling. Probably you’re using emulated compilation. If you want to cross compile, one way would be to do this:
IIUC there is another approach where your NixOS configuration knows nothing about the cross-compilation and instead you deal with it when you instantiate nixpkgs. For example:
In this commit I figured out a way to use deploy-rs from an x86 host for deploying to a Raspberry Pi. The key magic here is:
pkgs = import nixpkgs { system = "x86_64-linux"; };
# This is a rather bananas dance to create a cross-compiled deploy-rs.
# There is a binary in there that needs to be build for the target
# architecture, so this sets up a version of nixpkgs that's cross-compiled
# to aarch64. Then deploy-rs provides an overlay that will build the
# pacakge via this cross compilation. This does still require building
# rustc though lmao.
# Note this ISN'T used for the actual NixOS system, for that it's just
# built "natively" so you'll need the binfmt_misc magic to make it work.
# That is fine in practice because you just get everything from the binary
# cache.
# https://nixos.wiki/wiki/Cross_Compiling has a section about "lazy
# cross-compiling" that seems like a more elegant way to achieve something
# kinda similar to this.
pkgsCross = import nixpkgs {
localSystem = "x86_64-linux";
crossSystem = { config = "aarch64-unknown-linux-gnu"; };
overlays = [ deploy-rs.overlays.default ];
};
Then wherever you need a cross-compiled package you refer to it from pkgsCross instead of pkgs.
(Note this example is kinda confusing because pkgs refers to an x86 nixpkgs, pkgsCross refers to an aarch64-built-on-x86 nixpkgs, and then I think there is a third version of nixpkgs that is being used implicitly as a result of me passing system = "aarch64-linux" to nixpkgs.lib.nixosSystem, and that’s the native aarch64 one. But in my current configuration I never need to refer to that explicitly).
This is obviously also pretty janky but it seem like it might be an avenue worth exploring if you want fine-grained control. For example I use it as a hack to cross-compile stuff that I need to compile, and then use the natively-compiled versions of packages that are in the binary cache. So I don’t have to cross-compile the whole of NixOS.
