Hi!
I’d like to be able to build my NixOS config for an RPi on my x86_64 laptop, as the eval/build on the Pi takes a lot of time.
However, when I do that, everything is compiled on the build host which takes even more time.
I’ve seen this topic but that’s about non-NixOS machine and I don’t really understand if it is related to the fact or not.
Is there even a way to build NixOS config for different architecture without having to actually compile everything?
Thanks,
-j.
If you set boot.binfmt.emulatedSystems = ["aarch64-linux"]; on your build system, you can compile programs as if your build system is an arm computer.
Emulation is so slow that it’s probably faster just to build on the pi.
That is exactly the issue. If you’re doing cross compilation, you are not building the same packages that cache.nixos.org keeps cached. A cross compiled package is a different package than its natively compiled counterpart.
You can mix and match though. You can make it so that everything by default is built natively, but specific packages are cross compiled.
{ lib, pkgs, ... }:
let
crossPkgs = import pkgs.path {
localSystem = "x86_64-linux";
crossSystem = "aarch64-linux";
};
in {
# I'm being overly explicit for demonstration purposes
nixpkgs.buildPlatform = "aarch64-linux";
nixpkgs.hostPlatform = "aarch64-linux";
# Cross compile just the kernel
boot.kernelPackages = crossPkgs.linuxPackages;
}
I second the mix and match approach, this is exactly what I do for my raspies - the default package set is set up for cross-compilation, but I have a native one as another input, so I can override things that would take ages to compile even with cross-compilation (like kernels - or were those GUI machines - browsers) or that fail to cross compiler for some reason (e.g. ncdu is now zig-based, which doesn’t/didn’t support cross-compilation in nixpkgs).
I imagine another option could be using system.replaceDependencies possibly, but haven’t tried using it that way, so not sure it would work.
Hopefully when CA derivations get stabilised we could think about making native and cross-compiled packages equivalent? Barring any bugs in compiler and non-determinism the resulting binary shouldn’t differ between cross-compilation and a native compiler, right?
This requires the replacements to have nearly identical path names (such as the same length), but cross compiled packages will have e.g. -aarch64-unknown-linux-gnu added to the package name which breaks this.
I think we would be very lucky if CA derivations accomplished this with any regularity. Just getting stdenv to be similar enough to do this reliably would probably be a large task.
Ah, right, good point about the paths, I guess it’s the mix & march approach only. Good thing it works well enough.
Shame about CA derivations, hoped this could have been one of improvements coming from them.
Still not sure I understand everything that had been said.
So if I put this
in the rpi config and build it on my laptop, it should work? Or did I miss something between the lines?
Because it doesn’t, it still tries to compile HASS, kernel, and other big things. It doesn’t do that when I set --build-host to the rpi.
What I don’t fully understand is that it doesn’t try to cross-compile everything. Stuff like busybox for example, and a bunch of other packages.
You really need to share you config and the exact commands you’re using to build it.
ok, let’s start from the basics (my config is rather complicated so I’m trying to create a minimal example instead):
I got this base flake:
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
nixos-hardware.url = "github:NixOS/nixos-hardware/master";
};
outputs = { self, nixpkgs, nixos-hardware }: {
nixosConfigurations = {
rpi = let system = "aarch64-linux";
in nixpkgs.lib.nixosSystem {
specialArgs = { inherit system nixos-hardware; };
pkgs = import nixpkgs { inherit system; };
modules = [ ./rpi ];
};
};
};
}
and `rpi/default.nix’:
{ nixos-hardware, modulesPath, ...}:
{
imports = [
nixos-hardware.nixosModules.raspberry-pi-4
(modulesPath + "/installer/scan/not-detected.nix")
];
# nixpkgs.buildPlatform = "aarch64-linux";
# nixpkgs.hostPlatform = "aarch64-linux";
fileSystems."/" = {
device = "/dev/mmc0p1";
fsType = "ext4";
};
fileSystems."/boot" =
{ device = "/dev/disk/by-label/BOOT";
fsType = "ext4";
options = [ "nofail" ];
};
virtualisation = {
podman = {
enable = true;
};
};
system.stateVersion = "22.05";
}
running
$ uname -m
x86_64
$ nixos-rebuild build --flake '.#rpi' --build-host localhost
apparently downloads packages from NixOS cache:
copying path '/nix/store/...' from 'https://cache.nixos.org'...
but then builds the derivations in qemu-aarch64:
building '/nix/store/....drv'...
Which itself is rather slow but at least it doesn’t try to compile anything. Uncommenting the *platform lines doesn’t make a difference as far as I can tell.
If I understand what was said above tho, even this shouldn’t happen and the build should happen natively.
I’m trying to add more things to see what exactly triggers the compile. In the mean time, I would like to at least understand what (if anything) I’m doing wrong when the drv builds happens in the aarch64 qemu VM.
I do have boot.binfmt.emulatedSystems = ["aarch64-linux"]; on the build machine (obviously), running NixOS 24.11.
ok, that didn’t take long. I use ZFS in fact and adding this:
fileSystems."/" = {
device = "tank/root";
fsType = "zfs";
};
Triggers the build of zfs-kernel
building '/nix/store/6hv7n3y7zlv7mkaz2idxfxyyvqmxczkp-zfs-kernel-2.2.7-6.6.31-stable_20240529.drv'...
which in turn probably triggers rebuild of dozens of other packages.
The latest RPi kernel in nixpkgs is 6.6.31, but when I try to search Hydra for this exact build, I get a 500 error. I guess that means it’s not there because searching for other builds yields results.
Well, don’t ever do this in the nixosSystem arguments. You’re overriding all of the logic nixos does to determine how to form pkgs and replacing it with your own pkgs. So anything you do like nixpkgs.buildPlatform will have no effect. Most people just pass the system argument to nixosSystem like this:
nixosConfigurations = {
rpi = nixpkgs.lib.nixosSystem {
system = "aarch64-linux";
/* ... */
};
But if you’re wanting to play around with what platform is the build vs host platform, then leave that system argument out too and just use the nixpkgs.buildPlatform and nixpkgs.hostPlatform nixos options.