Error building a raspberry pi image using nix on a x86_64 system

Hi *,

I use the following flake to build a raspberry pi image

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    nixos-hardware.url = "github:nixos/nixos-hardware";
  };
  outputs = {
    self,
    nixpkgs,
    nixos-hardware,
  }: {
    images = {
      rpi-4 =
        (self.nixosConfigurations.rpi.extendModules {
          modules = ["${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64-new-kernel-no-zfs-installer.nix"];
        })
        .config
        .system
        .build
        .sdImage;
    };

    nixosConfigurations = {
      rpi = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          nixos-hardware.nixosModules.raspberry-pi-4
          ./hosts/rpi4/configuration.nix
        ];
      };
    };
  };
}

On my NixOS (x86_64) with boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; everything works fine and the image builds as expected.

Trying to build the image on a Ubuntu 22.04 (x86_64) fails with the following message:

mi@mi-ThinkCentre-M75q-1:~/nixos-raspi$ nix build .#images.rpi-4
warning: Git tree '/home/mi/nixos-raspi' is dirty
error: a 'aarch64-linux' with features {} is required to build '/nix/store/dg8m1dlmqb9i11290ywlan5bwmj49fdf-config.txt.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, nixos-test, uid-range}

After installing/enabling binfmt and qemu-user-static I am still getting the same error message.

Any ideas on how to build this flake.nix on a non NixOS system?

Many thanks,
qitta

Briefly, the reason this fails is that Nix requires a derivation’s system parameter to match the current system in order to perform a local build. It works for you on NixOS because of the emulation, since the system will set extra-platforms (see the module source for details).

You could try pkgsCross by following the cross-compilation tutorial. Please report your results. That would also help us improve the documentation (in fact, the ideal form of feedback is a pull request). If it works, @jade’s post may need some corrections in the overview and the section on cross-compilation.

A workaround would be to set extra-platforms = aarch64_linux in nix.conf on Ubuntu and somehow make sure that those instructions actually run on that system (the NixOS module may give some hints how to do that). Another workaround would be to configure remote build machines and delegate building derivations for “foreign” systems. If that’s too much non-declarative hassle, consider trying System Manager (I haven’t and would like to know how it goes).

An alternative would be to use niv for dependency management (or npins, which is very similar but maintained), since with your particular configuration you probably don’t have any advantage from using flakes.

To OP: just set extra-platforms and since you have binfmt-misc set up, I expect it will just work.

To Valentin: There are a few errata in your post.

You could try pkgsCross by following the cross-compilation tutorial.

pkgsCross is the incorrect approach for cross-building NixOS specifically (unless you use pkgsCross.blah.nixos as an entry point, but that’s extremely obscure, doesn’t work with non-flakes, and is not the clean way to do it).

The reason for this is that you either need a native-arch builder (such as by setting extra-platforms and using binfmt-misc or having a native machine), or you need to fully cross-build the system, since you need to be able to build any arbitrary derivation required by the system, which, for NixOS, will not all be in cache; thus, you need to import nixpkgs with the right architecture settings, which can be achieved by setting the nixpkgs.* NixOS options.

The correct approach to build a non-native NixOS architecture to set system in the nixosSystem invocation to aarch64-linux in the case of native-building with emulation (or indeed, not set it/set it to null and then set nixpkgs.system). For cross-building from x86_64-linux, set nixpkgs.buildPlatform and nixpkgs.hostPlatform and don’t pass system.

jade’s post may need some corrections in the overview and the section on cross-compilation.

Nope. I just don’t bother mentioning how to actually do the cross compilation in there, just how to support it as a downstream flake, which is perhaps an omission, but the post is 5k words long already. The point with suggesting overlays there is making it possible to “bring your own pkgs”. That’s the entire reason.

A workaround would be to set extra-platforms = aarch64_linux in nix.conf on Ubuntu

Yep, and set up binfmt-misc properly with the Ubuntu package; should be easy enough. I believe the extra-platforms is the only thing that the original poster is actually missing, and their config would just work then.

You could even set it inside your flake with nixConfig, I suppose. For remote build machines, you can set up the Nix builders config file, or you can use --build-machine of nixos-rebuild which will uhhh do a shitload of extremely unsurprising crimes involving nix eval, nix copy and nix-store -r to avoid the Nix remote build protocol, probably because it is really slow on non-local connections.

A full example of a command to (cross, or otherwise) build a flake based NixOS configuration locally and deploy it to another machine is:
NIX_SSHOPTS="-l root" nixos-rebuild switch --target-host YOUR_HOSTNAME --fast --flake .#yourConfig

One very embarrassing docs omission in the NixOS docs and arguably nix.dev is that it doesn’t mention anywhere in the manual to use --fast on nixos-rebuild, which is necessary for cross building a config, since otherwise it will try to re-exec Nix of the wrong architecture.

An alternative would be to use niv for dependency management

This is basically irrelevant to the question at hand, and I would argue that flakes are much more pleasant to use for NixOS since you can control the entry point to your configuration. But you could use this. It wouldn’t change the cross compilation steps, you just would have to use -I nixos-config=./mymachine.nix and then uhhhhhhhhh write a bad nixos-rebuild wrapper script to nix-instantiate --eval the store path out of npins/niv, then pass it as -I nixpkgs=/nix/store/..... I don’t particularly wonder why most people use flakes instead of doing this.

1 Like

@jade thanks for the elucidation. I find that many user stories, and definitely also documentation, around NixOS are very incomplete. We’re building many things from the ground up with nix.dev and in the Nix manual, and haven’t even reached the more advanced use cases that have been already been touched upon to some extent, including cross compilation and essentially NixOS.

@fricklerhandwerk @jade thanks for the fast reply and explanation.

I am currently using flakes, because this flake is supposed to be passed to co-workers and customers to build a specific and identical image without hassle. So having a remote builder is not a option for my use case.

I also would rather just want to stick to flakes without additional toolset to minimize moving parts.

Setting extra-platforms = aarch64-linux in nix.conf solves the previous issue but gives me somehow another weird error message:

warning: Git tree '/home/mi/nixos-raspi' is dirty
error: builder for '/nix/store/md9h0vj7v1gy02473jdpz320hjl6hfpn-builder.pl.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/md9h0vj7v1gy02473jdpz320hjl6hfpn-builder.pl.drv'.
error: builder for '/nix/store/j20aqm9anwsf0yy12f6s4h7sgqxs8vw4-chfn.pam.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/j20aqm9anwsf0yy12f6s4h7sgqxs8vw4-chfn.pam.drv'.
error: builder for '/nix/store/k91h034bxak5ms53qv06wss1sc1159sq-config.txt.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/k91h034bxak5ms53qv06wss1sc1159sq-config.txt.drv'.
error: builder for '/nix/store/nciy3n0vxwf6w7zcpvwym5v3x8v0srsh-extlinux-conf-builder.sh.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/nciy3n0vxwf6w7zcpvwym5v3x8v0srsh-extlinux-conf-builder.sh.drv'.
error: builder for '/nix/store/c388jcvvyqg86ixa3kvgshwnalc5wf5j-mounts.sh.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/c388jcvvyqg86ixa3kvgshwnalc5wf5j-mounts.sh.drv'.
error: builder for '/nix/store/lgbvf6wgzlf8z7y8snbaxzh1arclbb4i-users-groups.json.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/lgbvf6wgzlf8z7y8snbaxzh1arclbb4i-users-groups.json.drv'.
error: 1 dependencies of derivation '/nix/store/16jmbi9vj8i2ywckqwvhdj2zk27nn1nw-nixos-sd-image-24.05.20240106.46ae021-aarch64-linux.img.drv' failed to build
mi@mi-ThinkCentre-M75q-1:~/nixos-raspi$ nix log /nix/store/md9h0vj7v1gy02473jdpz320hjl6hfpn-builder.pl.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/md9h0vj7v1gy02473jdpz320hjl6hfpn-builder.pl.drv^*'
error: executing '/nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash': No such file or directory

Using a nix shell with bash inside or installing bash to nix profile does not help even though bash seems to be available:

mi@mi-ThinkCentre-M75q-1:~/nixos-raspi$ /nix/store/l34lhs5k16k8pydj2rjq70pqxz86kd3f-bash-5.2-p21/bin/bash
bash: shopt: progcomp: invalid shell option name
bash: complete: command not found
[...]
bash: complete: command not found
bash: shopt: hostcomplete: invalid shell option name
bash: complete: command not found
[...]
bash: complete: command not found
\[\]mi@mi-ThinkCentre-M75q-1:~/nixos-raspi$ 

Any ideas what might go wrong here?

Many thanks
qitta

Adding the nixpkgs.buildPlatform and nixpkgs.hostPlatform like this started the build, even though it seems as there is everything being recompiled - at least it seems like this because it takes forever. Is there a way to just trigger recompilation of stuff that is required?

[...]
 nixosConfigurations = {
      rpi-4 = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [

	({ pkgs, config, ...}: {
	 nixpkgs.buildPlatform = { system = "x86_64-linux"; };
	 nixpkgs.hostPlatform = { system = "aarch64-linux"; };
	 })

          nixos-hardware.nixosModules.raspberry-pi-4
          ./hosts/rpi4/configuration.nix
        ];
      };
    };
[...]

Does it seem like the proper way to set host and build platform like this?

But let’s see if the build will complete successful…

One generally annoying limitation which follows from how Nix works (namely, Nix itself doesn’t know what cross compilation is so cross-built packages have different derivations to native-built ones), is that the binary cache will not include many cross-built packages so if you cross compile NixOS you may be building a fair number of packages from source. My suggestion for this is that you just set up binfmt-misc and native-build the config (that is, system = "aarch64-linux"). For particularly heavy to build packages that you want to build faster, you can import nixpkgs again as a cross-compilation setup and pull them from there (with something like import pkgs.path { config = config.nixpkgs.config; overlays = config.nixpkgs.overlays; buildPlatform = { system = "x86_64-linux"; }; hostPlatform = { system = "aarch64-linux"; }; }; I would suggest putting it into an option that you reuse everywhere you use it such that you only eat the evaluation time once). However, this aggravatingly has a caveat that you will waste a bunch of Nix store space since the dependencies will be duplicated. I am not surprised that cross-compiling NixOS is not a commonly used configuration on account of this.

Something is awry with binfmt-misc, but i cannot say what it is; possibly due to sandbox chroot not allowing access to the binfmt handler or something? As for why that bash is behaving weird, it’s probably reading your bashrc and doing wrong stuff but I’m not sure why it’s broken.

EDIT: yeah it’s extra sandbox paths, look at the module there

@jade, thanks for explaining. My build finished after multiple hours on the Ubuntu system.

When running the cross compilation on my NixOS (x86_64) system it finishes in 372 seconds because the majority is coming from the cache, but when I set the hostPlatform and the buildPlatform see below, it also takes on the NixOS machine forever (symlink was removed every time and nix-store garbage collected).

Is this behavior expected? Can I somehow replicate the "mainly from cache behavior’’ for building the image that I have using NixOS also under a non-NixOS system?

    nixosConfigurations = {                                                                                        
      rpi4 = nixpkgs.lib.nixosSystem {                                                                             
        system = "aarch64-linux";                                                                                  
        modules = [                                                                                                
                                                                                                                   
        {                                                                                                          
          nixpkgs.config.allowUnsupportedSystem = true;                                                            
          nixpkgs.hostPlatform.system = "aarch64-linux";                                                           
          nixpkgs.buildPlatform.system = "x86_64-linux";                                                           
        }                                                                                                          
                                                                                                                   
          nixos-hardware.nixosModules.raspberry-pi-4                                                               
          ./hosts/rpi4/configuration.nix                                                                           
        ];                                                                                                         
      };                                                                                                           
    };                                                                                                             

Many thanks,
qitta

Yes. This is expected behaviour.

Remember: there are two modes (if you are building an aarch64 configuration and you have an x86_64 computer):

  • not cross compiling, and running the build under emulation. Here you get slower compile speeds but less to compile overall because you can use the cache. This is what happens when you set system to aarch64-linux.
  • cross compiling from x86_64 to aarch64. Here you cannot use the cache, mostly, but the compilation speeds are faster. If buildPlatform is set to x86_64-linux and hostPlatform (or system; system is ignored if you set hostPlatform), this is what you are doing.

@jade, okay now I got it with the two modes.

Any hints/ideas how I can workaround/fix the sandbox path issue that gives me the bash error?

Read the NixOS module linked above and set extra-sandbox-paths in a similar fashion

@jade, thanks, I will have a look into it :).

Edit: @jade, sorry to ask again, but which module exactly do you mean? I did not find any sandbox paths, I guess I was looking into the wrong module. Do you mean the binfmt.nix module?

Update: @jade and @fricklerhandwerk thanks for the input again, I managed not to make it build/cross-compile on a Ubuntu 22.04 LTS. For it to build on a this system you just need to set extra-platforms = aarch64-linux (nix.conf) and install the qemu-user-static-package from the Ubuntu repository.

1 Like