Cross compiling / building a flake for raspberry pi taking too long?

Help
1

I’m trying to build a nix image to put on a raspberry pi (aarch64-linux) using raspberry-pi-nix. I’m building this image on my x86_64-linux nix laptop, and it’s taking far too long to be reasonable.

I’m building a very basic config to start with, basically the default given in the github repo, but every time i try to build it the process gets stuck compiling linux. I can see that it is doing something, but I’ve left it running for almost two hours with cpu and fans on full and it didn’t complete; obviously nix build doesn’t show you how close to complete a specific step of the build process is, though.

I’m just a bit lost about what the right way to go about this is, because this doesn’t seem to be it. My laptop is pretty beefy, it shouldn’t take this long. I suspect it’s compiling through an emulator, which is why it’s taking so long, but I’m not sure how to check for sure or how to force it to cross-compile properly. Any advice much appreciated.

I have the following line in my system flake config which I think enables building / compilation through an emulator:

  # Enable emulation of other systems
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

And the flake.nix I’m trying to compile looks like this:

{
  description = "Your new nix config";

  inputs = {
    # Nixpkgs
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    # Stylix
    stylix.url = "github:danth/stylix";

    # Home manager
    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    # Firefox addons flake
    firefox-addons = {
      url = "gitlab:rycee/nur-expressions?dir=pkgs/firefox-addons";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    # Sddm theme
    sddm-sugar-candy-nix = {
      url = "gitlab:Zhaith-Izaliel/sddm-sugar-candy-nix";
      # Optional, by default this flake follows nixpkgs-unstable.
      inputs.nixpkgs.follows = "nixpkgs";
    };
    # AGS for widgets
    ags.url = "github:Aylur/ags";
  };

  outputs =
    { self, nixpkgs, home-manager, stylix, sddm-sugar-candy-nix, ... }@inputs:
    let
      inherit (self) outputs;
      defaultWallpaper = ./images/wallpaper_default.jpg;
    in {
      # NixOS configuration entrypoint
      nixosConfigurations = {
        # Config for gpteapot system
        gpteapot = nixpkgs.lib.nixosSystem {
          specialArgs = { inherit inputs outputs defaultWallpaper; };
          modules = [
            # System-level configuration
            ./system/configuration.nix
            sddm-sugar-candy-nix.nixosModules.default

            # Stylix
            stylix.nixosModules.stylix

            home-manager.nixosModules.home-manager
            {
              # Home manager modules
              home-manager.useGlobalPkgs = true;
              home-manager.useUserPackages = true;
              home-manager.extraSpecialArgs = { inherit inputs; };
              # User-specific configs
              home-manager.users.username = import ./home;
            }
          ];
        };

      };
    };
}

(configuration.nix has no effect on the process if removed, it just configures ssh and wifi)

I’m building with the following command:

nix build '.#nixosConfigurations.rpi.config.system.build.sdImage'

But i’ve tried a couple different settings like adding --system aarch64-linux with seemingly no effect.

2

Exactly, boot.binfmt.emulatedSystems = [ "aarch64-linux" ]; enabled the QEMU emulaor for aarch64-linux.

For cross-compling, you need something like this:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-23.11";
  };

  outputs = inputs@{ self, nixpkgs, ... }: {
    nixosConfigurations.lp4a = nixpkgs.lib.nixosSystem {
      # native platform
      system = "x86_64-linux";
      modules = [

        # add this module, to enable cross-compilation.
        {
          nixpkgs.crossSystem = {
            # target platform
            system = "riscv64-linux";
          };
        }

        # ...... other modules
      ];
    };
  };
}

More details:

3 Likes
3

Okay, this looks like it works, but I now run into another error which I can’t find the solution for:

error: builder for '/nix/store/da0rfihqp8yjifgkm5fyya0agdix88xc-linux-6.6.31.drv' failed with exit code 1;
       last 1 log lines:
       > error: executing '/nix/store/1fzg4cl3k2n9yq80vg6y1vcmvx3qm682-bash-5.2p26/bin/bash': No such file or directory
       For full logs, run 'nix log /nix/store/da0rfihqp8yjifgkm5fyya0agdix88xc-linux-6.6.31.drv'.
error: 1 dependencies of derivation '/nix/store/nrqbb4k6ds6abvnn6v2znv53vpk9l771-nixos-sd-image-23.11.20240709.205fd42-aarch64-linux.img-aarch64-unknown-linux-gnu.drv' failed to build

To clarify, I added the lines specified to the flake for the raspberry pi config, and removed the line that enables emulation in my host system config, so my raspberry pi flake now looks like this:

{
  description = "NixOS Raspberry Pi configuration";

  inputs = {
    ...
  };

  outputs = { self, nixpkgs, raspberry-pi-nix }:
    let
      inherit (nixpkgs.lib) nixosSystem;
      basic-config = { pkgs, lib, ... }: {
        ...
      };

    in {
      nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux"; # Changing this to the host system has no effect
        modules = [
          {
              nixpkgs.crossSystem = {
                # target platform
                system = "aarch64-linux";
              };
          }
          ./configuration.nix
          raspberry-pi-nix.nixosModules.raspberry-pi
          basic-config
        ];
      };
    };
}

And I’m running the code using

nix build '.#nixosConfigurations.rpi.config.system.build.sdImage' --system aarch64-linux

My best guess is I need to “inherit” more packages or something? Or inject some dependencies? The bash that the error says is unavailable is in fact available on the system, so I have no idea why it’s not seeing it, and I can only assume it expects it to be an aarch64 version of bash? But I’m not sure how to fix that.

4

OK Fixed.

You can try to avoid compilation locally by adding nix-community’s cache server, an example:

{
  description = "raspberry-pi-nix example";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    raspberry-pi-nix.url = "github:nix-community/raspberry-pi-nix";
  };

  nixConfig = {
    extra-substituters = [
      "https://nix-community.cachix.org"
    ];
    extra-trusted-public-keys = [
      "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
    ];
  };

  outputs = { self, nixpkgs, raspberry-pi-nix }:
    let
      inherit (nixpkgs.lib) nixosSystem;
      basic-config = { pkgs, lib, ... }: {
        # bcm2711 for rpi 3, 3+, 4, zero 2 w
        # bcm2712 for rpi 5
        # See the docs at:
        # https://www.raspberrypi.com/documentation/computers/linux_kernel.html#native-build-configuration
        raspberry-pi-nix.board = "bcm2711";
        time.timeZone = "America/New_York";
        users.users.root.initialPassword = "root";
        networking = {
          hostName = "basic-example";
          useDHCP = false;
          interfaces = {
            wlan0.useDHCP = true;
            eth0.useDHCP = true;
          };
        };
        hardware = {
          bluetooth.enable = true;
          raspberry-pi = {
            config = {
              all = {
                base-dt-params = {
                  # enable autoprobing of bluetooth driver
                  # https://github.com/raspberrypi/linux/blob/c8c99191e1419062ac8b668956d19e788865912a/arch/arm/boot/dts/overlays/README#L222-L224
                  krnbt = {
                    enable = true;
                    value = "on";
                  };
                };
              };
            };
          };
        };
      };

    in {
      nixosConfigurations = {
        rpi-example = nixosSystem {
          system = "aarch64-linux";
          modules = [ raspberry-pi-nix.nixosModules.raspberry-pi basic-config ];
        };
      };
    };
}

Or if you really want to cross compile the whole system locally, I have tested this demo works on my system:

{
  description = "raspberry-pi-nix example";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
    raspberry-pi-nix.url = "github:nix-community/raspberry-pi-nix";
  };

  outputs = {
    self,
    nixpkgs,
    raspberry-pi-nix,
  }: let
    inherit (nixpkgs.lib) nixosSystem;
    basic-config = {
      pkgs,
      lib,
      ...
    }: {
      # bcm2711 for rpi 3, 3+, 4, zero 2 w
      # bcm2712 for rpi 5
      # See the docs at:
      # https://www.raspberrypi.com/documentation/computers/linux_kernel.html#native-build-configuration
      raspberry-pi-nix.board = "bcm2711";
      time.timeZone = "America/New_York";
      users.users.root.initialPassword = "root";
      networking = {
        hostName = "basic-example";
        useDHCP = false;
        interfaces = {
          wlan0.useDHCP = true;
          eth0.useDHCP = true;
        };
      };
      hardware = {
        bluetooth.enable = true;
        raspberry-pi = {
          config = {
            all = {
              base-dt-params = {
                # enable autoprobing of bluetooth driver
                # https://github.com/raspberrypi/linux/blob/c8c99191e1419062ac8b668956d19e788865912a/arch/arm/boot/dts/overlays/README#L222-L224
                krnbt = {
                  enable = true;
                  value = "on";
                };
              };
            };
          };
        };
      };
    };
  in {
    nixosConfigurations = {
      rpi-example = nixosSystem {
        system = "x86_64-linux";
        modules = [
          raspberry-pi-nix.nixosModules.raspberry-pi
          basic-config
          {
            nixpkgs.crossSystem = {
              # target platform
              system = "aarch64-linux";
            };
          }
        ];
      };
    };
  };
}

The dependency graph is generated by GitHub - maralorn/nix-output-monitor: Pipe your nix-build output through the nix-output-monitor a.k.a nom to get additional information while building.

nix run nixpkgs#nix-output-monitor -- build '.#nixosConfigurations.rpi-example.config.system.build.sdImage'

image
image1920×960 160 KB

5

Yes, I was just starting to look into cachix when I saw this reply, thank you for explaining the correct way to use it. I should have done it to start with, really, but trying to figure out cross compilation was educational, even if I couldn’t quite get it to work for some reason.

1 Like