How to cross-compile the SD Card images?

Help
1

I saw this example of producing an armv7l SD Card image, but it assumes that the host you’re running nix-build on is an armv7l machine. In my case, I want to produce an SD Card image for an Orange Pi Zero with 256MB of RAM, which will never be able to run the Nix tooling natively.

So, how do I cross-compile an SD Card image for armv7l from an X86 host? I’ve read a bit about crossSystem and assume that this is how it’s supposed to be done, but get lost very quickly. Is it possible someone could give me a minimal example for this?

1 Like
2

Despite being commented out, rpizero1 in my flake is a cross compiled system (x86_64-linux → armv7l-linux): https://github.com/colemickens/nixcfg/blob/b9b033a12cc8aa140e847aa27c951f9eead314eb/flake.nix#L188

But the important parts are:

  1. a specific pinned nixpkgs. for me, whatever crosspkgs is in my flake inputs was a known good revision for (x86_64->aarch64 crossing). Your luck will vary with your cross, whatever rev you’re on, what exact packages you need, what the caches have, etc.

  2. that nixpkgs was imported with system = x86_64-linux: https://github.com/colemickens/nixcfg/blob/b9b033a12cc8aa140e847aa27c951f9eead314eb/flake.nix#L188

  3. and then later, in the machine config, https://github.com/colemickens/nixcfg/blob/b9b033a12cc8aa140e847aa27c951f9eead314eb/hosts/rpizero1/configuration.nix#L25:

      nixpkgs.crossSystem = lib.systems.examples.raspberryPi;

EDIT: Sorry, rpizero1 = (Raspberry Pi Zero W). So, my cross target was armv6, not armv7l, so again, see caveat about finding a known good nixpkgs where the core pkgs cross compile successfully. This can be confirmed by following systems.examples.raspberryPi to here: https://github.com/NixOS/nixpkgs/blob/1c65a509fbbc17a2853a657ea1391de0aab9e793/lib/systems/examples.nix#L37-L39

  raspberryPi = {
    config = "armv6l-unknown-linux-gnueabihf";
  } // platforms.raspberrypi;
1 Like
3

I haven’t personally tried it but the wiki mentions an interesting alternative to cross-compiling namely “native”-compilation using QEMU, which could remove some of the problems that cross-compilation comes with.
https://nixos.wiki/wiki/NixOS_on_ARM
Have you already seen or tried that?

4

I’ve tried that, at least for armv7l it is unbearably slow. It was taking many many minutes to compile the hello derivation on a Ryzen 3900X, I cancelled after 20 minutes.

5

Oh. That’s unfortunate.

6

I am afraid that it’s not going to help but this is what I did:
GitHub - gytis-ivaskevicius/orangepi-r1-plus-nixos-image: NixOS port for OrangePi R1 Plus. WIP (one of my unfinished projects, need to go over some changes that are pushed to separate branch)

And this for cross compilation:

As for compile speed - I’d say its totally fine if not great, but then again - I have 5950x :smiley:

1 Like
7

So how would I build this? rpizero1 is not a derivation, so nix build .#nixosConfigurations.rpizero1 would fail with error: flake output attribute 'nixosConfigurations.rpizero1' is not a derivation.

8

I’m lazy, so I can do nix build .#toplevels.rpizero1. Which is defined here: https://github.com/colemickens/nixcfg/blob/b788e1678ca88d4840b688aa9d20b44f153ac3db/flake.nix#L206-L208

toplevels = genAttrs
   (builtins.attrNames inputs.self.outputs.nixosConfigurations)
   (attr: nixosConfigurations.${attr}.config.system.build.toplevel);

which can be read as “generate an attribute set, use the attribute names from nixosConfiguration as keys, and use the nixosConfigurations.<x>.config.system.build.toplevel as the value”.

Thus, nix build .#nixosConfigurations.rpizero1.config.system.build.toplevel evaluates to the same thing as nix build .#toplevels.rpizero1.

1 Like
9

I have managed to do that with armv7l in place of your rpizero1 config, but what do I do with the result? How am I intended to flash this to an SD card and boot it? I was expecting an ISO, not an activation script. What is your process?

[matthew@swordfish:~/tmp/armv7l/nixcfg]$ ls result
activate               dtbs                init                    kernel-modules  sw
append-initrd-secrets  etc                 init-interface-version  kernel-params   system
bin                    extra-dependencies  initrd                  nixos-version   systemd
configuration-name     firmware            kernel                  specialisation
10

Same as usual with these sorts of NixOS-level build artifacts – somewhere in my config I added an import. That import added stuff into the nixos build that resulted in an extra output attribute. So, in addition to the config.system.build.toplevel, the rpizero1 build also now has config.system.build.sdImage.

So, either of:

  • nix build github:colemickens/nixcfg#images.rpizero1
  • nix build github:colemickens/nixcfg#nixosConfigurations.rpizero1.config.system.build.sdImage

will give an img you can flash.

In my case, for some reason it seems like I carried my own uboot config? But then I include a regular nixos module that has the rest of the sdImage magic: https://github.com/colemickens/nixcfg/blob/ac4971f7285af12514ab597b5257ed4ffe22baa0/hosts/rpizero1/sd-image-raspberrypi.nix

3 Likes
11

While I’m sure it’s far from idiomatic, searchers may be interested in my example below:

Points of interest include:

  • cross compiles an SD image that is bootable on an RPi3 with nix build
  • uses pkgs.vmTools.runInLinuxVM to create a subvolumed, compressed, BTRFS-root partition layout for the image
  • adds some customizations to uboot to support BTRFS and ZSTD compression
  • provides a booting VM target for local testing (though doesn’t do much other than boot – thread)
  • uses GitHub Actions to automatically build the image in CI (on tagged releases), and uploads the image to Releases · n8henrie/nixos-btrfs-pi · GitHub where then can be directly downloaded and burned to an SD card

I’m sure others will want to customize and probably take out a lot of cruft, but I’m blown away that this can all be done with nix + GitHub Actions, newbies (like myself) might find the example interesting.

4 Likes