Compiling rpi4 kernel in on x86_64 nixos-hardware slow

[penguin_brian]

As in it takes many hours, and there doesn’t appear to be any cached build available.

Any better options?

It is slow because it wants to build via emulator. So I guess in theory setting up a cross build would be faster. But my attempts to cross build the rpi4 kernel have been failures so far. As in it typically wants to cross compile all the build tools too, and that ends being very slow and failing.

It use to be the rpi4 kernel was available in nixpkgs, and cached, but I think this is no longer the case.

[codesupply]

Having been down the cross compilation/emulation rabbit hole I now keep a remote VM running aarch64-linux for this purpose. I set it up as a remote builder.

[codesupply]

Also see the nixos raspberry pi project, which has some cachix caching.

[adomixaszvers]

I have migrated my RPI4 to a generic kernel rpi4: migrate to UEFI boot · adomixaszvers/dotfiles-nix@6c05cd9 · GitHub but it was not straightforward because I did not know which kernel modules were needed to boot.

[penguin_brian]

I would like to go to the mainstream kernel also. But most of my RPI4 use the “Raspberry [7 inch] Pi Touch Display”[1]. And last time I tried to do that with the mainstream kernel I wasted hours and never got anywhere. While it has been a while since I tested this last, LLM seems to think I would still have problems - it says there are missing parts from the mainstream kernel still.

Notes

[1] There are other displays - of varying degrees of goodness - I can get that use HDMI, but none of them (that I could find) have integrated cases for both both Rpi4 and the display.

[penguin_brian]

Not clear here, is nixos-raspberrypi something I use instead of nixos-hardware? Or in addition?

[StepBroBD]

curious if anyone got pi5 to boot on generic kernel

[tree-sapii]

I encountered similar problem. My pi5 kernel build ( only change was size optimizations ) took more than 6 hours ( and then I gave up ), only I had binfmt enabled instead of a proper cross comp setup, but I swear that QEMU doesn’t have this much overhead, a x86 kernel build took less than 2 hours on the same machine.

[eblechschmidt]

I’m building it over night. Never looked at the exact time it took but was always done the next morning.

[penguin_brian]

Tried nixos-raspberrypi out.

Apparently a significant benefit is you get a chose of bootloaders. And uboot - required by nixos-hardware - supposedly has an issue on rpi5 in that USB devices such as keyboards don’t work during boot (this is what LLM told me; I haven’t verified this).

But I noticed the kernel version is old, so I filled a bug report. kernel version old and insecure · Issue #191 · nvmd/nixos-raspberrypi · GitHub

If I need to manually update the kernel, I lose the benefit of the caching, and it probably isn’t worth while for me.

[eblechschmidt]

Funny thing I have to revisit this topic again. This time the kernel did not compile over night.

[penguin_brian]

Seems my bug report against nixos-raspberrypi was resolved rather promptly - (in the develop branch not the main branch), so continuing to try that out again.

Config is rather different from nixos-hardware, and they appear to be mutually exclusive. So will be a question of if I dare risk trying deploying

With this, building linux-config is still on the slow side, but unlike building the entire kernel before, but this takes minutes, not hours.

Setting up a remote aarch64-linux VM is not something I thought of. If this doesn’t work, that might be a good fall back.

[penguin_brian]

OK, nixos-raspberrypi worked fine.

Initially had some issues with /boot/firmware. First it wasn’t mounted by default, and it now is required by the deployment:

[…]
uboot-builder: -f /boot/firmware -b /boot -c /nix/store/qkvr20wnp7p7m2aph81a6al85f6l7k83-nixos-system-brian-26.05.20260531.b51242d
copying firmware…
using vendor firmware from /nix/store/fk8ls7ihhp39rwsq5kilwgqzdqwcrqzq-raspberrypi-firmware-1.20260521/share/raspberrypi/boot
cp: cannot create regular file ‘/boot/firmware/bcm2708-rpi-b.dtb.tmp’: No such file or directory
Failed to install bootloader
[…]

Fix: change config to remove the noauto on the mount point, and run manually on each host:

mkdir -p /boot/firmware && mount /dev/disk/by-label/FIRMWARE /boot/firmware

Then I ran out of diskspace on this partition, because it was full of legacy deployment attempts with rpi2 and rpi3 files that I no longer required. And the partition I got was small (30M).

rm /boot/firmware/*

This seemed scary, but deployment worked, and reboot worked.

[rnhmjoj]

@tree-sapii

I encountered similar problem. My pi5 kernel build ( only change was size optimizations ) took more than 6 hours ( and then I gave up )

@eblechschmidt

I’m building it over night. Never looked at the exact time it took but was always done the next morning.

If a custom kernel build takes that long there’s definitely something wrong.
My guess would be that you’re using the default config, which pratically builds every kernel module, including those that are completely unrelated to a raspberry pi.

See this:

Kernel build uses more than 16 GB of space

The default kernel configuration in NixOS is insanely bloated, don’t even try to compile it unless you have some server as a dedicated build farm. It can easily take hours to build on a desktop, even if you manage to get enough disk and memory to do it. So, you can’t simply add your patch to boot.kernelPatches and call it a day, but there is a way around. You can generate a kernel configuration that enables only the modules you really need by looking at what is currently loaded: you have to run…

[eblechschmidt]

Thanks for your reply. Tbh. my knowledge about compiling kernels is quite limited. I use nixos-hardware which probably provides the kernel I use?! From a quick check I would suspect it is this derivation: nixos-hardware/raspberry-pi/common/kernel.nix at 6358ff76821101c178e3ab4919a62799bfe3652e · NixOS/nixos-hardware · GitHub

which is referenced like this:

kernelPackages = lib.mkDefault (
      pkgs.linuxPackagesFor (pkgs.callPackage ../common/kernel.nix { rpiVersion = 4; })
    );

How can I see if this is using the default config and what would I need to change?

[penguin_brian]

As per my original message, I think a big part of the slow down is that most people have amd64, and rpi uses arch64, a completely difference instruction set. To do this, nix by default wants to run everything via an emulator. Which I think is very slow when running gcc.

Some people have said it is a lot faster compiling on a native arch64 platform.

Which does raise some questions.

Some people feel that QEMU should not be so slow compiling the kernel. It is not just QEMU here, it is emulated instruction set too. Maybe that makes the difference.

And in theory setting up proper cross-compilation should work, but it looks like I am not the only one who failed trying to do that.

[rnhmjoj]

You can build the attribute config.boot.kernelPackages.kernel.configfile to get the kernel config file.

Take a look at the file, see what modules are enabled and then compare to make localmodconfig generates.

If you don’t have any experience building a custom kernel, you can start from here.