Planning for a better NixOS on ARM

Hi! :wave: been a while since I was active on the Discourse.

I have been hinting about it a lot on the Matrix NixOS on ARM room, and beforehand on the IRC channel, but I have vague ideas for making the NixOS on ARM better, and I hope the best showcase of a totally generic mainline-based ARM Linux distribution.

While I’m the de facto lead on ARM things, I wouldn’t dare imply my word is final and the only way forward. (Though I do think it is the way forward.)

First, we need to look at the current state of things. Then I’ll point out issues. Finally I’ll be barfing out general notes about the overarching plan.

Current NixOS on ARM

Frankly speaking, we should be proud of how generic it already is for AArch64 and ARMv7.

While yes, we’re building SD images with the platform firmware required for the Raspberry Pi family of devices already built-in, they both use the mainline Linux kernel, which makes the images usable with other mainline-supported devices with their own platform firmware installs out of the box.

Less known, and less popular, are the UEFI iso. Yes, UEFI iso just like the boring x86_64 UEFI iso, but for ARM! These don’t have the drawbacks of the current SD images. Though they require the target system to have a platform firmware that knows how to boot using UEFI. Your favourite system is more likely than not to be able to boot from a USB UEFI iso than not.

:bulb: For ARMv6 it’s a bit of a different story. It is built specifically for the Raspberry Pi 1 family, but we can handwave this with something along the line “it’s a best effort unsupported image”. And it is.

Current issues

SD image conflicts

Those SD images are “pre-installed” systems which somewhat rehydrate on first boot. There are issues with this scheme.

First is that the filesystem UUIDs and labels are static, and the same across all images. This can cause issues when multiple storage medium have SD images written to them.

These pre-installed systems also make it needlessly hard to customize the system setup. For example, full-disk encryption, or alternative filesystems.

Further conflicts

Some platform firmware require installation on the same storage medium used for the SD image.

Currently this is handled through an out of band operation, and requires users to “burn” U-Boot to a specific offset that is platform dependent.

This offset can and for some platforms does conflict with the FAT32 partition used only for the Raspberry Pi’s own platform firmware.

Additionally, I find it’s bad form to ship images that work out of the box with the Raspberry Pi platform firmware, but none for any other systems. This is done only because it is really painful to install the platform firmware for the Raspberry Pi with out of band operations with such images.

Platform firmware and NixOS generations

Simply put, they exist in mismatched lifecycles. There is no way a NixOS system can manage the platform firmware through the generations lifecycle and get away with it.

But we do it with bootloaders already!

Yes, and it’s hard! Don’t confuse the platform firmware with the bootloader!

On most platforms there is no way to correctly rollback a botched update for the platform firmware. This makes it excessively inconvenient to recover, especially when most devices fallback to a “bricked-like” state when the platform firmware is invalid.

But we do it with bootloaders already!!

Yes, but there’s always a platform firmware to recover a b0rked update with!

If your GRUB update breaks booting on your x86_64 laptop, plop in a NixOS USB installer, do something, rebuild (reinstall) and voilà repaired!

Comparatively, if your platform firmware breaks, in the most common setups you will need to first learn about what a platform firmware is, e.g. learn what U-Boot is, re-install it with device-specific instructions, most likely from another computer.

If we think of ARM as an esoteric system composed entirely of purpose-specific Raspberry Pi boards, it doesn’t sound too bad.

ARM is not only for embedded and purpose-specific use cases. Look at the Pinebook Pro!

But we do it with bootloaders ALREADY!!!

And it’s way easier with bootloaders. Generally they have a very limited scope! Think about how many different GRUBs there are for your x86_64 systems. There’s about two! One for UEFI, one for legacy boot! systemd-boot only exists in one flavour!

Now that’s much easier! They use generally stable and agreed upon protocols (e.g. UEFI), and are universal for an architecture.

Now think about how many SBCs and ARM systems you know. It’s possible you have to pick that number and assume there’s that many different schemes for platform firmwares for those devices. Maybe more, maybe fewer.

On a single system, there may be multiple valid ways ways to install the platform firmware, in addition to many more (in my opinion) invalid ways.

Implementing the required machinery to update platform firmware automatically in a safe manner would be extremely hard. And the total benefit weighed against the drawbacks not worth it.

Raspberry pains

This is somewhat tangential, but must be handled with the NixOS on ARM revamp.

But we already manage the platform firmware for the Raspberry Pi??

Not really. If you’re thinking about boot.loader.raspberryPi, these options may not be doing what you think they are doing, and are part of the confusion. These come from an older boot scheme not really supported anymore, but kept around because of our inability to just delete possibly-still-in-use options.

The future NixOS on ARM

NOTE: This assumes a target board is already supported by mainline Linux.

Only support a single entirely device-agnostic boot image per architecture.

The way forward is to support the standards, the Arm Base Boot Requirements. Part of the Arm BBR are SBBR and EBBR. And from those, the only pertinent detail here is that they specify usage of UEFI.

What does this actually mean for NixOS? Not much! We’d be dropping the current SD images, and rely entirely on the existing and relatively well tested UEFI images for ARM.

It sounds like change, but in a way this is reducing complexity as it allows using the same concepts compared to x86_64.

With one caveat…

Bring your own platform firmware

Part of this plan requires re-thinking the way everyone thinks about the common ARM platforms. Everyone includes other distributions, and users outside of NixOS. Not really, we can go our own way and do it the right way anyway, but it would be easier if we can rally everyone behind this new way of doing things.

As the title implies, distributions should not provide platform firmware builds. Relatedly, device-specific images shouldn’t really be a common thing anymore.

With this in mind, I already started with the Tow-Boot project. This aims to provide opinionated generic U-Boot builds for the express purpose of using them like your old boring laptop’s UEFI.

Supporting only Tow-Boot is not a goal. Said images should work with standards and specifications in mind. So other options like the Raspberry Pi Tianocore EDKII builds should be as supported.

And this is where a change in philosophy is necessary. The first step in installing a Linux distribution on ARM will not start at the distribution. The first step will be, paraphrased, “Prepare your system according to the platform firmware setup instructions”.

Then, at this point, all ARM boards should share the same installation instructions. Yes it’s possible. We can do it today.

What’s left then?

Given Tow-Boot is a thing, and you can already install with UEFI, what’s left?

First order would be properly surveying users to understand what use cases are missing here.

In parallel, UEFI instructions should be put first as the “happy path” for NixOS on ARM and tested thoroughly to file down rough edges.

Then, the SD images should be removed.

But I rely on SD image builds in my custom use case

Sorry to hear that. They never were intended to be used through composition, and it’s a real thorn on the side of NixOS on ARM.

But fear not! I purposefully made it sound grim by saying SD images should be removed. The real way forward is rather: better semantics for image building are required. And this is not an ARM issue, this is a generic NixOS issue outright. This is not a topic for this thread. If you rely on custom image builds, assume this is handled at the same time completely externally to this plan.

Additionally, some work should be done to make it easier for users of systems not yet supported by mainline Linux to build correct UEFI iso with the required (presumably vendor) kernel forks. Not really hard or problematic, only needs to be handled. (Yes, it’s mostly as simple as you think it is.)

Work to be done (elsewhere)

While this is all a good plan already, there is some work I know is required elsewhere.

Documentation from at least one platform firmware provider should be written up and completed so we can start referencing to them. In an ideal world there would even be a “Linux on ARM” generic organization where the starting point would live for such documentation.

Figuring out an upgrade mechanism for the platform firmware. Not in NixOS, entirely out of band, provided by the platform firmware provider.

Then, not strictly required, but in my mind it’s always been part of the plan, is to complete enough of the UX work on Tow-Boot. Main missing features are a TUI to configure some things, like the boot order, and device-specific options handling.

Open questions

  • Who should manage the device tree used by the operating system?

    • Currently Linux assumes distros provide their binary builds and will be booted using the dtb files generated from the kernel build, and it’s a real world problem already.
    • It would be better to do as device tree was designed and leave it to the platform firmware’s FDT, but this requires the kernel people to stop fixing things entirely in their device tree “forks”.
  • (Assuming we need to load dtb files) how can GRUB (or any other EFI bootloader) load the correct dtb file according to the kernel’s own desires?

    • U-Boot has some semantics already for that, in which it (somewhat) hardcodes a platform name in the environment, which is used in the extlinux-compatible boot flow.
    • I’m proposing adding the /linux,dtb-name property to the platform firmware’s FDT which can be used like the (somewhat) hardcoded platform name in U-Boot.
  • Converting from “old scheme” images to “new scheme” images.

    • Will it be necessary? How can we ease the pain?


  • SD image painful, remove.
  • UEFI all the things.
  • Conquer ARM.


Technical notes

U-Boot is not a bootloader

It is way more than a bootloader.

It is first and foremost the platform firmware for the device. Think of U-Boot not like GRUB, but like your BIOS or UEFI in your boring “normal” computer.

The main point of confusion is that U-Boot also knows how to act as a bootloader. It can do so through direct boot methods (bootm, booti and similar), or schemes like extlinux compatible boot. U-Boot also can boot UEFI programs. Read more in README.distro.


Platform Firmware

The platform firmware is the softwarey-firmwarey program that executes before the bootloader. Some platform firmware may also provide bootloader facilities. Example platform firmware include Tianocore EDKII and U-Boot.


The bootloader is the software part that handles loading the operating system image as needed for the specific target operating system. Generally speaking, common modern bootloaders are implemented as UEFI programs. Some platform firmware include *bootloader facilities, like U-Boot does.

Example bootloaders include: GRUB, systemd-boot, syslinux and rEFInd.



Questions from this thread

None yet

Thanks for excellent writeup! I haven’t used NixOS on ARM yet, and am bit of a noon, but I have a Raspberry Pi coming up and would love to devote it for testing purposes if need be!

I did an installation of NixOS on a RPi3 with the EDK2 UEFI implementation. It worked surprisingly well (only minor issues).

1 Like

I’m using SkiffOS as the “platform firmware” part with NixOS running within a container. Then the container and nix configs are portable across arm machines and even different architectures.

The only issue at the moment is Nix has limited caching for arm32 builds.

SkiffOS wouldn’t be defined as a platform firmware here. But fear not, this is not a negative thing!

SkiffOS is more akin to an hypervisor. I guess it would be called a container host operating system.

And yeah, the lack of an ARMv7l is annoying, but luckily a distinct issue from what I’m looking to work on and solve here.

Given that I’m looking to solve booting up to the kernel starting, it should change nothing about the container-based experience with SkiffOS.

1 Like

Nothing in particular. Don’t devote your Raspberry Pi for testing, use it, for whatever purposes you want!

I was mainly making the plans public so I could refer to them, rather than explaining every time from fresh :). Also taking the pulse of the community on the topic.

1 Like