Planning for a better NixOS on ARM (and other non-x86_64 systems)

Development
1

I have been hinting about it a lot on the Matrix NixOS on ARM room, and beforehand on the IRC channel, but I have vague ideas for making the NixOS on ARM better, and I hope the best showcase of a totally generic mainline-based ARM Linux distribution.

While I’m the de facto lead on ARM things, I wouldn’t dare imply my word is final and the only way forward. (Though I do think it is the way forward.)

First, we need to look at the current state of things. Then I’ll point out issues. Finally I’ll be barfing out general notes about the overarching plan.

Current NixOS on ARM

Frankly speaking, we should be proud of how generic it already is for AArch64 and ARMv7.

While yes, we’re building SD images with the platform firmware required for the Raspberry Pi family of devices already built-in, they both use the mainline Linux kernel, which makes the images usable with other mainline-supported devices with their own platform firmware installs out of the box.

Less known, and less popular, are the UEFI iso. Yes, UEFI iso just like the boring x86_64 UEFI iso, but for ARM! These don’t have the drawbacks of the current SD images. Though they require the target system to have a platform firmware that knows how to boot using UEFI. Your favourite system is more likely than not to be able to boot from a USB UEFI iso than not.

:bulb: For ARMv6 it’s a bit of a different story. It is built specifically for the Raspberry Pi 1 family, but we can handwave this with something along the line “it’s a best effort unsupported image”. And it is.

Current issues

SD image conflicts

Those SD images are “pre-installed” systems which somewhat rehydrate on first boot. There are issues with this scheme.

First is that the filesystem UUIDs and labels are static, and the same across all images. This can cause issues when multiple storage medium have SD images written to them.

These pre-installed systems also make it needlessly hard to customize the system setup. For example, full-disk encryption, or alternative filesystems.

Further conflicts

Some platform firmware require installation on the same storage medium used for the SD image.

Currently this is handled through an out of band operation, and requires users to “burn” U-Boot to a specific offset that is platform dependent.

This offset can and for some platforms does conflict with the FAT32 partition used only for the Raspberry Pi’s own platform firmware.

Additionally, I find it’s bad form to ship images that work out of the box with the Raspberry Pi platform firmware, but none for any other systems. This is done only because it is really painful to install the platform firmware for the Raspberry Pi with out of band operations with such images.

Platform firmware and NixOS generations

Simply put, they exist in mismatched lifecycles. There is no way a NixOS system can manage the platform firmware through the generations lifecycle and get away with it.

But we do it with bootloaders already!

Yes, and it’s hard! Don’t confuse the platform firmware with the bootloader!

On most platforms there is no way to correctly rollback a botched update for the platform firmware. This makes it excessively inconvenient to recover, especially when most devices fallback to a “bricked-like” state when the platform firmware is invalid.

But we do it with bootloaders already!!

Yes, but there’s always a platform firmware to recover a b0rked update with!

If your GRUB update breaks booting on your x86_64 laptop, plop in a NixOS USB installer, do something, rebuild (reinstall) and voilà repaired!

Comparatively, if your platform firmware breaks, in the most common setups you will need to first learn about what a platform firmware is, e.g. learn what U-Boot is, re-install it with device-specific instructions, most likely from another computer.

If we think of ARM as an esoteric system composed entirely of purpose-specific Raspberry Pi boards, it doesn’t sound too bad.

ARM is not only for embedded and purpose-specific use cases. Look at the Pinebook Pro!

But we do it with bootloaders ALREADY!!!

And it’s way easier with bootloaders. Generally they have a very limited scope! Think about how many different GRUBs there are for your x86_64 systems. There’s about two! One for UEFI, one for legacy boot! systemd-boot only exists in one flavour!

Now that’s much easier! They use generally stable and agreed upon protocols (e.g. UEFI), and are universal for an architecture.

Now think about how many SBCs and ARM systems you know. It’s possible you have to pick that number and assume there’s that many different schemes for platform firmwares for those devices. Maybe more, maybe fewer.

On a single system, there may be multiple valid ways ways to install the platform firmware, in addition to many more (in my opinion) invalid ways.

Implementing the required machinery to update platform firmware automatically in a safe manner would be extremely hard. And the total benefit weighed against the drawbacks not worth it.

Raspberry pains

This is somewhat tangential, but must be handled with the NixOS on ARM revamp.

But we already manage the platform firmware for the Raspberry Pi??

Not really. If you’re thinking about boot.loader.raspberryPi, these options may not be doing what you think they are doing, and are part of the confusion. These come from an older boot scheme not really supported anymore, but kept around because of our inability to just delete possibly-still-in-use options.

The future NixOS on ARM

NOTE: This assumes a target board is already supported by mainline Linux.

Only support a single entirely device-agnostic boot image per architecture.

The way forward is to support the standards, the Arm Base Boot Requirements. Part of the Arm BBR are SBBR and EBBR. And from those, the only pertinent detail here is that they specify usage of UEFI.

What does this actually mean for NixOS? Not much! We’d be dropping the current SD images, and rely entirely on the existing and relatively well tested UEFI images for ARM.

It sounds like change, but in a way this is reducing complexity as it allows using the same concepts compared to x86_64.

With one caveat…

Bring your own platform firmware

Part of this plan requires re-thinking the way everyone thinks about the common ARM platforms. Everyone includes other distributions, and users outside of NixOS. Not really, we can go our own way and do it the right way anyway, but it would be easier if we can rally everyone behind this new way of doing things.

As the title implies, distributions should not provide platform firmware builds. Relatedly, device-specific images shouldn’t really be a common thing anymore.

With this in mind, I already started with the Tow-Boot project. This aims to provide opinionated generic U-Boot builds for the express purpose of using them like your old boring laptop’s UEFI.

Supporting only Tow-Boot is not a goal. Said images should work with standards and specifications in mind. So other options like the Raspberry Pi Tianocore EDKII builds should be as supported.

And this is where a change in philosophy is necessary. The first step in installing a Linux distribution on ARM will not start at the distribution. The first step will be, paraphrased, “Prepare your system according to the platform firmware setup instructions”.

Then, at this point, all ARM boards should share the same installation instructions. Yes it’s possible. We can do it today.

What’s left then?

Given Tow-Boot is a thing, and you can already install with UEFI, what’s left?

First order would be properly surveying users to understand what use cases are missing here.

In parallel, UEFI instructions should be put first as the “happy path” for NixOS on ARM and tested thoroughly to file down rough edges.

Then, the SD images should be removed.

But I rely on SD image builds in my custom use case

Sorry to hear that. They never were intended to be used through composition, and it’s a real thorn on the side of NixOS on ARM.

But fear not! I purposefully made it sound grim by saying SD images should be removed. The real way forward is rather: better semantics for image building are required. And this is not an ARM issue, this is a generic NixOS issue outright. This is not a topic for this thread. If you rely on custom image builds, assume this is handled at the same time completely externally to this plan.

Additionally, some work should be done to make it easier for users of systems not yet supported by mainline Linux to build correct UEFI iso with the required (presumably vendor) kernel forks. Not really hard or problematic, only needs to be handled. (Yes, it’s mostly as simple as you think it is.)

Work to be done (elsewhere)

While this is all a good plan already, there is some work I know is required elsewhere.

Documentation from at least one platform firmware provider should be written up and completed so we can start referencing to them. In an ideal world there would even be a “Linux on ARM” generic organization where the starting point would live for such documentation.

Figuring out an upgrade mechanism for the platform firmware. Not in NixOS, entirely out of band, provided by the platform firmware provider.

Then, not strictly required, but in my mind it’s always been part of the plan, is to complete enough of the UX work on Tow-Boot. Main missing features are a TUI to configure some things, like the boot order, and device-specific options handling.

Open questions

  • Who should manage the device tree used by the operating system?

    • Currently Linux assumes distros provide their binary builds and will be booted using the dtb files generated from the kernel build, and it’s a real world problem already.
    • It would be better to do as device tree was designed and leave it to the platform firmware’s FDT, but this requires the kernel people to stop fixing things entirely in their device tree “forks”.

  • (Assuming we need to load dtb files) how can GRUB (or any other EFI bootloader) load the correct dtb file according to the kernel’s own desires?

    • U-Boot has some semantics already for that, in which it (somewhat) hardcodes a platform name in the environment, which is used in the extlinux-compatible boot flow.
    • I’m proposing adding the /linux,dtb-name property to the platform firmware’s FDT which can be used like the (somewhat) hardcoded platform name in U-Boot.

  • Converting from “old scheme” images to “new scheme” images.

    • Will it be necessary? How can we ease the pain?

TLDR

  • SD image painful, remove.
  • UEFI all the things.
  • Conquer ARM.

Appendix

Technical notes

U-Boot is not a bootloader

It is way more than a bootloader.

It is first and foremost the platform firmware for the device. Think of U-Boot not like GRUB, but like your BIOS or UEFI in your boring “normal” computer.

The main point of confusion is that U-Boot also knows how to act as a bootloader. It can do so through direct boot methods (bootm, booti and similar), or schemes like extlinux compatible boot. U-Boot also can boot UEFI programs. Read more in README.distro.

Glossary

Platform Firmware

The platform firmware is the softwarey-firmwarey program that executes before the bootloader. Some platform firmware may also provide bootloader facilities. Example platform firmware include Tianocore EDKII and U-Boot.

Bootloader

The bootloader is the software part that handles loading the operating system image as needed for the specific target operating system. Generally speaking, common modern bootloaders are implemented as UEFI programs. Some platform firmware include *bootloader facilities, like U-Boot does.

Example bootloaders include: GRUB, systemd-boot, syslinux and rEFInd.

41 Likes
2

FAQ

Questions from this thread

None yet

3

Thanks for excellent writeup! I haven’t used NixOS on ARM yet, and am bit of a noon, but I have a Raspberry Pi coming up and would love to devote it for testing purposes if need be!

4

I did an installation of NixOS on a RPi3 with the EDK2 UEFI implementation. It worked surprisingly well (only minor issues).

2 Likes
5

I’m using SkiffOS as the “platform firmware” part with NixOS running within a container. Then the container and nix configs are portable across arm machines and even different architectures.

The only issue at the moment is Nix has limited caching for arm32 builds.

6

SkiffOS wouldn’t be defined as a platform firmware here. But fear not, this is not a negative thing!

SkiffOS is more akin to an hypervisor. I guess it would be called a container host operating system.

And yeah, the lack of an ARMv7l is annoying, but luckily a distinct issue from what I’m looking to work on and solve here.

Given that I’m looking to solve booting up to the kernel starting, it should change nothing about the container-based experience with SkiffOS.

1 Like
7

Nothing in particular. Don’t devote your Raspberry Pi for testing, use it, for whatever purposes you want!

I was mainly making the plans public so I could refer to them, rather than explaining every time from fresh :). Also taking the pulse of the community on the topic.

2 Likes
8

This installer image should be written to a USB drive, like usual. In a pinch, it may also be written to an SD image, if your target’s platform firmware does not need to be written to that same SD image. This UEFI iso is incompatible (by design) with shared firmware storage.

(from NixOS on ARM/UEFI - NixOS Wiki)

Will there be UEFI images that are compatible with shared firmware storage? It’s not a huge deal, but it feels silly to flash two drives when doing a fresh installation.

9

I hope we can end up providing the tooling for more enterprising individuals (or organizations) to build their own “final” disk images with their requirements, rather than try to ship a one-size-fits-all image that has some subtle, and some less subtle, issues. But that’s a separate goal from ARM things.

The whole reason two images are flashed with the shared storage strategy is to work around a deficiency in the target device; the “BIOS” (so to speak) of the system is missing and has been put on the Operating System storage of the computer. Flashing the platform firmware to the SD card is an operation that I would compare to “completing the assembly” of the device. Now your device is ready to be used with any standards-based boot operating system (like NixOS).

While extremely principled as an approach, I think we’re at a point in time where it is necessary to signal strongly that this is an issue, and this is the only solution for platforms without dedicated storage for the firmware.

The main benefit in the end is that this ends up abstracting all platforms into common steps:

  1. Get a standards-based boot platform firmware
  2. Install your OS with whichever partitioning scheme you want

Now, maybe I misread your question. Maybe your question was something like: can the firmware be on the USB drive with the installer iso? No, most platforms can’t start the platform firmware from USB, and the iso is built in a way where it’s likely impossible to add them anyway. Furthermore, it would only be “moving cheese”; the arduous device-specific steps of somehow getting the platform firmware for your specific SBC would need to exist still. And yet still, your target platform wouldn’t be bootable without the USB drive, unless you also somehow managed to flash the platform firmware in device-specific ways before installing.

Maybe I still misread your concerns. The issue may be about flashing two USB drives to then install to the SD card (e.g. on a Raspberry Pi). That wouldn’t be the case. You would prepare the SD card for the Raspberry Pi by flashing the platform firmware onto it. This also provides you with a started partition scheme that protects the platform firmware. Then you have to flash a secondary media (USB drive) to do the installation from.

Yet still, maybe there’s something I’m missing, if so do ask :).

(I preferred over-answering as other people may have had the questions I additionally answered.)

2 Likes
10

That’s close to what I was trying to ask, but not exactly. From the wiki page, my understanding is that when installing to SD you can currently have:

  • Firmware on dedicated storage, installer on USB drive
  • Firmware on dedicated storage, installer on target SD
  • Firmware on shared storage, installer on USB drive

Would it be possible in the future to have the firmware on shared storage and installer on the target SD?

11

I think I see a problem point. Let me try to clarify things. If that helps that means I need to work on improving the messaging about that.

When talking about shared storage and dedicated storage, these are mutually exclusive situations. We are talking about where the platform firmware (“BIOS” if you allow the anachronistic metaphor) of the system lives.

When a device supports dedicated storage, the platform firmware will be installed to a location that is dedicated for that firmware. It means that even if you removed the SD or eMMC storage, the firmware is still present and able to do some useful work. As an implementation detail, it will be commonly implemented as an “SPI Flash” type of storage.

When a device does not support dedicated storage for the firmware, we’re left to work with a shared storage strategy to work around the issue. The main idea here is that external projects will provide a base image with a partition scheme pre-configured to protect the platform firmware, and make it a tangible piece of the puzzle. Only a limited subset of platforms already require a partition (e.g. the Raspberry Pi family of hardware). Every other platforms, it’s more often than not done through splatting the U-Boot build to an arbitrary location, where you have to hope the end-user won’t write data into.

So, coming back to your concerns, when installing you have two different scenarios:

With a platform that supports installing as dedicated storage

  1. (Done at least once) install the platform firmware to the board.
  2. Install any operating system using standards-based boots (e.g. UEFI boot) to wherever the platform firmware can boot from. Be free to format and partition as you want.

With a platform that does nos support installing as dedicated storage

  1. Write the platform firmware image to a storage the platform can start from, e.g. SD card for Raspberry Pi; eMMC or SD for many A64 targets.
  2. Install any operating system using standards-based boots (e.g. UEFI boot) to either the internal storage, or any other storage that the platform firmware can boot from.
    • If installing to the same storage the platform firmware is installed to, be mindful about the existing partition scheme.
    • Else you’re free to partition other target storage as you want.

With the first scenario, you need to “finish assembling” the board by installing the firmware where it should have been installed by the factory (imo!). With the second scenario, the end-user has to start from a known-good starting point for the board’s internal storage.

Finally, to directly answer with concrete examples

With the ever-popular Raspberry Pi platform, you would:

  1. Burn an SD card with the shared storage strategy pre-installed firmware.
  2. Burn a USB drive with the UEFI NixOS installer
  3. When installing, before using nixos-generate-config, you partition and format starting from the existing partitions on the SD card.

With the Pinebook Pro (or really any RK3399 platforms, and many other boards)

  1. (Do at least once) start from a supported storage device (e.g. SD card) and install to the dedicated storage. (This storage device can be erased/reset once this is done.)
  2. Burn a USB drive with the UEFI NixOS installer
  3. When installing, before using nixos-generate-config, you partition and format starting from scratch.

There are multiple strong benefits here from not using a pre-partitioned-and-formatted SD card:

  • Unique identifiers are generated for your install (partition UUID, filesystem UUID)
  • Freedom to choose the filesystem(s) to use
  • Easier possible to do full-disk-encryption

When starting from a pre-baked SD card, you’re stuck with the choices made for you. It becomes harder than necessary to change these choices.

And a final benefit here, is that the installation instructions can become unified across the different platforms. No need for special AArch64 instructions as far as the installation goes! Conversely: your knowledge about installing on x86_64 applies to AArch64 too.

Don’t feel bad if I missed anything still. It’s on me to make the messaging as crystal clear as possible, and I’m cursed with having all the knowledge already at the top of my mind :). So just ask more clarification questions if you have some more.

2 Likes
12

I think the answer is that this isn’t possible, but I also think I’m doing a really bad job of asking what I want. What I’d like is this installation process for systems that only support shared storage:

  1. Flash firmware to boot SD (somehow)
  2. Flash installer to boot SD (somehow, possibly fused with the previous step)
  3. Boot installer from SD to RAM
  4. Delete installer partitions from SD, leaving only firmware
  5. Create new target partition(s)
  6. Install targeting there
13

Specifically, my thoughts are that while the process for systems that already have firmware on dedicated storage is the same as on x86_64, we don’t live in a world where that’s the case for most ARM systems. It isn’t a huge amount of friction, but it’d be great if a standards-based boot process didn’t involve more work when installing compared to distributions like Raspbian.

I’d also note that I strongly agree with your approach in general, and it’s how I’ve had my Pine A64+ (currently running Debian, I plan to move it to NixOS when 21.11 releases) set up for a while now. I also think that the method I’m describing should work for Debian, but it didn’t work in practice for reasons I wasn’t able to figure out (a naive installation would fail with a confusing error message, I eventually found a workaround but the installer tried and failed to replace U-Boot with one from the Debian repositories which I couldn’t find a way to bypass).

14

This could be possible… Assuming there’s enough RAM. It would require a different installer build I think. Same story as on x86_64, the installer is built in a special way that would break when not written at the start of a storage media.

The only reason one would need to do this (imo) if it they don’t have an additional valid boot medium handy.

And it is a shame. Speaking of shame, there is a not-so-hidden agenda here to shame ARM platforms where EBBR can’t be followed correctly due to the lack of a dedicated storage area for the platform firmware. These lacking platforms will get more cumbersome to install to, in a way that I hope will provide incentives for board makers to build “finished” products, especially if other distros follow suit.

I personally think the “less” work Raspbian-like distros is an illusion. It seems all of these distros have long post-setup steps to follow anyway for a lot of use cases.

Furthermore, the actual real reason they’re bad is it promotes an non-sustainable workflow where distributions need to build and provide images for every single boards. We’re already not doing this for non-Raspberry Pi systems by providing a unique image, and leaving final assembly to the end-user. In many ways, we’re mostly shuffling existing NixOS installation steps around.

Yes, such standards-based boot methods should work with other distros, assuming a UEFI install. Though a lot of distros have assumptions that don’t hold with the limited UEFI support U-Boot provides. Mainly Ubuntu and Debian, for different reasons actually, both fail hard the installation since UEFI vars can’t be changed to add the UEFI boot entry for the bootloader.

When working on Tow-Boot, I tested other distros’ UEFI installers on ARM too, since it’s not meant to be a NixOS project. It is a project for all distros and all boards.

Assuming non-UEFI installs, this is the flip side of the coin: distributions shouldn’t manage the platform firmware. The distributions themselves shouldn’t, but other tools like fwupd could, as long as it’s done just like it would be for an x86_64 UEFI firmware. This goes in the same direction about things becoming unmaintainable: every distributions would otherwise have to provide their own updater scheme, provided for every boards.

With that said…

I hope to figure out a good strategy for enterprising individuals that want to build their own pre-configured disk images as a whole “product” output. So with Nix it may be possible to have your cake and eat it too. Provide UEFI installers for standards-based boot and “classic” installation schemes for “pet”-like management of an SBC. Provide the tooling necessary for “cattle”-like deployment to SBCs (of standards-based boot still).

1 Like
15

Thanks, that answers all my questions.

The cost-benefit ratio here is very questionable, but the RAM usage could be heavily reduced by network mounting /nix/.ro-store. I don’t think this is worthwhile, though.

16

And that would quickly increase complexity!

1 Like
17

How will this interact with android based devices?

18

It will not. Even other mobile devices that could follow this scheme don’t have to.

This plan is squarely for platforms that already or can be made to follow standards-based boots.

Mobile devices are varied, complex enough, and boot in annoying ways. Mobile NixOS continues providing the necessary abstractions up to and including the kernel boot.

19

I did not realize generic images were possible for ARM! That is massively exciting, thank you for working to (hopefully) bring about a future where that’s the typical way things are done.

4 Likes
20

This plan is absolutely fantastic IMO.
I tried this week to get started with a Pi 4 and was successful but it annoyed me that I couldn’t use my standard setup and had some problems with Flakes.

I’m going to test if the UEFI approach works better, at least I’m familiar with it :slight_smile: .

The firmware part could be quite easy no?
Flash an image with a small partition and that’s that? Looks like that’s exactly what you have to do with Tow-Boot💪.

The cattle part sounds interesting as well but I haven’t tried yet to build an image myself.

1 Like