# Planning for a better NixOS on ARM (and other non-x86\_64 systems)
**URL:** https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346
**Category:** Development
**Created:** [October 4, 2021, 10:05pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346 "2021-10-04T22:05:23Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 4, 2021, 10:05pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/1 "2021-10-04T22:05:23Z")
I have been hinting about it a lot on the Matrix [NixOS on ARM](https://matrix.to/#/#nixos-on-arm:nixos.org) room, and beforehand on the IRC channel, but I have vague ideas for making the NixOS on ARM better, and I hope **the best** showcase of a totally generic mainline-based ARM Linux distribution.
While I’m the de facto lead on ARM things, I wouldn’t dare imply my word is final and the only way forward. (Though I do think it _is_ the way forward.)
First, we need to look at the current state of things. Then I’ll point out issues. Finally I’ll be barfing out general notes about the overarching plan.
# Current NixOS on ARM
Frankly speaking, we should be proud of _how generic_ it already is for `AArch64` and `ARMv7`.
While yes, we’re building **SD images** with the _platform firmware_ required for the Raspberry Pi family of devices already built-in, they both use the _mainline_ Linux kernel, which makes the images usable with other mainline-supported devices with their own _platform firmware_ installs out of the box.
Less known, and less popular, are the **UEFI iso**. Yes, UEFI iso just like the boring `x86_64` UEFI iso, but for ARM! These don’t have the drawbacks of the current SD images. Though they require the target system to have a _platform firmware_ that knows how to boot using UEFI. Your favourite system is more likely than not to be able to boot from a USB UEFI iso than not.
> 💡 For `ARMv6` it’s a bit of a different story. It is built specifically for the Raspberry Pi 1 family, but we can handwave this with something along the line “it’s a best effort unsupported image”. And it is.
## Current issues
### SD image conflicts
Those SD images are “pre-installed” systems which somewhat rehydrate on first boot. There are issues with this scheme.
First is that the filesystem UUIDs and labels are static, and the same across all images. This can cause issues when multiple storage medium have SD images written to them.
These pre-installed systems also make it needlessly hard to customize the system setup. For example, full-disk encryption, or alternative filesystems.
### Further conflicts
Some _platform firmware_ require installation on the same storage medium used for the SD image.
Currently this is handled through an out of band operation, and requires users to “burn” U-Boot to a specific offset that is platform dependent.
This offset _can_ and for some platforms _does_ conflict with the FAT32 partition used only for the Raspberry Pi’s own _platform firmware_.
Additionally, I find it’s bad form to ship images that work out of the box with the Raspberry Pi _platform firmware_, but none for any other systems. This is done only because it is _really painful_ to install the _platform firmware_ for the Raspberry Pi with out of band operations with such images.
### Platform firmware and NixOS generations
Simply put, they exist in mismatched lifecycles. There is no way a NixOS system can manage the _platform firmware_ through the generations lifecycle and get away with it.
> But we do it with _bootloaders_ already!
Yes, and it’s hard! Don’t confuse the _platform firmware_ with the _bootloader_!
On most platforms there is no way to correctly rollback a botched update for the _platform firmware_. This makes it excessively inconvenient to recover, **especially** when most devices fallback to a “bricked-like” state when the _platform firmware_ is invalid.
> But we do it with _bootloaders_ already!!
Yes, but there’s always a _platform firmware_ to recover a b0rked update with!
If your GRUB update breaks booting on your `x86_64` laptop, plop in a NixOS USB installer, do _something_, rebuild (reinstall) and _voilà_ repaired!
Comparatively, if your _platform firmware_ breaks, in the most common setups you will need to first _learn about what a platform firmware is_, e.g. learn what U-Boot is, re-install it with device-specific instructions, most likely from another computer.
If we think of ARM as an esoteric system composed entirely of purpose-specific Raspberry Pi boards, it doesn’t sound too bad.
ARM is **not** only for embedded and purpose-specific use cases. Look at the Pinebook Pro!
> But we do it with _bootloaders_ ALREADY!!!
And it’s way easier with _bootloaders_. Generally they have a very limited scope! Think about how many different GRUBs there are for your `x86_64` systems. There’s about two! One for UEFI, one for legacy boot! `systemd-boot` only exists in one flavour!
Now that’s much easier! They use generally stable and agreed upon protocols (e.g. UEFI), and are universal for an architecture.
Now think about how many SBCs and ARM systems you know. It’s possible you have to pick that number and assume there’s that many different _schemes_ for _platform firmwares_ for those devices. Maybe more, maybe fewer.
On a single system, there may be _multiple valid ways_ ways to install the _platform firmware_, in addition to _many more (in my opinion) invalid ways_.
Implementing the required machinery to update _platform firmware_ automatically in a safe manner _would be extremely hard_. And the total benefit weighed against the drawbacks not worth it.
### Raspberry pains
This is somewhat tangential, but must be handled with the NixOS on ARM revamp.
> But we already manage the _platform firmware_ for the Raspberry Pi??
Not really. If you’re thinking about [`boot.loader.raspberryPi`](https://search.nixos.org/options?channel=unstable&query=boot.loader.raspberryPi), these options _may not be doing what you think they are doing_, and are part of the confusion. These come from an older boot scheme not really supported anymore, but kept around because of our inability to just delete possibly-still-in-use options.
# The future NixOS on ARM
> **NOTE** : This assumes a target board is already supported by mainline Linux.
Only _support_ a single **entirely** device-agnostic boot image per architecture.
The way forward is to support the standards, the [_Arm Base Boot Requirements_](https://developer.arm.com/documentation/den0044/latest). Part of the _Arm BBR_ are _SBBR_ and _EBBR_. And from those, the _only_ pertinent detail here is that they specify usage of _UEFI_.
What does this actually mean _for NixOS_? Not much! We’d be dropping the current SD images, and rely entirely on the existing and relatively well tested UEFI images for ARM.
It sounds like change, but in a way this is reducing complexity as it allows using _the same concepts_ compared to `x86_64`.
With one caveat…
## Bring your own platform firmware
Part of this plan requires re-thinking the way **everyone** thinks about the common ARM platforms. Everyone includes other distributions, and users outside of NixOS. Not really, we can go our own way and do it the right way anyway, but it would be easier if we can rally everyone behind this new way of doing things.
As the title implies, _distributions should not provide platform firmware builds_. Relatedly, device-specific images shouldn’t really be a common thing anymore.
With this in mind, _I already started with the [Tow-Boot](https://tow-boot.org/) project_. This aims to provide opinionated generic U-Boot builds for the express purpose of using them like your old boring laptop’s UEFI.
Supporting _only_ Tow-Boot is **not** a goal. Said images should work with standards and specifications in mind. So other options like the [Raspberry Pi Tianocore EDKII builds](https://rpi4-uefi.dev/) should be as supported.
And this is where a change in philosophy is necessary. **The first step in installing a Linux distribution on ARM will not start at the distribution**. The first step will be, paraphrased, “Prepare your system according to the _platform firmware_ setup instructions”.
Then, at this point, **all ARM boards should share the same installation instructions**. Yes it’s possible. [We can do it today](https://nixos.wiki/wiki/NixOS_on_ARM/UEFI).
## What’s left then?
> Given [Tow-Boot](https://tow-boot.org/) is a thing, and [you can already install with UEFI](https://nixos.wiki/wiki/NixOS_on_ARM/UEFI), what’s left?
First order would be properly surveying users to understand what use cases are missing here.
In parallel, UEFI instructions should be put first as the “happy path” for NixOS on ARM and tested thoroughly to file down rough edges.
Then, the SD images should be removed.
> But I rely on SD image builds in my custom use case
Sorry to hear that. They never were intended to be used through composition, and it’s a real thorn on the side of NixOS on ARM.
But fear not! I purposefully made it sound grim by saying SD images should be removed. The real way forward is rather: **better semantics for image building are required**. And this is not an ARM issue, this is a generic NixOS issue outright. This is **not** a topic for this thread. If you rely on custom image builds, **assume this is handled** at the same time completely externally to this plan.
Additionally, some work should be done to make it easier for users of systems not yet supported by mainline Linux to build correct UEFI iso with the required (presumably vendor) kernel forks. Not really hard or problematic, only needs to be handled. (Yes, it’s mostly as simple as you think it is.)
## Work to be done (elsewhere)
While this is all a good plan already, there is some work I know is required elsewhere.
Documentation from at least one _platform firmware_ provider should be written up and completed so we can start referencing to them. In an ideal world there would even be a “Linux on ARM” generic organization where the starting point would live for such documentation.
Figuring out an upgrade mechanism for the _platform firmware_. Not in NixOS, entirely out of band, provided by the _platform firmware_ provider.
Then, not strictly required, but in my mind it’s always been part of the plan, is to complete enough of the UX work on _Tow-Boot_. Main missing features are a TUI to configure some things, like the boot order, and _device-specific_ options handling.
## Open questions
- Who should manage the device tree used by the operating system?
- (Assuming we need to load dtb files) how can GRUB (or any other EFI bootloader) load the _correct_ dtb file according to the kernel’s own desires?
- Converting from “old scheme” images to “new scheme” images.
* * *
# TLDR
- SD image painful, remove.
- UEFI all the things.
- Conquer ARM.
* * *
# Appendix
## Technical notes
### U-Boot is not a bootloader
**It is way more than a bootloader.**
It is first and foremost the _platform firmware_ for the device. Think of _U-Boot_ not like _GRUB_, but like your _BIOS_ or _UEFI_ in your boring “normal” computer.
The main point of confusion is that _U-Boot_ **also** knows how to act as a bootloader. It can do so through direct boot methods (`bootm`, `booti` and similar), or schemes like _extlinux compatible_ boot. _U-Boot_ **also** can boot UEFI programs. [Read more in `README.distro`.](https://source.denx.de/u-boot/u-boot/-/blob/master/doc/README.distro)
## Glossary
### Platform Firmware
The **platform firmware** is the softwarey-firmwarey program that executes before the bootloader. Some **platform firmware** may also provide _bootloader_ facilities. Example **platform firmware** include _Tianocore EDKII_ and _U-Boot_.
### Bootloader
The **bootloader** is the software part that handles loading the operating system image as needed for the specific target operating system. Generally speaking, common modern bootloaders are implemented as UEFI programs. Some _platform firmware_ include \*_bootloader_ facilities, like U-Boot does.
Example bootloaders include: _GRUB_, _systemd-boot_, _syslinux_ and _rEFInd_.
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 4, 2021, 10:05pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/2 "2021-10-04T22:05:30Z")
# FAQ
Questions from this thread
_None yet_
---
### Author:  [@payas](https://discourse.nixos.org/u/payas)
#### Post date: [October 5, 2021, 5:15am UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/3 "2021-10-05T05:15:29Z")
Thanks for excellent writeup! I haven’t used NixOS on ARM yet, and am bit of a noon, but I have a Raspberry Pi coming up and would love to devote it for testing purposes if need be!
---
### Author:  [@markuskowa](https://discourse.nixos.org/u/markuskowa)
#### Post date: [October 5, 2021, 2:56pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/4 "2021-10-05T14:56:14Z")
I did an installation of NixOS on a RPi3 with the [EDK2 UEFI](https://github.com/pftf/RPi3) implementation. It worked surprisingly well (only minor issues).
---
### Author:  [@paralin](https://discourse.nixos.org/u/paralin)
#### Post date: [October 7, 2021, 10:13pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/5 "2021-10-07T22:13:59Z")
I’m using SkiffOS as the “platform firmware” part with NixOS running within a container. Then the container and nix configs are portable across arm machines and even different architectures.
The only issue at the moment is Nix has limited caching for arm32 builds.
> **[GitHub - skiffos/SkiffOS: Any Linux distribution, anywhere.](https://github.com/skiffos/skiffos)**
>
> Any Linux distribution, anywhere. Contribute to skiffos/SkiffOS development by creating an account on GitHub.
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 7, 2021, 10:40pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/6 "2021-10-07T22:40:32Z")
_SkiffOS_ wouldn’t be defined as a _platform firmware_ here. But fear not, this is not a negative thing!
_SkiffOS_ is more akin to an hypervisor. I guess it would be called a _container host operating system_.
And yeah, the lack of an ARMv7l is annoying, but luckily a distinct issue from what I’m looking to work on and solve here.
Given that I’m looking to solve booting up to the kernel starting, it should change nothing about the container-based experience with _SkiffOS_.
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 7, 2021, 10:42pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/7 "2021-10-07T22:42:05Z")
Nothing in particular. Don’t devote your Raspberry Pi for testing, **use it** , for whatever purposes you want!
I was mainly making the plans public so I could refer to them, rather than explaining every time from fresh :). Also taking the pulse of the community on the topic.
---
### Author:  [@leo60228](https://discourse.nixos.org/u/leo60228)
#### Post date: [October 28, 2021, 3:55pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/8 "2021-10-28T15:55:28Z")
> This installer image should be written to a USB drive, like usual. In a pinch, it may also be written to an SD image, if your target’s platform firmware does not need to be written to that same SD image. This UEFI iso is incompatible (by design) with _shared firmware storage_.
(from [NixOS on ARM/UEFI - NixOS Wiki](https://nixos.wiki/wiki/NixOS_on_ARM/UEFI#Getting_the_installer_image))
Will there be UEFI images that are compatible with shared firmware storage? It’s not a huge deal, but it feels silly to flash two drives when doing a fresh installation.
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 28, 2021, 6:07pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/9 "2021-10-28T18:07:47Z")
I hope we can end up providing the tooling for more enterprising individuals (or organizations) to build their own “final” disk images with their requirements, rather than try to ship a one-size-fits-all image that has some subtle, and some less subtle, issues. But that’s a separate goal from ARM things.
The whole reason two images are flashed with the _shared storage_ strategy is to work around a deficiency in the target device; the “BIOS” (so to speak) of the system is missing and has been put on the Operating System storage of the computer. Flashing the _platform firmware_ to the SD card is an operation that I would compare to “completing the assembly” of the device. Now your device is ready to be used with **any** standards-based boot operating system (like NixOS).
While extremely principled as an approach, I think we’re at a point in time where it is necessary to signal strongly that this is an issue, and this is the only solution for platforms without _dedicated storage_ for the firmware.
The main benefit in the end is that this ends up abstracting all platforms into common steps:
1. Get a standards-based boot platform firmware
2. Install your OS with _whichever_ partitioning scheme you want
Now, maybe I misread your question. Maybe your question was something like: _can the firmware be on the USB drive with the installer iso?_ No, most platforms can’t start the platform firmware from USB, and the iso is built in a way where it’s likely impossible to add them anyway. Furthermore, it would only be “moving cheese”; the arduous device-specific steps of _somehow_ getting the platform firmware for your specific SBC would need to exist still. And yet still, your target platform wouldn’t be bootable without the USB drive, unless you _also_ somehow managed to flash the platform firmware in device-specific ways before installing.
Maybe I still misread your concerns. The issue may be about flashing _two USB drives_ to then install to the SD card (e.g. on a Raspberry Pi). That wouldn’t be the case. You would prepare the SD card for the Raspberry Pi by flashing the _platform firmware_ onto it. This also provides you with a started partition scheme that protects the _platform firmware_. Then you have to flash a secondary media (USB drive) to do the installation from.
Yet still, maybe there’s something I’m missing, if so do ask :).
(I preferred over-answering as other people may have had the questions I additionally answered.)
---
### Author:  [@leo60228](https://discourse.nixos.org/u/leo60228)
#### Post date: [October 28, 2021, 11:12pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/10 "2021-10-28T23:12:53Z")
That’s close to what I was trying to ask, but not exactly. From the wiki page, my understanding is that when installing to SD you can currently have:
- Firmware on dedicated storage, installer on USB drive
- Firmware on dedicated storage, installer on target SD
- Firmware on shared storage, installer on USB drive
Would it be possible in the future to have the firmware on shared storage _and_ installer on the target SD?
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 28, 2021, 11:57pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/11 "2021-10-28T23:57:27Z")
I think I see a problem point. Let me try to clarify things. If that helps that means I need to work on improving the messaging about that.
When talking about _shared storage_ and _dedicated storage_, these are mutually exclusive situations. We are talking about where the _platform firmware_ (“BIOS” if you allow the anachronistic metaphor) of the system lives.
When a device supports _dedicated storage_, the _platform firmware_ will be installed to a location that is dedicated for that _firmware_. It means that even if you removed the SD or eMMC storage, the firmware is still present and able to do some useful work. As an implementation detail, it will be commonly implemented as an “SPI Flash” type of storage.
When a device does not support _dedicated storage_ for the firmware, we’re left to work with a _shared storage strategy_ to work around the issue. The main idea here is that _external projects_ will provide a base image with a partition scheme pre-configured to **protect** the platform firmware, and make it a tangible piece of the puzzle. Only a limited subset of platforms already require a partition (e.g. the Raspberry Pi family of hardware). Every other platforms, it’s more often than not done through _splatting_ the U-Boot build to an arbitrary location, where you have to hope the end-user won’t write data into.
* * *
So, coming back to your concerns, when installing you have two different scenarios:
With a platform that supports installing as _dedicated storage_
1. (Done at least once) install the _platform firmware_ to the board.
2. Install any operating system using standards-based boots (e.g. UEFI boot) to wherever the _platform firmware_ can boot from. Be free to format and partition as you want.
With a platform that does nos support installing as _dedicated storage_
1. Write the _platform firmware_ image to a storage the platform can start from, e.g. SD card for Raspberry Pi; eMMC or SD for many A64 targets.
2. Install any operating system using standards-based boots (e.g. UEFI boot) to either the internal storage, or any other storage that the _platform firmware_ can boot from.
- If installing to the same storage the _platform firmware_ is installed to, be mindful about the existing partition scheme.
- Else you’re free to partition other target storage as you want.
With the first scenario, you need to “finish assembling” the board by installing the firmware where it should have been installed by the factory (imo!). With the second scenario, the end-user has to start from a known-good starting point for the board’s internal storage.
* * *
Finally, to directly answer with concrete examples
With the ever-popular Raspberry Pi platform, you would:
1. Burn an SD card with the _shared storage strategy_ pre-installed firmware.
2. Burn a USB drive with the UEFI NixOS installer
3. When installing, before using `nixos-generate-config`, you partition and format _starting from the existing partitions on the SD card_.
With the Pinebook Pro (or really any RK3399 platforms, and many other boards)
1. (Do at least once) start from a supported storage device (e.g. SD card) and install to the dedicated storage. (This storage device can be erased/reset once this is done.)
2. Burn a USB drive with the UEFI NixOS installer
3. When installing, before using `nixos-generate-config`, you partition and format _starting from scratch_.
There are multiple **strong benefits** here from not using a pre-partitioned-and-formatted SD card:
- Unique identifiers are generated for your install (partition UUID, filesystem UUID)
- Freedom to choose the filesystem(s) to use
- ~~Easier~~ **possible** to do full-disk-encryption
When starting from a pre-baked SD card, you’re stuck with the choices made for you. It becomes harder than necessary to change these choices.
And a final benefit here, is that the installation instructions can become unified across the different platforms. No need for special AArch64 instructions as far as the installation goes! Conversely: your knowledge about installing on x86\_64 applies to AArch64 too.
* * *
Don’t feel bad if I missed anything still. It’s on me to make the messaging as crystal clear as possible, and I’m cursed with having all the knowledge already at the top of my mind :). So just ask more clarification questions if you have some more.
---
### Author:  [@leo60228](https://discourse.nixos.org/u/leo60228)
#### Post date: [October 29, 2021, 1:22pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/12 "2021-10-29T13:22:08Z")
I think the answer is that this isn’t possible, but I also think I’m doing a really bad job of asking what I want. What I’d like is this installation process for systems that only support shared storage:
1. Flash firmware to boot SD (somehow)
2. Flash installer to boot SD (somehow, possibly fused with the previous step)
3. Boot installer from SD to RAM
4. Delete installer partitions from SD, leaving only firmware
5. Create new target partition(s)
6. Install targeting there
---
### Author:  [@leo60228](https://discourse.nixos.org/u/leo60228)
#### Post date: [October 29, 2021, 1:29pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/13 "2021-10-29T13:29:11Z")
Specifically, my thoughts are that while the process for systems that already have firmware on dedicated storage is the same as on x86\_64, we don’t live in a world where that’s the case for most ARM systems. It isn’t a _huge_ amount of friction, but it’d be great if a standards-based boot process didn’t involve more work when installing compared to distributions like Raspbian.
I’d also note that I strongly agree with your approach in general, and it’s how I’ve had my Pine A64+ (currently running Debian, I plan to move it to NixOS when 21.11 releases) set up for a while now. I also think that the method I’m describing _should_ work for Debian, but it didn’t work in practice for reasons I wasn’t able to figure out (a naive installation would fail with a confusing error message, I eventually found a workaround but the installer tried and failed to replace U-Boot with one from the Debian repositories which I couldn’t find a way to bypass).
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 29, 2021, 4:44pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/14 "2021-10-29T16:44:15Z")
> [@leo60228](#):
>
> [Installing from a booted-to-RAM system on AArch64]
This _could_ be possible… Assuming there’s enough RAM. It would require a different installer build I think. Same story as on x86\_64, the installer is built in a special way that would break when not written at the start of a storage media.
The only reason one would need to do this (imo) if it they don’t have an additional valid boot medium handy.
> [@leo60228](#):
>
> Specifically, my thoughts are that while the process for systems that already have firmware on dedicated storage is the same as on x86\_64, we don’t live in a world where that’s the case for most ARM systems.
And it is a shame. Speaking of shame, there is a not-so-hidden agenda here to shame ARM platforms where EBBR can’t be followed correctly due to the lack of a _dedicated storage_ area for the _platform firmware_. These lacking platforms will get more cumbersome to install to, in a way that I hope will provide incentives for board makers to build “finished” products, especially if other distros follow suit.
> [@leo60228](#):
>
> It isn’t a _huge_ amount of friction, but it’d be great if a standards-based boot process didn’t involve more work when installing compared to distributions like Raspbian.
I personally think the “less” work Raspbian-like distros is an illusion. It seems all of these distros have long post-setup steps to follow _anyway_ for a lot of use cases.
Furthermore, the actual real reason they’re bad is it promotes an non-sustainable workflow where distributions need to build and provide images for every single boards. We’re already not doing this for non-Raspberry Pi systems by providing a unique image, and leaving final assembly to the end-user. In many ways, we’re mostly shuffling existing NixOS installation steps around.
> [@leo60228](#):
>
> I’d also note that I strongly agree with your approach in general, and it’s how I’ve had my Pine A64+ (currently running Debian, I plan to move it to NixOS when 21.11 releases) set up for a while now. I also think that the method I’m describing _should_ work for Debian, but it didn’t work in practice for reasons I wasn’t able to figure out (a naive installation would fail with a confusing error message, I eventually found a workaround but the installer tried and failed to replace U-Boot with one from the Debian repositories which I couldn’t find a way to bypass).
Yes, such standards-based boot methods **should** work with other distros, assuming a UEFI install. Though a lot of distros have assumptions that don’t hold with the limited UEFI support U-Boot provides. Mainly Ubuntu and Debian, for _different reasons actually_, both fail hard the installation since UEFI vars can’t be changed to add the UEFI boot entry for the bootloader.
When working on Tow-Boot, I tested other distros’ UEFI installers on ARM too, since it’s not meant to be a NixOS project. It is a project for all distros and all boards.
Assuming non-UEFI installs, this is the flip side of the coin: **distributions shouldn’t manage the _platform firmware_**. The distributions themselves shouldn’t, but other tools like `fwupd` could, as long as it’s done just like it would be for an x86\_64 UEFI firmware. This goes in the same direction about things becoming unmaintainable: every distributions would otherwise have to provide their own updater scheme, provided for every boards.
* * *
With that said…
I hope to figure out a good strategy for enterprising individuals that want to build their own pre-configured disk images as a whole “product” output. So with Nix it may be possible to have your cake and eat it too. Provide UEFI installers for standards-based boot and “classic” installation schemes for **“pet”-like** management of an SBC. Provide the tooling necessary for **“cattle”-like** deployment to SBCs (of standards-based boot still).
---
### Author:  [@leo60228](https://discourse.nixos.org/u/leo60228)
#### Post date: [October 29, 2021, 4:57pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/15 "2021-10-29T16:57:24Z")
Thanks, that answers all my questions.
> [@samueldr](#):
>
> This _could_ be possible… Assuming there’s enough RAM. It would require a different installer build I think. Same story as on x86\_64, the installer is built in a special way that would break when not written at the start of a storage media.
The cost-benefit ratio here is very questionable, but the RAM usage could be heavily reduced by network mounting `/nix/.ro-store`. I don’t think this is worthwhile, though.
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [October 29, 2021, 5:13pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/16 "2021-10-29T17:13:50Z")
> [@leo60228](#):
>
> The cost-benefit ratio here is very questionable, but the RAM usage could be heavily reduced by network mounting `/nix/.ro-store` . I don’t think this is worthwhile, though.
_And_ that would quickly increase complexity!
---
### Author:  [@xaverdh](https://discourse.nixos.org/u/xaverdh)
#### Post date: [November 1, 2021, 11:38am UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/17 "2021-11-01T11:38:09Z")
How will this interact with android based devices?
---
### Author:  [@samueldr](https://discourse.nixos.org/u/samueldr)
#### Post date: [November 1, 2021, 5:48pm UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/18 "2021-11-01T17:48:18Z")
It will not. Even other mobile devices that could follow this scheme don’t have to.
This plan is squarely for platforms that already or can be made to follow standards-based boots.
Mobile devices are varied, complex enough, and boot in annoying ways. Mobile NixOS continues providing the necessary abstractions up to and including the kernel boot.
---
### Author:  [@NoahAndrews](https://discourse.nixos.org/u/NoahAndrews)
#### Post date: [January 12, 2022, 4:42am UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/19 "2022-01-12T04:42:35Z")
I did not realize generic images were possible for ARM! That is massively exciting, thank you for working to (hopefully) bring about a future where that’s the typical way things are done.
---
### Author:  [@Nebucatnetzer](https://discourse.nixos.org/u/Nebucatnetzer)
#### Post date: [March 17, 2022, 9:12am UTC](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346/20 "2022-03-17T09:12:19Z")
This plan is absolutely fantastic IMO.
I tried this week to get started with a Pi 4 and was successful but it annoyed me that I couldn’t use my standard setup and had some problems [with Flakes](https://discourse.nixos.org/t/how-to-install-a-flake-onto-a-raspberry-pi/18200).
I’m going to test if the UEFI approach works better, at least I’m familiar with it 🙂 .
The firmware part could be quite easy no?
Flash an image with a small partition and that’s that? Looks like that’s exactly what you have to do with Tow-Boot💪.
The cattle part sounds interesting as well but I haven’t tried yet to build an image myself.
[Next page](https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm-and-other-non-x86-64-systems/15346.md?page=2)