Newbie installs NixOS on an ARM SBC, or how patience is a virtue

Hi, a relatively new NixOS user here. I’ve been a Linux user for years in the x86_64 & non-Nix world, but in January I got myself an ARM single board computer to take over the server tasks of my old PC home server. I’ve learned a lot, and thought it might be worthwile to share some of the things I’ve gone through, and how things are now a lot better for the SBC I got, Hardkernel’s ODROID-M1.

The most important details and code snippets are in a tl;dr at the end, if the story part isn’t interesting to you.

The story part

ODROID-M1 seemed appealing to me because it has an NVMe and a SATA port on it and can boot from them too. It uses a bootloader called Petitboot by default, which means that the board’s firmware basically contains a tiny Linux distribution which in turn boots the actual OS. In the beginning of this year, all the necessary drivers for the hardware existed already, but the device tree information (which is needed for the kernel to understand what hardware the board has) wasn’t in any officially released kernel version yet, so I ended up compiling a kernel by instructions from the forums of a distribution I was originally trying out.

Luckily for me, the firmware included an utility to start installer for a couple Linux distributions. The installers and the installations they produce are customized by Hardkernel. I started out by getting myself a working Debian installation that way, but didn’t intend to stay on it. It provided a proper Linux environment on the board itself, though, so I didn’t have to rely on cross-compilation.

Petitboot has its downsides, but a neat part is that only a simple plain text file is needed to boot into Linux from it. It supports (at least based on the GitHub repo - the Hardkernel version seems to be modified) several configuration file formats, including GRUB’s file format. That doesn’t mean it includes GRUB itself, it just has a parser that can read basic GRUB file syntax to obtain the location of kernel, initrd and device tree files. This means getting a distribution to boot was actually relatively easy, if it just otherwise had all the needed things enabled. As a beginner in the ARM world, I don’t know if I would have been successful in getting something like U-Boot to work by myself.

After trying out other distributions, I ended up trying NixOS, at first by using an SD card image, but it didn’t have the required drivers for SATA and NVMe drives to work. The device tree file I had from the other distro attempts. Installing Nix on the Debian installation, and then creating a NixOS installation on an NVMe drive worked well. It took a while to figure out the needed extra kernel compilation options, but I ultimately got myself a working installation.

Since then, things have become slowly more “out of the box”. When kernel 6.2 was released, I didn’t have to hang on to the old device tree file anymore. For a long time, I was considering making an issue in the nixpkgs issue tracker on GitHub to request that the needed drivers would be enabled by the distribution, so I wouldn’t have to recompile the kernel myself. (Compilation takes something like eight hours on the board itself!) However, there was an old pull request to enable them waiting on a kernel mailing list, so I thought I’d just wait and see if they would be enabled in the official kernel default config. And in kernel 6.6 they were, so my long wait is now finally over and I can just use linux_latest.

(Sorry, the story part ended up being way longer I had intended, the reference in the title to patience wasn’t for that!)

Summary and details

It seems like the board is for the kernel’s part now working by default on NixOS. I belive the SD card image with latest kernel should now work (see NixOS on ARM page in the wiki), so no bootstrapping through other distributions should not be needed to install NixOS on ODROID-M1 anymore. (I haven’t tested this myself, though.) The NixOS installation on my board is now running with linux_latest, so on that part I can confirm that no kernel customizations are needed anymore.

Booting into NixOS is the remaining “tricky” part. As I told above, Petitboot supports some of GRUB’s configuration syntax. However, the default GRUB config generated by NixOS is too complicated for Petitboot’s parser so it’s not usable by itself. Here are my board-specific configuration options:

# Use kernel >6.6
boot.kernelPackages = pkgs.linuxPackages_latest;

# I'm not completely sure if some of these could be omitted,
# but want to make sure disk access works
boot.initrd.availableKernelModules = [
# Petitboot uses this port and baud rate on the boards serial port,
# it's probably good to keep the options same for the running
# kernel for serial console access to work well
boot.kernelParams = [ "console=ttyS2,1500000" ];

# Enable GRUB 2, even though we don't really use it
boot.loader.grub.enable = true;
# Disable installing GRUB to the start of the disk
boot.loader.grub.device = "nodev";

# This script extracts the necessary lines from
# the NixOS-generated GRUB config into one that Petitboot accepts
boot.loader.grub.extraInstallCommands = ''
grep -E '(menuentry|initrd|linux)' "/boot/grub/grub.cfg"|
  sed 's#($drive1)/##'|
  sed -re 's#-initrd$#-initrd\n  devicetree /rk3568-odroid-m1.dtb\n}#' > "/boot/grub.cfg"
  cp "${config.boot.kernelPackages.kernel.outPath}/dtbs/rockchip/rk3568-odroid-m1.dtb" /boot

As you can see, the Petitboot’s boot menu entries are extracted from the “real” GRUB config, which I personally consider fragile (what if the NixOS generated file gets formatted differently?), but it has worked so far. It would probably be better to generate the entries based on boot.json files. The device tree file is also always just copied to same place in the root of boot partition. A cleaner version would have a separate .dtb for each kernel version.

Edit: Just as I had sent the post, I realized that the SD card installation probably doesn’t work, if the initrd options I’ve used on my installation are necessary.



Why is this, is petiboot memory constrained or something?

just curios…

Nice work!

I can only guess, but GRUB 2 is surprisingly advanced system based on its manual and all the commands listed there. Reimplementing support for all of that, or even to a significant degree, can’t be easy. Petitboot has support for some other foreign bootloader config formats too, based on a quick look at its source code on Github (there are files for Yaboot and Syslinux along the GRUB parser code), so maybe it’s more about offering familiarity to users of other bootloaders instead of targeting full compatibility?

1 Like

grub has to support of legacy stuff and probably has a lot cruft in it from bios and systems from 1992 ;-).

Have you seen this ? Might be of interest to you.

I haven’t, thanks for the link! I should look into that kboot.conf format, in case it’s nicer to work with than the (restricted) GRUB syntax, and see what’s there to learn about the script otherwise.

@hellabyte any chance of documenting te steps of how you got these up an running? There’s a thread of people on the odroid forums who are looking for how to get NixOS onto the M1.


Nevermind :slight_smile: I found some time today and got it working

Nice! I really should take the time to look into using that kboot conf myself too, it seems more robust than my grep+sed GRUB config munging.

I finally did some testing myself today before coming back to this thread, so I’ll add my notes too in case they’re helpful for someone in the future. I tried out a generic SD card image, and turns out only adding the boot configuration is needed now. So the steps to get started are roughly:

  1. Obtain an SD card image from the NixOS on ARM wiki page. The unstable (latest kernel) is the correct one at the moment. (At some point in future it should matter less, when all kernels are 6.6 or later.) Write it to SD card.
  2. Mount the second partition on the card. It’s the one with boot, nix, etc and other system directories. The first partition has files relevant to other boards, and isn’t useful to us.
  3. Note the contents of the file boot/extlinux/extlinux.conf.
  4. Create a file with name grub.cfg to the root of the partition, meaning next to the directories I mentioned above. Other paths are possible too, according to this. Write following contents to the file:
menuentry "NixOS SD" {
  linux /boot/nixos/ivffbmvkp0sf406sg92k94aalxizysmj-linux-6.6.1-Image init=/nix/store/p5x1a07dxji4af7habxzha5br1291336-nixos-system-nixos-23.11pre549786.c757e9bd77b1/init console=ttyS2,1500000 loglevel=7
  initrd /boot/nixos/dqcdaxgxclp59yqa7yc95cy7nv6p5svq-initrd-linux-6.6.1-initrd
  devicetree /boot/nixos/ivffbmvkp0sf406sg92k94aalxizysmj-linux-6.6.1-dtbs/rockchip/rk3568-odroid-m1.dtb

…well, not actually exactly those contents. You need to use the paths from the extlinux.conf mentioned earlier, they vary over time as new versions come. Also the extlinux.conf is organized a bit differently, here we have the contents of the LINUX and APPEND lines combined into a single linux line. Note that I’ve changed the console argument from the one in extlinux.conf, so that serial access works properly.

  1. Unmount the SD card, put it into the ODROID M1 and turn the thing on. The boot menu should have the NixOS SD entry on it. Boot into it. If you’re using an HDMI display, you won’t be able to see the boot messages, because the HDMI/video drivers aren’t loaded yet (the’re missing from the initrd), but after a moment the command line should show up. I didn’t test it today, but the serial console should show the full boot message properly.
  2. Continue however you want from here. The storage drives should be available (at least the NVMe drive was visible in my case), so you don’t need to stick on the SD card.

Additionally when making a new installation, Petitboot supports ext2, ext3 and ext4 and a couple other filesystems like vfat and iso9660, so /boot should be on one of those. (To see it for yourself, use the exit to console option in Petitboot, and do cat /proc/filesystems.) I use btrfs, so I needed a separate /boot partition, but I suppose if your installation / is on ext4, a single system partition should be enough. It might even be possible to configure the entries so that they point directly into files in /nix?