Hi, a relatively new NixOS user here. I’ve been a Linux user for years in the x86_64 & non-Nix world, but in January I got myself an ARM single board computer to take over the server tasks of my old PC home server. I’ve learned a lot, and thought it might be worthwile to share some of the things I’ve gone through, and how things are now a lot better for the SBC I got, Hardkernel’s ODROID-M1.
The most important details and code snippets are in a tl;dr at the end, if the story part isn’t interesting to you.
The story part
ODROID-M1 seemed appealing to me because it has an NVMe and a SATA port on it and can boot from them too. It uses a bootloader called Petitboot by default, which means that the board’s firmware basically contains a tiny Linux distribution which in turn boots the actual OS. In the beginning of this year, all the necessary drivers for the hardware existed already, but the device tree information (which is needed for the kernel to understand what hardware the board has) wasn’t in any officially released kernel version yet, so I ended up compiling a kernel by instructions from the forums of a distribution I was originally trying out.
Luckily for me, the firmware included an utility to start installer for a couple Linux distributions. The installers and the installations they produce are customized by Hardkernel. I started out by getting myself a working Debian installation that way, but didn’t intend to stay on it. It provided a proper Linux environment on the board itself, though, so I didn’t have to rely on cross-compilation.
Petitboot has its downsides, but a neat part is that only a simple plain text file is needed to boot into Linux from it. It supports (at least based on the GitHub repo - the Hardkernel version seems to be modified) several configuration file formats, including GRUB’s file format. That doesn’t mean it includes GRUB itself, it just has a parser that can read basic GRUB file syntax to obtain the location of kernel, initrd and device tree files. This means getting a distribution to boot was actually relatively easy, if it just otherwise had all the needed things enabled. As a beginner in the ARM world, I don’t know if I would have been successful in getting something like U-Boot to work by myself.
After trying out other distributions, I ended up trying NixOS, at first by using an SD card image, but it didn’t have the required drivers for SATA and NVMe drives to work. The device tree file I had from the other distro attempts. Installing Nix on the Debian installation, and then creating a NixOS installation on an NVMe drive worked well. It took a while to figure out the needed extra kernel compilation options, but I ultimately got myself a working installation.
Since then, things have become slowly more “out of the box”. When kernel 6.2 was released, I didn’t have to hang on to the old device tree file anymore. For a long time, I was considering making an issue in the nixpkgs issue tracker on GitHub to request that the needed drivers would be enabled by the distribution, so I wouldn’t have to recompile the kernel myself. (Compilation takes something like eight hours on the board itself!) However, there was an old pull request to enable them waiting on a kernel mailing list, so I thought I’d just wait and see if they would be enabled in the official kernel default config. And in kernel 6.6 they were, so my long wait is now finally over and I can just use linux_latest.
(Sorry, the story part ended up being way longer I had intended, the reference in the title to patience wasn’t for that!)
Summary and details
It seems like the board is for the kernel’s part now working by default on NixOS. I belive the SD card image with latest kernel should now work (see NixOS on ARM page in the wiki), so no bootstrapping through other distributions should not be needed to install NixOS on ODROID-M1 anymore. (I haven’t tested this myself, though.) The NixOS installation on my board is now running with linux_latest, so on that part I can confirm that no kernel customizations are needed anymore.
Booting into NixOS is the remaining “tricky” part. As I told above, Petitboot supports some of GRUB’s configuration syntax. However, the default GRUB config generated by NixOS is too complicated for Petitboot’s parser so it’s not usable by itself. Here are my board-specific configuration options:
# Use kernel >6.6
boot.kernelPackages = pkgs.linuxPackages_latest;
# I'm not completely sure if some of these could be omitted,
# but want to make sure disk access works
boot.initrd.availableKernelModules = [
"nvme"
"nvme-core"
"phy-rockchip-naneng-combphy"
"phy-rockchip-snps-pcie3"
];
# Petitboot uses this port and baud rate on the boards serial port,
# it's probably good to keep the options same for the running
# kernel for serial console access to work well
boot.kernelParams = [ "console=ttyS2,1500000" ];
# Enable GRUB 2, even though we don't really use it
boot.loader.grub.enable = true;
# Disable installing GRUB to the start of the disk
boot.loader.grub.device = "nodev";
# This script extracts the necessary lines from
# the NixOS-generated GRUB config into one that Petitboot accepts
boot.loader.grub.extraInstallCommands = ''
grep -E '(menuentry|initrd|linux)' "/boot/grub/grub.cfg"|
sed 's#($drive1)/##'|
sed -re 's#-initrd$#-initrd\n devicetree /rk3568-odroid-m1.dtb\n}#' > "/boot/grub.cfg"
cp "${config.boot.kernelPackages.kernel.outPath}/dtbs/rockchip/rk3568-odroid-m1.dtb" /boot
'';
As you can see, the Petitboot’s boot menu entries are extracted from the “real” GRUB config, which I personally consider fragile (what if the NixOS generated file gets formatted differently?), but it has worked so far. It would probably be better to generate the entries based on boot.json files. The device tree file is also always just copied to same place in the root of boot partition. A cleaner version would have a separate .dtb for each kernel version.
Edit: Just as I had sent the post, I realized that the SD card installation probably doesn’t work, if the initrd options I’ve used on my installation are necessary.
I found some time today and got it working