NixOS on Rock64Pro

Boondocks8194 · August 12, 2024, 5:46pm

Hi folks,

I’m using a Rock64Pro from Pine64 as my offsite backup device with some HDDs attached.
To get NixOS working on the device I followed all instructions from the Pine64-Wiki and used the NixOS build from here. First boot from the MicroSD card worked and after that I generated a configuration file and rebuilt the system again with some config changes, no errors so far. The new built is instantly running. But after rebooting the device, it booted up into an old built. While booting I have no possibility to select the newest built. It always boots up like this:

.
As you can see, it’s obviously not the newest NixOS-Version and built. After running a nixos-rebuilt switch --upgrade again, it upgrades everything to the newest channel and also config. Running neofetch again, it now displays this:

Rock64Pro uses uboot as boot manager, so I looked up all boot entries in /boot/extlinux/extlinux.conf:

# Generated file, all changes will be lost on nixos-rebuild!

# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 50

LABEL nixos-default
MENU LABEL NixOS - Default
LINUX …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-Image
INITRD …/nixos/hqrr20g1vg40bw1f4scwfc2pkpp7h7jg-initrd-linux-6.6.43-initrd
APPEND init=/nix/store/888xmyq6pqizx758111zs487wkc4jgdw-nixos-system-backy-24.05.3523.a633d89c6dc9/init loglevel=4
FDTDIR …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-dtbs

LABEL nixos-5-default
MENU LABEL NixOS - Configuration 5-default (2024-08-05 10:30 - 24.05.3523.a633d89c6dc9)
LINUX …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-Image
INITRD …/nixos/hqrr20g1vg40bw1f4scwfc2pkpp7h7jg-initrd-linux-6.6.43-initrd
APPEND init=/nix/store/888xmyq6pqizx758111zs487wkc4jgdw-nixos-system-backy-24.05.3523.a633d89c6dc9/init loglevel=4
FDTDIR …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-dtbs

LABEL nixos-4-default
MENU LABEL NixOS - Configuration 4-default (2024-07-31 20:52 - 24.05.3407.12bf09802d77)
LINUX …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-Image
INITRD …/nixos/c86rrzbk7mqsadvc6533jh2vv9ks8gav-initrd-linux-6.6.43-initrd
APPEND init=/nix/store/ay5fpvmbd733l8ikrbj0qvm1gp6vr8nx-nixos-system-nixos-24.05.3407.12bf09802d77/init loglevel=4
FDTDIR …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-dtbs

LABEL nixos-3-default
MENU LABEL NixOS - Configuration 3-default (2024-07-31 20:44 - 24.05.3407.12bf09802d77)
LINUX …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-Image
INITRD …/nixos/c86rrzbk7mqsadvc6533jh2vv9ks8gav-initrd-linux-6.6.43-initrd
APPEND init=/nix/store/dyd4g0wx4c0l89z63ihk748g609sznb4-nixos-system-nixos-24.05.3407.12bf09802d77/init loglevel=4
FDTDIR …/nixos/gq6wvbgp73fgk6r92ma89csq7fdm85yb-linux-6.6.43-dtbs

LABEL nixos-2-default
MENU LABEL NixOS - Configuration 2-default (2024-07-31 20:31 - 23.05.5533.70bdadeb94ff)
LINUX …/nixos/9b3yq6x6h5nx5v35g631yfg3fv73flz3-linux-6.1.69-Image
INITRD …/nixos/lr7rkdyi92adv4r49pzqg251g0br6shj-initrd-linux-6.1.69-initrd
APPEND init=/nix/store/p74g18fx8dy8mp3nkxnmqs6h15m7z776-nixos-system-nixos-23.05.5533.70bdadeb94ff/init loglevel=4
FDTDIR …/nixos/9b3yq6x6h5nx5v35g631yfg3fv73flz3-linux-6.1.69-dtbs

LABEL nixos-1-default
MENU LABEL NixOS - Configuration 1-default (1970-01-01 00:00 - 23.05.3427.e5f018cf150e)
LINUX …/nixos/l4dqp8306smayy34ab6wcc8ilvm0gba2-linux-6.1.52-Image
INITRD …/nixos/44hj5qq79q2v24776wsy7jjlqjy1macn-initrd-linux-6.1.52-initrd
APPEND init=/nix/store/7wvkshfylm6cilj3jxkdaycarrbbd22h-nixos-system-nixos-23.05.3427.e5f018cf150e/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 nohibernate loglevel=7
FDTDIR …/nixos/l4dqp8306smayy34ab6wcc8ilvm0gba2-linux-6.1.52-dtbs

As you can see, all newer builts are properly listed in this boot configuration, even the default is correct.
But after rebooting the device it always starts into nixos-2-default.

is there anything I need to know about uboot, maybe running a specific command I forgot to refresh the default boot entry?

Thanks in advance!

mightyiam · August 16, 2024, 4:24am

Did you figure this out?

Boondocks8194 · August 17, 2024, 12:20pm

Not yet…
Stell waiting for an answer…

mightyiam · August 18, 2024, 12:38pm

Does uboot have a feature of remembering a default?

Boondocks8194 · August 20, 2024, 4:54pm

Sadly I have no idea, there are no special options set in my config, just the basics with

nixos-generate-config

edwinistrator · August 20, 2024, 7:43pm

I can’t really help you with the specific issue of booting into the wrong generation. But I’ve been running NixOS on my RockPro64 for while now.

Instead of using u-boot as a bootloader directly, I’ve flashed Tow-Boot - “an opinionated distribution of U-Boot” - which gives a you a nice interface and UEFI working out of the box.
Then I use systemd-boot as the UEFI bootloader, like I do on my other x86 and ARM systems. The installation is exactly the same as on x86: let NixOS install it on the ESP and u-boot will find it. So u-boot really just serves as a UEFI firmware here.
systemd-boot also gives you a nixos generation selector (it should work over a serial cable; while HDMI output works IIRC, using a USB keyboard has been flaky).

I’ve put the ESP (EFI system partition) on an SD card. Having the ESP on the disk itself might also work, I only have it on the SD card because I previously had the rootfs on a ZFS RAID. NixOS itself is installed on an SSD connected via a PCIE SATA controller.

Other than that, the only bit of hardware-specific configuration I’ve got in my NixOS configuration are a few kernel modules for the initrd, a few kernel paramters, and the CPU performance governor:

{...}: {
  boot = {
    initrd = {
      availableKernelModules = [
        "ahci"
        "pcie_rockchip_host"
        "phy_rockchip_pcie"
        "rockchip_dfi"
        "rockchip_thermal"
        "rtc_rk808"
        "rockchip_saradc"
        "uas"
        "fusb302"
      ];
    };
    kernelParams = [
      "console=ttyS2,115200"
      "nohibernate"
    ];
  };
  powerManagement.cpuFreqGovernor = "ondemand";
}

(I think the uas module was from when I had some storage attached to USB. And the nohibernate option is a remnant from when it was a ZFS NAS. But now it’s a router, so it shouldn’t hibernate anyway).

I can really recommend Tow-Boot. I use it on my two ODROID-N2s as well. It’s not perfect, it has bugs; but having a working UEFI environment and being able to use the same bootloader like on x86 just makes things easier. Also it’s built with Nix, so you don’t need to figure out how to built u-boot and inject weird vendor blobs at the right magic offsets.

Boondocks8194 · October 8, 2024, 6:21pm

Hey @edwinistrator, sorry for the late answer, I was pretty busy.

Thank you for your comment! Could you give me some more information about installing tow-boot and NixOS onto Rock64Pro? Mine is running with an SD-card only.

Thanks in advance!

edwinistrator · October 9, 2024, 10:57am

Hi @Boondocks8194, no worries. We all have other things to do.

It has been a while since I flashed Tow-Boot, so I hope I’m not forgetting anything here. But this is rougly how it should work:

Clone the Tow-Boot repo, checkout the released branch to get the latest stable release
Build Tow-Boot via nix-build -A pine64-rockpro64 (the repo doesn’t have a flake). The build should be reasonably fast, as it is automatically using cross-compilation.
Flash spi.installer.img to an SD card
Boot RockPro64 from that SD card and follow the instructions on the serial console. I don’t remember the details here, but it should be straightforward. It might even support HDMI output and USB keyboard input, but if IIRC keyboard input was flaky.

After this, the RockPro64 should automatically boot from the SPI flash. If it doesn’t work, you might have to jumper two GPIO pins to force it to boot from the SD card again, see the Pine64 wiki.

Boondocks8194 · October 27, 2024, 5:51pm

Hi @edwinistrator,

Thanks for your answer!
I tried flashing Tow-Boot to SPI, but it doesn’t do anything. No matter if I use the self-compiled image or the pre-compiled one, in both cases the board doesn’t give me any answer via serial console…
I’M running out of ideas…

edwinistrator · October 28, 2024, 10:32am

Maybe you’re using the wrong baud rate on the serial console? Tow-Boot uses 115200 for all devices, while upstream U-Boot uses 1500000 for the RockPro64.

Boondocks8194 · October 30, 2024, 9:06am

No, serial connection is working just fine, I tested it with my “old” NixOS build and I can see everything while booting. I was able to detect the error I described at the beginning. It tried to boot into the latest configuration but then fails because the image has a collission with another image or something like that. Than it tries the next config, which fails too and so on. Only the oldest config works and it boots up into the old config as in my screenshot above.
I searched the issue and found out, that this is a bug with u-boot and I should install tow-boot instead.
So here we are…

zimward · October 31, 2024, 1:34pm

Could you share the exact error message? To me it sounds simmilar to an issue i had with getting the PinePhone Pro to boot when the initrd was bigger than 1GiB, wich is either a u-boot limit or a RK33xx limitation IIRC.

Boondocks8194 · October 31, 2024, 2:42pm

@zimward there you go:

github.com/NixOS/nixpkgs

RD image overlaps OS image

opened 05:47PM - 13 Mar 24 UTC

closed 02:56PM - 21 Oct 24 UTC

Martin1887

0.kind: bug

### Describe the bug In a Pinebook Pro using the SD image instead of nixos-inst…all and TowBoot, as recommended in the documentation, modifying the kernel or the kernel parameters produces a generation that fails to boot with the obscure message 'RD image overlaps OS image'. ### Steps To Reproduce 1. Create a new configuration that modifies the `boot.kernelPackages` attribute. 2. `sudo nixos-rebuild switch`. 3. `sudo reboot`. 4. Select the last generation in the boot entry. 5. The last generation cannot be booted, with the message `RD image overlays OS image`, and the previous generation is booted instead. ### Expected behavior The last generation is booted after selecting it after the reboot. ### Screenshots N/A. ### Additional context I think the problem is modifying the kernel or kernel parameters in the SD image, but I'm not sure. The problem could also be specific of Pinebook Pro, but I think it is a problem of using a SD image instead of nixos-install. This problem should be mentioned in the documentation if not solved, and probably recommending using nixos-install with detailed instructions instead of using the SD image for a serious usage. --- Add a :+1: [reaction] to [issues you find important]. [reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/ [issues you find important]: https://github.com/NixOS/nixpkgs/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc