Can't boot into new configuration after nix-rebuild, raspberry pi3 B

I have been following this guide to create a headless sd image for my raspberry pi 3 B. It works great and I can create a bootable nixos on my pi3 and I can ssh into it perfectly fine.

For context:
  1. The method I’m using to create the sd image is with https://github.com/nix-community/nixos-generators and with this patch: nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs by misuzu · Pull Request #82718 · NixOS/nixpkgs · GitHub. Command: nixos-generate -f sd-aarch64-installer --system aarch64-linux -c ./pi3-sd.nix -I nixpkgs=./nixpkgs --cores 8
  2. I’m using pkgs.linuxPackages_latest since pkgs.linuxPackages_rpi3 doesn’t work for me. Possibly related with this: Raspberry Pi 3 B+ doesn't boot on kernel 5.4.23 · Issue #82455 · NixOS/nixpkgs · GitHub
  3. nixpkgs is at release-20.03 brach
Steps I take to install a new nixos configuration:
  1. mkdir /boot/firmware && mount /dev/disk/by-label/FIRMWARE /boot/firmware

  2. I scp over the machine specific /etc/nixos/configuration.nix and I run nixos-rebuild boot (switch sometimes doesn’t work but that is for another post). The machine configuration.nix has the following:

/etc/nixos/configuration.nix
  1. Reboot
{ config, pkgs, lib, ... }:
{
  imports = [
    #/etc/nixos/hardware-configuration.nix # not importing this for now, see config.fileSystems 
    /etc/nixos/common.nix # contains ssh, wireless configs, and pkgs
  ];

  hardware = {
    enableRedistributableFirmware = true;
    firmware = [ pkgs.wireless-regdb ];
  };

  # Defining fileSystems here allow sme to not have to run nixos-generate-config
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-uuid/44444444-4444-4444-8888-888888888888";
      fsType = "ext4";
    };
    "/boo/firmware" = {
      device = "/dev/disk/by-uuid/2178-694E";
      fsType = "vfat";
    };
  };

  boot = {
    #kernelPackages = pkgs.linuxPackages_rpi3;
    kernelPackages = pkgs.linuxPackages_latest;
    kernelParams = ["cma=32M"];
    loader = {
      grub.enable = false;
      systemd-boot.enable = true;
      raspberryPi.firmwareConfig = ''
        boot_delay=1
      '';
      efi.canTouchEfiVariables = true;
      timeout = 1;
    };
    extraModprobeConfig = ''
      options cf680211 ieee80211_regdom="GB"
    '';
  };

  networking = {
    enableIPv6 = false;
    hostName = "robert";
    defaultGateway = {
      address = "192.168.1.254";
      interface = "wlan0";
    };
}
The problem:

The new configuration.nix isn’t loaded at boot, the old configuration persists. I think my problem is similar to this one: ZFS system: non-ZFS filesystems not mounting · Issue #85875 · NixOS/nixpkgs · GitHub. The diference here being that I there exists the firmware partition because the sd image has the boot dir in the root partition, but before running nixos-rebuild boot I make sure I mount the firmware paritition in step 1. Am I missing something here?

After step 2 nixos-rebuild boot:

[root@nixos:~]# sudo nix-env --list-generations --profile /nix/var/nix/profiles/system
   1   1970-01-01 00:00:13
   2   2020-06-23 14:45:46   (current)

After reboot I checked the contents of /run/current-system/configuration.nix and it is the same config as the one used to create the sd image, so to me it seems that I haven’t booted into the new configuration (I have system.copySystemConfiguration = true; in both the machine configuration.nix and the sd-image.nix).

Possible solutions and what I need help with

I checked the contents of the raspbian os image and saw that /boot contains all of the firmware files, in contrast to the sd-image.nix which has all the firmware files in /boot/firmware. So what I would like to is overwrite the configuration.fileSystems in nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix

  config = {
    fileSystems = {
      "/boot/firmware" = {
        device = "/dev/disk/by-label/FIRMWARE";
        fsType = "vfat";
        # Alternatively, this could be removed from the configuration.
        # The filesystem is not needed at runtime, it could be treated
        # as an opaque blob instead of a discrete FAT32 filesystem.
        options = [ "nofail" "noauto" ];
      };
      "/" = {
        device = "/dev/disk/by-label/NIXOS_SD";
        fsType = "ext4";
      };
    };
  ....
  };

from https://github.com/NixOS/nixpkgs/blob/39da4240609ee0d8ea533f142ae4c7e25df95980/nixos/modules/installer/cd-dvd/sd-image.nix#L114-#L123

And use this instead in my sd-image.nix:

    fileSystems = {
      "/boot" = {
        device = "/dev/disk/by-label/FIRMWARE";
        fsType = "vfat";
      };
      "/" = {
        device = "/dev/disk/by-label/NIXOS_SD";
        fsType = "ext4";
      };
    };

That way all the firmware files are written to /boot instead of /boot/firmware. I just don’t know how to overwrite attributes yet: The option fileSystems./boot.’ has conflicting definitions…. Maybe this is completely the wrong way to approach this problem because what I'm trying to do is not part f the attributes of sdImage`.

Thanks for reading!

Edit: I didn’t know that discourse would comment on all the github issues. Sorry for the spam!

I changed the config.fileSystems directly in nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix in the local copy I have of nixpkgs and that didn’t work… for understandable reasons.

This is what I changed:

diff --git a/nixos/modules/installer/cd-dvd/sd-image.nix b/nixos/modules/installer/cd-dvd/sd-image.nix
index 901c60be..80b6e715 100644
--- a/nixos/modules/installer/cd-dvd/sd-image.nix
+++ b/nixos/modules/installer/cd-dvd/sd-image.nix
@@ -113,13 +113,13 @@ in

   config = {
     fileSystems = {
-      "/boot/firmware" = {
+      "/boot" = {
         device = "/dev/disk/by-label/FIRMWARE";
         fsType = "vfat";
         # Alternatively, this could be removed from the configuration.
         # The filesystem is not needed at runtime, it could be treated
         # as an opaque blob instead of a discrete FAT32 filesystem.
-        options = [ "nofail" "noauto" ];
+        #options = [ "nofail" "noauto" ];
       };
       "/" = {
         device = "/dev/disk/by-label/NIXOS_SD";

The error I get:

Hey! I am the author of the story you used to create the SD image. Glad that it went smoothly for you!

You’re using a Pi 3 which is actually the easiest model to get going with NixOS. You don’t need all of that cruft in your /etc/nixos/configuration.nix, especially everything related to the firmware partition!

On the repo associated with my article there is an example configuration for the Pi 3 which should work out of the box. Here is it for visibility:

# Please read the comments!
{ config, pkgs, lib, ... }:
{
  # Boot
  boot.loader.grub.enable = false;
  boot.loader.raspberryPi.enable = true;
  boot.loader.raspberryPi.version = 3;
  boot.loader.raspberryPi.uboot.enable = true;

  # Kernel configuration
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = ["cma=32M"];

  # Enable additional firmware (such as Wi-Fi drivers).
  hardware.enableRedistributableFirmware = true;

  # Filesystems
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  # Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
  networking.hostName = "nixpi"; # unleash your creativity!

  # Packages
  environment.systemPackages = with pkgs; [
    # customize as needed!
    vim git htop
  ];

  # Users
  # === IMPORTANT ===
  # Change `yourName` here with the name you'd like for your user!
  users.users.yourName = {
    isNormalUser = true;
    # Don't forget to change the home directory too.
    home = "/home/yourName";
    # This allows this user to use `sudo`.
    extraGroups = [ "wheel" ];
    # SSH authorized keys for this user.
    openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ];
  };

  # Miscellaneous
  time.timeZone = "Europe/Rome"; # you probably want to change this -- otherwise, ciao!
  services.openssh.enable = true;

  # WARNING: if you remove this, then you need to assign a password to your user, otherwise
  # `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
  # by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
  security.sudo.wheelNeedsPassword = false;

  # Nix
  nix.gc.automatic = true;
  nix.gc.options = "--delete-older-than 30d";
  boot.cleanTmpDir = true;

  # https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
  system.stateVersion = "20.03";
}

After putting this in /etc/nixos/configuration.nix, a simple nixos-rebuild switch should do.

There is no need to import any hardware configuration or any other file with this. It can also be quite easily modularized in multiple files!

Hopefully this helps!
Roberto

1 Like

Hi @robertof!

Thank you very much for jumping in here and proving a solution. I think the trick that made the new configuration load at boot was adding the fileSystems config without the extra firmware mounting stuff like you said. I also added the raspberry pi options:

{ config, pkgs, lib, ... }:
{
  imports = [
    /etc/nixos/common.nix
    /etc/nixos/uk_wifi.nix
  ];

  hardware = {
    enableRedistributableFirmware = true;
    firmware = [ pkgs.wireless-regdb ];
  };

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };

  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  boot = {
    kernelPackages = pkgs.linuxPackages_latest;
    kernelParams = ["cma=32M"];
    loader = {
      grub.enable = false;
      raspberryPi = {
        enable = true;
        version = 3;
        uboot.enable = true;
      };
      timeout = 1;
    };
    cleanTmpDir = true;
    extraModprobeConfig = ''
      options cf680211 ieee80211_regdom="GB"
    '';
  };

  nix.gc = {
    automatic = true;
    options = "--delete-older-than 10d";
  };

  networking = {
    enableIPv6 = false;
    hostName = "robert";
    wireless.enable = true;
    defaultGateway = {
      address = "192.168.1.254";
      interface = "wlan0";
    };
}

Thank you very very very much for your original article, it is a really great explanation.

1 Like

@sebohe @Mic92 , when you switched to not mounting /boot, did you also have to make / a bootable partition and make /boot non-bootable?

If you are talking about the RPI3, no.

I still have the original firmware directory in p1 but it is not mounted:

This is the output of fdisk:

Disk /dev/mmcblk0: 29.03 GiB, 31167873024 bytes, 60874752 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos 
Disk identifier: 0x2178694e
                                       
Device         Boot Start      End  Sectors Size Id Type
/dev/mmcblk0p1      16384    77823    61440  30M  b W95 FAT32
/dev/mmcblk0p2 *    77824 60874751 60796928  29G 83 Linux

This is first partition that has the boot loader:

-rwxr-xr-x  1 root root     512 Jun 22 18:33 armstub8-gic.bin
-rwxr-xr-x  1 root root   49090 Jun 22 18:33 bcm2711-rpi-4-b.dtb
-rwxr-xr-x  1 root root   52456 Jun 22 18:33 bootcode.bin
-rwxr-xr-x  1 root root     650 Jun 22 18:33 config.txt
-rwxr-xr-x  1 root root    3187 Jun 22 18:33 fixup4cd.dat
-rwxr-xr-x  1 root root    5448 Jun 22 18:33 fixup4.dat
-rwxr-xr-x  1 root root    8452 Jun 22 18:33 fixup4db.dat
-rwxr-xr-x  1 root root    8454 Jun 22 18:33 fixup4x.dat
-rwxr-xr-x  1 root root    3187 Jun 22 18:33 fixup_cd.dat
-rwxr-xr-x  1 root root    7314 Jun 22 18:33 fixup.dat
-rwxr-xr-x  1 root root   10298 Jun 22 18:33 fixup_db.dat
-rwxr-xr-x  1 root root   10298 Jun 22 18:33 fixup_x.dat
-rwxr-xr-x  1 root root  793116 Jun 22 18:33 start4cd.elf
-rwxr-xr-x  1 root root 3722504 Jun 22 18:33 start4db.elf
-rwxr-xr-x  1 root root 2228800 Jun 22 18:33 start4.elf
-rwxr-xr-x  1 root root 2981192 Jun 22 18:33 start4x.elf
-rwxr-xr-x  1 root root  793116 Jun 22 18:33 start_cd.elf
-rwxr-xr-x  1 root root 4794472 Jun 22 18:33 start_db.elf
-rwxr-xr-x  1 root root 2952960 Jun 22 18:33 start.elf
-rwxr-xr-x  1 root root 3704808 Jun 22 18:33 start_x.elf
-rwxr-xr-x  1 root root  540032 Jun 22 18:33 u-boot-rpi3.bin
-rwxr-xr-x  1 root root  584824 Jun 22 18:33 u-boot-rpi4.bin

Here is my rpi3 config: https://github.com/Mic92/dotfiles/blob/08b8181ad31836e24cb0f87319461e34b5df792d/nixos/modules/rpi3.nix

1 Like
Device         Boot Start      End  Sectors Size Id Type
/dev/mmcblk0p1      16384    77823    61440  30M  b W95 FAT32
/dev/mmcblk0p2 *    77824 60874751 60796928  29G 83 Linux

Looks like you’ve removed the boot flag on the firmware partition and added it to the main partition.

I tried doing this but I am running into issues with booting on NixOS 21.05.

U-Boot says:

ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree

Then it tries the next entry. On the devices where I still have a 19.03 entry, it works for that one.

Here’s my configuration:

{ config, pkgs, lib, ... }:

{

  nixpkgs.localSystem = {
    config = "aarch64-unknown-linux-gnu";
    system = "aarch64-linux";
  };

  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = [ "cma=256M" ];
  boot.initrd.kernelModules = [ "vc4" "bcm2835_dma" "i2c_bcm2835" ];
  boot.loader = {
    grub.enable = false;
    generic-extlinux-compatible.enable = true;
    raspberryPi = {
      enable = true;
      version = 3;
      uboot.enable = true;
      firmwareConfig = ''
        hdmi_safe=1
      '';
    };
  };

  fileSystems."/" = {
    device = "/dev/disk/by-label/NIXOS_SD";
    fsType = "ext4";
  };
  # fileSystems."/boot" = {
  #   device = "/dev/disk/by-label/NIXOS_BOOT";
  #   fsType = "vfat";
  # };
  swapDevices = lib.mkIf (!config.our.basic.disableSwap) [
    { device = "/swapfile"; size = 1024; }
  ];

  services.logind.extraConfig = "RuntimeDirectorySize=50%";
  documentation.nixos.enable = false;

  environment.systemPackages = with pkgs; [
    libraspberrypi
  ];

  hardware.firmware = with pkgs; [
    raspberrypiWirelessFirmware
  ];

}

I also tried using boot.kernelPackages = linuxPackages_rpi3 but that doesn’t build on the nixos-21.05 channel.

Edit: this is with nixpkgs commit fd8a7fd07da0f3fc0e27575891f45c2f88e5dd44

Edit2: just tried nixos-unstable commit 31ffc50c571e6683e9ecc9dbcbd4a8e9914b4497 ; no change

After talking at length with @samueldr we were able to resolve my issue.

TLD to upgrade it and get booting working I did this:

  1. Delete files in NIXOS_BOOT (the old /boot partition) partition
  2. Copy files in the FIRMWARE partition of the latest nixos-21.05 Raspberry Pi SD image to NIXOS_BOOT
  3. un-mount NIXOS_BOOT
  4. use fdisk to modify the boot flag to be on NIXOS_SD (the root partition) and not on NIXOS_BOOT
  5. Confirm that you are going to use Linux 5.10 (5.14 doesn’t boot)
  6. run nixos-rebuild
  7. Confirm /boot looks good (has /boot/extlinux/extlinux.conf and /boot/nixos/)
  8. Reboot

Here’s my config that works:

{ config, pkgs, lib, ... }:

{

  nixpkgs.localSystem = {
    config = "aarch64-unknown-linux-gnu";
    system = "aarch64-linux";
  };

  boot.loader.grub.enable = false;
  boot.loader.generic-extlinux-compatible.enable = true;
  boot.kernelParams = [ "cma=32M" ];
  boot.initrd.kernelModules = [ "vc4" "bcm2835_dma" "i2c_bcm2835" ];
  hardware.enableRedistributableFirmware = true;

  fileSystems."/" = {
    device = "/dev/disk/by-label/NIXOS_SD";
    fsType = "ext4";
  };

  swapDevices = lib.mkIf (!config.our.basic.disableSwap) [
    { device = "/swapfile"; size = 1024; }
  ];

  services.logind.extraConfig = "RuntimeDirectorySize=50%";

  documentation.nixos.enable = false;

  environment.systemPackages = with pkgs; [
    libraspberrypi
  ];

}

If you can, I strongly recommend trying these instructions on some duplicate SD card before potentially getting your SD card into an unbootable state.

2 Likes

@ryantm I’m going through a somewhat similar issue with BTRFS root on an RPi3, with the initial boot working fine, but a boot loop after nixos-install.

My goal is to have everything booting from my ztsd-compressed @boot BTRFS subvolume. It looks like u-boot supports both zstd decompression and has no trouble booting from subvolumes, so I’m not sure where my hangup is.

I’ve been detailing my (slow) progress in a forum post as well as a repo:

Just thought I’d post here as well since I’m getting a vaguely similar error after nixos-install and reboot:

ERROR: arch-specific fdt fixup failed
 - must RESET the board to recover.

FDT creation failed!
resetting ...

I tried removing the bootable flag from /dev/disk/by-label/FIRMWARE after reading this thread, which didn’t help.

If anyone has any input, I’d greatly appreciate it!

EDIT: And apologize in advance for an only tangentially related post – not hoping to hijack!