Can't boot into new configuration after nix-rebuild, raspberry pi3 B

I have been following this guide to create a headless sd image for my raspberry pi 3 B. It works great and I can create a bootable nixos on my pi3 and I can ssh into it perfectly fine.

For context:
  1. The method I’m using to create the sd image is with https://github.com/nix-community/nixos-generators and with this patch: https://github.com/NixOS/nixpkgs/pull/82718. Command: nixos-generate -f sd-aarch64-installer --system aarch64-linux -c ./pi3-sd.nix -I nixpkgs=./nixpkgs --cores 8
  2. I’m using pkgs.linuxPackages_latest since pkgs.linuxPackages_rpi3 doesn’t work for me. Possibly related with this: https://github.com/NixOS/nixpkgs/issues/82455
  3. nixpkgs is at release-20.03 brach
Steps I take to install a new nixos configuration:
  1. mkdir /boot/firmware && mount /dev/disk/by-label/FIRMWARE /boot/firmware

  2. I scp over the machine specific /etc/nixos/configuration.nix and I run nixos-rebuild boot (switch sometimes doesn’t work but that is for another post). The machine configuration.nix has the following:

/etc/nixos/configuration.nix
  1. Reboot
{ config, pkgs, lib, ... }:
{
  imports = [
    #/etc/nixos/hardware-configuration.nix # not importing this for now, see config.fileSystems 
    /etc/nixos/common.nix # contains ssh, wireless configs, and pkgs
  ];

  hardware = {
    enableRedistributableFirmware = true;
    firmware = [ pkgs.wireless-regdb ];
  };

  # Defining fileSystems here allow sme to not have to run nixos-generate-config
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-uuid/44444444-4444-4444-8888-888888888888";
      fsType = "ext4";
    };
    "/boo/firmware" = {
      device = "/dev/disk/by-uuid/2178-694E";
      fsType = "vfat";
    };
  };

  boot = {
    #kernelPackages = pkgs.linuxPackages_rpi3;
    kernelPackages = pkgs.linuxPackages_latest;
    kernelParams = ["cma=32M"];
    loader = {
      grub.enable = false;
      systemd-boot.enable = true;
      raspberryPi.firmwareConfig = ''
        boot_delay=1
      '';
      efi.canTouchEfiVariables = true;
      timeout = 1;
    };
    extraModprobeConfig = ''
      options cf680211 ieee80211_regdom="GB"
    '';
  };

  networking = {
    enableIPv6 = false;
    hostName = "robert";
    defaultGateway = {
      address = "192.168.1.254";
      interface = "wlan0";
    };
}
The problem:

The new configuration.nix isn’t loaded at boot, the old configuration persists. I think my problem is similar to this one: https://github.com/NixOS/nixpkgs/issues/85875#issuecomment-624852290. The diference here being that I there exists the firmware partition because the sd image has the boot dir in the root partition, but before running nixos-rebuild boot I make sure I mount the firmware paritition in step 1. Am I missing something here?

After step 2 nixos-rebuild boot:

[root@nixos:~]# sudo nix-env --list-generations --profile /nix/var/nix/profiles/system
   1   1970-01-01 00:00:13
   2   2020-06-23 14:45:46   (current)

After reboot I checked the contents of /run/current-system/configuration.nix and it is the same config as the one used to create the sd image, so to me it seems that I haven’t booted into the new configuration (I have system.copySystemConfiguration = true; in both the machine configuration.nix and the sd-image.nix).

Possible solutions and what I need help with

I checked the contents of the raspbian os image and saw that /boot contains all of the firmware files, in contrast to the sd-image.nix which has all the firmware files in /boot/firmware. So what I would like to is overwrite the configuration.fileSystems in nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix

  config = {
    fileSystems = {
      "/boot/firmware" = {
        device = "/dev/disk/by-label/FIRMWARE";
        fsType = "vfat";
        # Alternatively, this could be removed from the configuration.
        # The filesystem is not needed at runtime, it could be treated
        # as an opaque blob instead of a discrete FAT32 filesystem.
        options = [ "nofail" "noauto" ];
      };
      "/" = {
        device = "/dev/disk/by-label/NIXOS_SD";
        fsType = "ext4";
      };
    };
  ....
  };

from https://github.com/NixOS/nixpkgs/blob/39da4240609ee0d8ea533f142ae4c7e25df95980/nixos/modules/installer/cd-dvd/sd-image.nix#L114-#L123

And use this instead in my sd-image.nix:

    fileSystems = {
      "/boot" = {
        device = "/dev/disk/by-label/FIRMWARE";
        fsType = "vfat";
      };
      "/" = {
        device = "/dev/disk/by-label/NIXOS_SD";
        fsType = "ext4";
      };
    };

That way all the firmware files are written to /boot instead of /boot/firmware. I just don’t know how to overwrite attributes yet: The option fileSystems./boot.’ has conflicting definitions…. Maybe this is completely the wrong way to approach this problem because what I'm trying to do is not part f the attributes of sdImage`.

Thanks for reading!

Edit: I didn’t know that discourse would comment on all the github issues. Sorry for the spam!

I changed the config.fileSystems directly in nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix in the local copy I have of nixpkgs and that didn’t work… for understandable reasons.

This is what I changed:

diff --git a/nixos/modules/installer/cd-dvd/sd-image.nix b/nixos/modules/installer/cd-dvd/sd-image.nix
index 901c60be..80b6e715 100644
--- a/nixos/modules/installer/cd-dvd/sd-image.nix
+++ b/nixos/modules/installer/cd-dvd/sd-image.nix
@@ -113,13 +113,13 @@ in

   config = {
     fileSystems = {
-      "/boot/firmware" = {
+      "/boot" = {
         device = "/dev/disk/by-label/FIRMWARE";
         fsType = "vfat";
         # Alternatively, this could be removed from the configuration.
         # The filesystem is not needed at runtime, it could be treated
         # as an opaque blob instead of a discrete FAT32 filesystem.
-        options = [ "nofail" "noauto" ];
+        #options = [ "nofail" "noauto" ];
       };
       "/" = {
         device = "/dev/disk/by-label/NIXOS_SD";

The error I get:

Hey! I am the author of the story you used to create the SD image. Glad that it went smoothly for you!

You’re using a Pi 3 which is actually the easiest model to get going with NixOS. You don’t need all of that cruft in your /etc/nixos/configuration.nix, especially everything related to the firmware partition!

On the repo associated with my article there is an example configuration for the Pi 3 which should work out of the box. Here is it for visibility:

# Please read the comments!
{ config, pkgs, lib, ... }:
{
  # Boot
  boot.loader.grub.enable = false;
  boot.loader.raspberryPi.enable = true;
  boot.loader.raspberryPi.version = 3;
  boot.loader.raspberryPi.uboot.enable = true;

  # Kernel configuration
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = ["cma=32M"];

  # Enable additional firmware (such as Wi-Fi drivers).
  hardware.enableRedistributableFirmware = true;

  # Filesystems
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  # Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
  networking.hostName = "nixpi"; # unleash your creativity!

  # Packages
  environment.systemPackages = with pkgs; [
    # customize as needed!
    vim git htop
  ];

  # Users
  # === IMPORTANT ===
  # Change `yourName` here with the name you'd like for your user!
  users.users.yourName = {
    isNormalUser = true;
    # Don't forget to change the home directory too.
    home = "/home/yourName";
    # This allows this user to use `sudo`.
    extraGroups = [ "wheel" ];
    # SSH authorized keys for this user.
    openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ];
  };

  # Miscellaneous
  time.timeZone = "Europe/Rome"; # you probably want to change this -- otherwise, ciao!
  services.openssh.enable = true;

  # WARNING: if you remove this, then you need to assign a password to your user, otherwise
  # `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
  # by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
  security.sudo.wheelNeedsPassword = false;

  # Nix
  nix.gc.automatic = true;
  nix.gc.options = "--delete-older-than 30d";
  boot.cleanTmpDir = true;

  # https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
  system.stateVersion = "20.03";
}

After putting this in /etc/nixos/configuration.nix, a simple nixos-rebuild switch should do.

There is no need to import any hardware configuration or any other file with this. It can also be quite easily modularized in multiple files!

Hopefully this helps!
Roberto

1 Like

Hi @robertof!

Thank you very much for jumping in here and proving a solution. I think the trick that made the new configuration load at boot was adding the fileSystems config without the extra firmware mounting stuff like you said. I also added the raspberry pi options:

{ config, pkgs, lib, ... }:
{
  imports = [
    /etc/nixos/common.nix
    /etc/nixos/uk_wifi.nix
  ];

  hardware = {
    enableRedistributableFirmware = true;
    firmware = [ pkgs.wireless-regdb ];
  };

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };

  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  boot = {
    kernelPackages = pkgs.linuxPackages_latest;
    kernelParams = ["cma=32M"];
    loader = {
      grub.enable = false;
      raspberryPi = {
        enable = true;
        version = 3;
        uboot.enable = true;
      };
      timeout = 1;
    };
    cleanTmpDir = true;
    extraModprobeConfig = ''
      options cf680211 ieee80211_regdom="GB"
    '';
  };

  nix.gc = {
    automatic = true;
    options = "--delete-older-than 10d";
  };

  networking = {
    enableIPv6 = false;
    hostName = "robert";
    wireless.enable = true;
    defaultGateway = {
      address = "192.168.1.254";
      interface = "wlan0";
    };
}

Thank you very very very much for your original article, it is a really great explanation.

1 Like