Nixos-rebuild switch generation not picked upon reboot

I am finding that rebooting my Raspberry PI 4 nixos host is not using the latest generation, but a previous one (I’m not sure but I think pre-upgrade to 22.11).

Once logged back in I can restore functionality via nixos-rebuild switch until the next reboot…

I’ve been investigating and I can’t see why the latest generation is not taking.

Looking in /boot/nixos shows some rather old files:

/boot/nixos directory listing
# ls -lt /boot/nixos
total 176152
dr-xr-xr-x 4 root root     4096 Oct  1  2021 3yf7xdilncqyf67ihdxr63nc3sqil33g-linux-5.10.52-1.20210805-dtbs
-r--r--r-- 1 root root  8958275 Oct  1  2021 hwg2qar5diqb2a3zx6k2g56azhl2dcyr-initrd-linux-5.10.52-1.20210805-initrd
-r--r--r-- 1 root root 28697088 Oct  1  2021 3yf7xdilncqyf67ihdxr63nc3sqil33g-linux-5.10.52-1.20210805-Image
dr-xr-xr-x 4 root root     4096 Sep 29  2021 k38gc6clwxnf8aqpvc5cn45nq61nmp1q-linux-5.10.52-1.20210805-dtbs
-r--r--r-- 1 root root  8966348 Sep 29  2021 wglp8q6zbigmiz1qmsp6alvbxciwwsyr-initrd-linux-5.10.52-1.20210805-initrd
-r--r--r-- 1 root root 28697088 Sep 29  2021 k38gc6clwxnf8aqpvc5cn45nq61nmp1q-linux-5.10.52-1.20210805-Image
dr-xr-xr-x 4 root root     4096 Sep 26  2021 59mpcrm3ndbg32834038v1c6lzjh9yi7-linux-5.10.52-1.20210805-dtbs
-r--r--r-- 1 root root  7265858 Sep 26  2021 3mjfvinj4csxpwihmqash1wa8g3cgg3q-initrd-linux-5.10.52-1.20210805-initrd
-r--r--r-- 1 root root 25354752 Sep 26  2021 59mpcrm3ndbg32834038v1c6lzjh9yi7-linux-5.10.52-1.20210805-Image
dr-xr-xr-x 4 root root     4096 Sep  1  2021 p0lii29xcml080975b74534vcq2j7p56-linux-5.10.52-1.20210805-dtbs
-r--r--r-- 1 root root  7264363 Sep  1  2021 4aqgfnn14y4bgn7bgi2r1ihkk56jb9qr-initrd-linux-5.10.52-1.20210805-initrd
-r--r--r-- 1 root root 25338368 Sep  1  2021 p0lii29xcml080975b74534vcq2j7p56-linux-5.10.52-1.20210805-Image
-r--r--r-- 1 root root  7224874 Aug 30  2021 mva4ywq73nhs0n1w2gkavyjn5n5jhsd4-initrd-linux-5.10.52-1.20210805-initrd
dr-xr-xr-x 4 root root     4096 Aug 25  2021 k7ilnxkr3pfbgmq2f8ikz7313xmf4zhd-linux-5.10.52-1.20210805-dtbs
-r--r--r-- 1 root root  7224849 Aug 25  2021 s7b5zfxximdjggky79mn90gsqmpa72kc-initrd-linux-5.10.52-1.20210805-initrd
-r--r--r-- 1 root root 25338368 Aug 25  2021 k7ilnxkr3pfbgmq2f8ikz7313xmf4zhd-linux-5.10.52-1.20210805-Image

And in /boot the significant files seem up to date:

/boot directory listing
# ls -lt /boot
total 63912
-r--r--r-- 1 root root      400 Mar 19 10:30 config.txt
-r--r--r-- 1 root root  3721992 Mar 19 10:30 start_x.elf
-r--r--r-- 1 root root  4819144 Mar 19 10:30 start_db.elf
-r--r--r-- 1 root root   805436 Mar 19 10:30 start_cd.elf
-r--r--r-- 1 root root  2998344 Mar 19 10:30 start4x.elf
-r--r--r-- 1 root root  3747240 Mar 19 10:30 start4db.elf
-r--r--r-- 1 root root   805436 Mar 19 10:30 start4cd.elf
-r--r--r-- 1 root root  2250848 Mar 19 10:30 start4.elf
-r--r--r-- 1 root root  2975104 Mar 19 10:30 start.elf
-r--r--r-- 1 root root    10229 Mar 19 10:30 fixup_x.dat
-r--r--r-- 1 root root    10229 Mar 19 10:30 fixup_db.dat
-r--r--r-- 1 root root     3170 Mar 19 10:30 fixup_cd.dat
-r--r--r-- 1 root root     8386 Mar 19 10:30 fixup4x.dat
-r--r--r-- 1 root root     8382 Mar 19 10:30 fixup4db.dat
-r--r--r-- 1 root root     3170 Mar 19 10:30 fixup4cd.dat
-r--r--r-- 1 root root     5398 Mar 19 10:30 fixup4.dat
-r--r--r-- 1 root root     7265 Mar 19 10:30 fixup.dat
-r--r--r-- 1 root root    52476 Mar 19 10:30 bootcode.bin
-rw-r--r-- 1 root root      116 Mar 19 10:30 cmdline.txt
-r-xr-xr-x 1 root root     4519 Mar 19 10:30 nixos-init
-r-xr-xr-x 1 root root    52424 Mar 19 10:30 bcm2838-rpi-4-b.dtb
-r-xr-xr-x 1 root root    31230 Mar 19 10:30 bcm2837-rpi-zero-2.dtb
-r-xr-xr-x 1 root root    30157 Mar 19 10:30 bcm2837-rpi-cm3.dtb
-r-xr-xr-x 1 root root    32533 Mar 19 10:30 bcm2837-rpi-3-b-plus.dtb
-r-xr-xr-x 1 root root    31922 Mar 19 10:30 bcm2837-rpi-3-b.dtb
-r-xr-xr-x 1 root root    32533 Mar 19 10:30 bcm2837-rpi-3-a-plus.dtb
-r-xr-xr-x 1 root root    50447 Mar 19 10:30 bcm2711-rpi-cm4s.dtb
-r-xr-xr-x 1 root root    53165 Mar 19 10:30 bcm2711-rpi-cm4.dtb
-r-xr-xr-x 1 root root    52424 Mar 19 10:30 bcm2711-rpi-4-b.dtb
-r-xr-xr-x 1 root root    52556 Mar 19 10:30 bcm2711-rpi-400.dtb
-r-xr-xr-x 1 root root    31230 Mar 19 10:30 bcm2710-rpi-zero-2-w.dtb
-r-xr-xr-x 1 root root    31230 Mar 19 10:30 bcm2710-rpi-zero-2.dtb
-r-xr-xr-x 1 root root    30157 Mar 19 10:30 bcm2710-rpi-cm3.dtb
-r-xr-xr-x 1 root root    32533 Mar 19 10:30 bcm2710-rpi-3-b-plus.dtb
-r-xr-xr-x 1 root root    31922 Mar 19 10:30 bcm2710-rpi-3-b.dtb
-r-xr-xr-x 1 root root    30170 Mar 19 10:30 bcm2710-rpi-2-b.dtb
-r--r--r-- 1 root root 11295105 Mar 19 10:30 initrd
-r--r--r-- 1 root root 30552576 Mar 19 10:30 kernel.img
drwxr-xr-x 2 root root    36864 Mar 19 09:07 old
-r--r--r-- 1 root root   106432 Jan 23  2022 recovery.bin
-rw-r--r-- 1 root root   524288 Jan 23  2022 pieeprom.upd
-rw-r--r-- 1 root root       80 Jan 23  2022 pieeprom.sig
drwxr-xr-x 2 root root     4096 Oct  2  2021 extlinux
drwxr-xr-x 7 root root     4096 Oct  1  2021 nixos
# nix-shell -p nix-info --run nix-info
system: "aarch64-linux", multi-user?: yes, version: nix-env (Nix) 2.13.3, channels(root): "nixos", nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos
# uname -a
Linux nixos-rpi4 5.10.52 #1-NixOS SMP Tue Jan 1 00:00:00 UTC 1980 aarch64 GNU/Linux
# nix-env --list-generations --profile /nix/var/nix/profiles/system | cat
 268   2023-03-18 04:40:23   (current)

I used the following to clean out old generations:

# nix-env --delete-generations old --profile /nix/var/nix/profiles/system
# /nix/var/nix/profiles/system/bin/switch-to-configuration switch
configuration.nix
# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [
      # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # RPI4
  boot = {
    kernelPackages = pkgs.linuxPackages_rpi4;
    # kernelPackages = pkgs.linuxPackages_latest;
    tmpOnTmpfs = true;
    initrd.availableKernelModules = [ "usbhid" "usb_storage" ];
  };
  boot.loader.raspberryPi = {
    enable = true;
    version = 4;
  };
  # Use the extlinux boot loader. (NixOS wants to enable GRUB by default)
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  #boot.loader.generic-extlinux-compatible.enable = true;

  # Required for the Wireless firmware
  hardware.enableRedistributableFirmware = true;

# … trimmed down ...

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "22.11"; # Did you read the comment?

}

Nixos-rebuild switch calls nix-env --set /nix/var/nix/profiles/system $drv before calling switch-to-configuration switch

Without that the switch isn’t persisted in the profile

I usually do a nixos-rebuild switch (after a nixos-rebuild build) as my usual practice… everything looks well, but rebooting doesn’t bring up that.

So I suppose what you are saying @arianvp is that the nix-env --set /nix/var/nix/profiles/system $drv isn’t updating the boot configuration correctly?