External luks storage is not mounted on raspberrypi boot

I have a raspberry pi 4 which I managed to setup with NixOS after a lot of work by reading resources like this and this. As in the blog post, I built the image with

nix run nixpkgs#nixos-generators -- -f sd-aarch64 --flake .#raspberrypi --system aarch64-linux -o ./raspberrypi.sd

and dd’ed into the SD card. With the image already inplace and the OS working, updates to the configuration are done directly via ssh and nixos-rebuild.

The problem is that I’m trying to mount an encrypted external storage using a keyfile present in the filesystem. It’s not the most secure approach but it’s what it’s worth for me right now. I used something like this in my hardware configuration:

  boot.initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  environment.etc.crypttab.text = ''
    backup UUID=bad5f4b7-fcdc-4b36-8887-357a629d7c00 /root/mykeyfile
  '';
  fileSystems."/mnt/backup" = {
    device = "/dev/mapper/backup";
  };

  fileSystems."/boot/firmware" = {
    device = "/dev/disk/by-label/FIRMWARE";
    fsType = "vfat";
    options = [ "nofail" "noauto" ];
  };

  fileSystems."/" = {
    device = "/dev/disk/by-label/NIXOS_SD";
    fsType = "ext4";
    options = [ "noatime" ];
  };

  swapDevices = [{ device = "/swapfile"; size = 128; }];

But when the system reboots I see logs in journalctl like this and the drive never mounts:

Mar 27 17:05:23 raspberrypi systemd[1]: Starting Cryptography Setup for backup...
Mar 27 17:05:23 raspberrypi systemd-cryptsetup[13088]: Volume backup already active.
Mar 27 17:05:23 raspberrypi systemd[1]: Finished Cryptography Setup for backup.

I’m not entirely sure but I believe when I used the same configuration in a different machine the external storage was succesfully mounted which leads me to think that is something related to the raspberry pi boot. This is the relevant part of my configuration:

  hardware = {
    enableRedistributableFirmware = true;
    raspberry-pi."4".apply-overlays-dtmerge.enable = true;
    deviceTree = {
      enable = true;
      filter = "*rpi-4-*.dtb";
    };
  };

  console.enable = true;

  boot = {
    # https://docs.syncthing.net/users/faq.html#inotify-limits
    kernel.sysctl = {
      "fs.inotify.max_user_watches" = 204800;
    };
    kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
    initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
    kernelParams = [
      "usb-storage.quirks=152d:0562:u,152d:2329:u,14cd:1212"
      "cgroup_enable=cpuset" "cgroup_enable=memory" "cgroup_memory=1" "swapaccount=1"
    ];
    loader = {
      grub.enable = false;
      generic-extlinux-compatible.enable = true;
    };
  };

Any thoughts or ideas?

What does systemctl status /mnt/backup say?

I tried two scenarios:

  1. power outtage (pulling the wall plug)
  2. sudo reboot now

first scenario

when there is a power outtage the drive does not mount and the output is

$ systemctl status /mnt/backup
○ mnt-backup.mount - /mnt/backup
     Loaded: loaded (/etc/fstab; generated)
     Active: inactive (dead)
      Where: /mnt/backup
       What: /dev/mapper/backup
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)

the only log in journalctl that says something about the drive is:

Mar 27 18:10:30 raspberrypi kernel: scsi 1:0:0:0: Direct-Access     KINGSTON  SA400S37960G    S340 PQ: 0 ANSI: 2 CCS
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] 1875385008 512-byte logical blocks: (960 GB/894 GiB)
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] Write Protect is off
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] Mode Sense: 28 00 00 00
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] No Caching mode page found
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] Assuming drive cache: write through
Mar 27 18:10:30 raspberrypi kernel: sd 1:0:0:0: [sdb] Attached SCSI disk

systemd-cryptsetup@backup.service says it failed. I believe the last lines lines are from a different run due to the date, maybe due the hard reboot. First line of the journal does say systemd-timesyncd[793]: System clock time unset or jumped backwards, restored from recorded timestamp though.

 $ systemctl status systemd-cryptsetup@backup.service                                                                                                                                   [S2][3]
× systemd-cryptsetup@backup.service - Cryptography Setup for backup
     Loaded: loaded (/etc/crypttab; generated)
     Active: failed (Result: signal) since Fri 2024-07-05 15:03:07 -03; 8 months 21 days ago
 Invocation: af59f2bf1c11457fa7220b4d5ad3bf29
       Docs: man:crypttab(5)
             man:systemd-cryptsetup-generator(8)
             man:systemd-cryptsetup@.service(8)
    Process: 752 ExecStart=/nix/store/1q4xgiznc9131d1zi7dk0b1q7xxjjdhc-systemd-256.10/bin/systemd-cryptsetup attach backup /dev/disk/by-uuid/bad5f4b7-fcdc-4b36-8887-357a629d7c00 /root/mykeyfile  (code=killed, sig>
   Main PID: 752 (code=killed, signal=TERM)
         IP: 0B in, 0B out
         IO: 2.2M read, 0B written
   Mem peak: 3.4M
        CPU: 61ms

Jul 05 15:03:06 raspberrypi systemd[1]: Starting Cryptography Setup for backup...
Jul 05 15:03:07 raspberrypi systemd[1]: systemd-cryptsetup@backup.service: Main process exited, code=killed, status=15/TERM
Jul 05 15:03:07 raspberrypi systemd[1]: systemd-cryptsetup@backup.service: Failed with result 'signal'.
Jul 05 15:03:07 raspberrypi systemd[1]: Stopped Cryptography Setup for backup.
Jul 05 15:03:07 raspberrypi systemd[1]: systemd-cryptsetup@backup.service: Consumed 61ms CPU time, 3.4M memory peak, 2.2M read from disk.

second scenario

When I reboot properly the drive mounts correctly

$ systemctl status /mnt/backup
● mnt-backup.mount - /mnt/backup
     Loaded: loaded (/etc/fstab; generated)
     Active: active (mounted) since Fri 2024-07-05 15:03:14 -03; 8 months 21 days ago
 Invocation: a1ec84b84859401288da2d589a3df351
      Where: /mnt/backup
       What: /dev/mapper/backup
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
         IP: 0B in, 0B out
         IO: 2.9M read, 4K written
      Tasks: 0 (limit: 4441)
     Memory: 492K (peak: 2.5M)
        CPU: 55ms
     CGroup: /system.slice/mnt-backup.mount

Notice: journal has been rotated since unit was started, output may be incomplete.
$ systemctl status systemd-cryptsetup@backup.service
● systemd-cryptsetup@backup.service - Cryptography Setup for backup
     Loaded: loaded (/etc/crypttab; generated)
     Active: active (exited) since Fri 2024-07-05 15:03:14 -03; 8 months 21 days ago
 Invocation: 24f26662614b4881bf2cb7f5b0128701
       Docs: man:crypttab(5)
             man:systemd-cryptsetup-generator(8)
             man:systemd-cryptsetup@.service(8)
    Process: 773 ExecStart=/nix/store/1q4xgiznc9131d1zi7dk0b1q7xxjjdhc-systemd-256.10/bin/systemd-cryptsetup attach backup /dev/disk/by-uuid/bad5f4b7-fcdc-4b36-8887-357a629d7c00 /root/mykeyfile  (code=exited, sta>
   Main PID: 773 (code=exited, status=0/SUCCESS)
         IP: 0B in, 0B out
         IO: 5.9M read, 0B written
   Mem peak: 347.3M
        CPU: 14.476s

Notice: journal has been rotated since unit was started, output may be incomplete.

All in all, the system does work when properly rebooted but not enough when there is a infrastructure/hardware issue.

bump, anyone has gone through this type of error?