Unexpected error with SSH during init

I wanted to do a remote LUKS unlock during boot so I followed the instructions and set it up like this:

  # boot.kernelParams = [ "rd.systemd.debug_shell=1" ];
  boot.initrd = {
    availableKernelModules = [ "r8169" ];
    luks.devices = {
      "root" = {
        device = "/dev/disk/by-uuid/ABC123DE-F456-ABC7-89DE-F012ABC345DE";
        allowDiscards = true;
        bypassWorkqueues = true;
      };
    };
    network = {
      enable = true;
      ssh = {
        enable = true;
        authorizedKeys = [ ''command="systemctl default" ssh-ed25519 <SNIP>'' ];
        hostKeys = [ "/etc/secrets/initrd/ssh_host_ed25519_key" ];
      };
    };
    systemd.network = {
      enable = true;
      networks."10-enp4s0" = {
        matchConfig.MACAddress = "AB:12:CD:34:EF:56";
        networkConfig.DHCP = "ipv4";
        linkConfig.RequiredForOnline = "routable";
      };
    };
  };

If I wait a couple of minutes, it’s perfectly fine. But if I connect too fast after the LUKS prompt appears, I get this error:

Failed to start default.target: Transaction for initrd.target/start is destructive (emergency.target has 'start' job queued, but 'stop' is included in transaction).

Apparently the parents are fighting over little TTY. They’ll get over it after a few minutes, but I assume it is unintended behavior.

Reads like your machine enters rescue mode. Could it be that it exits that mode somehow in the couple of minutes you mentioned, and it works then?

Do you have more info, like the other TTY’s/display output?

The part I don’t get is why it would fix itself. AI-powered rescue shell?

It’s been a while since I plugged in a screen but I don’t recall any strange messages except this one. What would you like to know?

There really is no way to know without more info.

$ journald -b might give you hints without connecting a screen: any failed jobs? dmsg doesn’t log the rescue shell tho, (if this is whats happening).

Propabably timeouts it wait’s for. DefaultDeviceTimeoutSec is 90 seconds for example.

Well, here’s the emergency part:

systemd[1]: dev-disk-by\x2duuid-xxxxxxx4\x2d92a3\xxxxxxxxxxx8ffa\x2d78xxxxxxxxxx.device: Job dev-disk-by\x2duuid-xxxxxxx4\x2d92a3\xxxxxxxxxxx8ffa\x2d78xxxxxxxxxx.>
systemd[1]: Timed out waiting for device /dev/disk/by-uuid/3bce73xxxxxxxxxx0b-8ffa-78xxxxxxxxxx.
systemd[1]: Dependency failed for /sysroot/var/log.
systemd[1]: Dependency failed for Initrd File Systems.
systemd[1]: initrd-fs.target: Job initrd-fs.target/start failed with result 'dependency'.
systemd[1]: initrd-fs.target: Triggering OnFailure= dependencies.
systemd[1]: sysroot-var-log.mount: Job sysroot-var-log.mount/start failed with result 'dependency'.
systemd[1]: Dependency failed for /sysroot/nix.
systemd[1]: Dependency failed for Find NixOS closure.
systemd[1]: initrd-find-nixos-closure.service: Job initrd-find-nixos-closure.service/start failed with result 'dependency'.
systemd[1]: sysroot-nix.mount: Job sysroot-nix.mount/start failed with result 'dependency'.
systemd[1]: Dependency failed for /sysroot/var/lib.
systemd[1]: sysroot-var-lib.mount: Job sysroot-var-lib.mount/start failed with result 'dependency'.
systemd[1]: Dependency failed for Initrd Root Device.
systemd[1]: initrd-root-device.target: Job initrd-root-device.target/start failed with result 'dependency'.
systemd[1]: initrd-root-device.target: Triggering OnFailure= dependencies.
systemd[1]: Dependency failed for /sysroot.
systemd[1]: Dependency failed for /sysroot/run.
systemd[1]: sysroot-run.mount: Job sysroot-run.mount/start failed with result 'dependency'.
systemd[1]: Dependency failed for Initrd Root File System.
systemd[1]: initrd-root-fs.target: Job initrd-root-fs.target/start failed with result 'dependency'.
systemd[1]: initrd-root-fs.target: Triggering OnFailure= dependencies.
systemd[1]: sysroot.mount: Job sysroot.mount/start failed with result 'dependency'.
systemd[1]: dev-disk-by\x2duuid-xxxxxxx4\x2d92a3\xxxxxxxxxxx8ffa\x2d78xxxxxxxxxx.device: Job dev-disk-by\x2duuid-xxxxxxx4\x2d92a3\xxxxxxxxxxx8ffa\x2d78xxxxxxxxxx.>
systemd[1]: panic-on-fail.service skipped, no trigger condition checks were met.
systemd[1]: Stopping Dispatch Password Requests to Console...
systemd[1]: Starting Create Volatile Files and Directories in the Real Root...
systemd[1]: Finished Create Volatile Files and Directories in the Real Root.
systemd[1]: systemd-ask-password-console.service: State 'stop-sigterm' timed out. Killing.
systemd[1]: systemd-ask-password-console.service: Killing process 278 (systemd-tty-ask) with signal SIGKILL.
systemd[1]: systemd-ask-password-console.service: Main process exited, code=killed, status=9/KILL
systemd[1]: systemd-ask-password-console.service: Failed with result 'timeout'.
systemd[1]: Stopped Dispatch Password Requests to Console.
systemd[1]: systemd-ask-password-console.path: Deactivated successfully.
systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
systemd[1]: Started Emergency Shell.
systemd[1]: Reached target Emergency Mode.

A few minutes later I log on, unlock the device and suddenly it found the device it was getting all moody about, stops the emergency shell, mounts the root, kicks me out of the SSH session as it stops initrd and continues booting.

So it’s panicking because it is looking for the BTRFS filesystem which is exposed by unlocking the luks.devices in the config?

EDIT: Is it because I created subvolumes for /var/lib and /var/log? Old-school partitioning habits, but I would hope it’s smart enough to realize LUKS unlock comes before filesystems. If not, I suspect the same situation occurs if I turn the sshd off, it’s caused by having systemd in the init?

What’s your fileSystems block looking like? I think there is a “gotcha” of sorts with using by-uuid for disks, whereas recent Nixpkgs expects or at least blesses /dev/mapper/<mapper> (in your case /dev/mapper/root) instead.