Systemd initrd for USB based LUKS unlocking root

I have used to approach from the wiki for usb unlocking before switching to systemd initrd. Currently I always get asked for the password, even when the stick is present.

I have the following config:

initrd.systemd = {
  enable = true;
  initrdBin = with pkgs; [
    util-linux
  ];
  services.mount-usb-key = {
    wantedBy = [ "initrd.target" ];
    before = [ "systemd-cryptsetup@cryptroot.service" ];
    serviceConfig.Type = "oneshot";
    script = ''
      mkdir -m 0755 -p /key

      sleep 2 # To make sure the usb key has been loaded

      device=$(findfs LABEL=KEYS)
      if [ -n "$device" ]; then
          mount -n -t vfat -o ro "$device" /key
      else
          echo "USB stick not found"
      fi
    '';
  };
};

The service runs without any errors but I get prompted for the password anyways. I came across multiple people that suggested mounting the stick using either systemd.mounts or fstab. But as far as I know these don’t allow for findfs or similar. I have multiple usb sticks and don’t want to add them all to my config and having to change it every time a new one comes or one leaves.

I came about a similar post where the op settled for using systemd.mounts but it is explained that systemd-cryptsetup is a template service and has a unknown suffix. I thought it might be the luks device name, which is cryptroot in my case but that assumptions seems to be false as it doesn’t work.

Does somebody know what add to the before list or an alternative method?

Are you 100% sure that this service is running before systemd-cryptsetup@your_disk_name.service? You can check after it boots up with “journalctl -o verbose” and search just like you would in vim.

You probably just need unitConfig.DefaultDependencies = false;. This is one of the most annoying gotchas about systemd initrd; almost everything ought to have this, and if you forget it on the wrong thing, systemd completely blows up the ordering of units, and you get things like this where you’re prompted for a password before a keyfile is mounted.

That said, I think you can simplify this with a mount unit instead of a service. One nice thing about systemd-cryptsetup@.service is that it gets ordered with RequiresMountsFor=/path/to/keyfile automatically. And a mount unit will automatically gain a dependency on the underlying device unit so you don’t need that sleep 2 (which you could do with the service as well, but mounts do it automatically).

boot.initrd.systemd.mounts = [
  {
    what = "/dev/disk/by-label/KEYS";
    where = "/key";
    options = [ "ro" ];
    type = "vfat";
  }
];

Sadly both options don’t work for me, unitConfig.DefaultDependencies = false; causes the script to be run before /dev has been populated. With mounts it also doesn’t work, not sure why tho.

I also had to change options = [ "ro" ]; to options = "ro"; as nixos would complain about wrong type otherwise.

The closes I have come is this:

boot = {
    initrd.kernelModules = [
      "uas"
      "usbcore"
      "usb_storage"
      "vfat"
      "nls_cp437"
      "nls_iso8859_1"
    ];

    initrd.systemd = {
      enable = true;
      initrdBin = with pkgs; [
        util-linux
      ];
      services.mount-usb-key = {
        wantedBy = [ "initrd.target" ];
        after = [ "initrd-root-device.target" ];
        before = [ "systemd-cryptsetup@cryptroot.service" ];
        serviceConfig.Type = "oneshot";
        unitConfig.DefaultDependencies = false;
        script = ''
          mkdir -m 0755 -p /key

          set +e
          device=$(findfs LABEL=KEYS)
          set -e
          lsblk
          if [ -n "$device" ]; then
              mount -n -t vfat -o ro "$device" /key
          else
              echo "USB stick not found"
          fi
        '';
      };
    };
  };

But with this I get stuck in initrd on the decrypt service, eg. it never finishes (at least not in 30min)

I have done some further debugging as @zachliebl suggested, my last snippet doesn’t finish because initrd-root-device.target only gets run after systemd-cryptsetup@cryptroot.service causing an impossible dependency layout.

When using @ElvishJerricco suggested mounts options I have no idea when it tries to mount or if at all. I could find no mention of it using journalctl -o verbose.

All other services configurations I tried get ran after systemd-cryptsetup@cryptroot.service or before any devices have been loaded. I tried looking for a service or similiar that guarantees that all devices have been setup but none I tried worked.

I’m also not sure if this matters, I’m using disko to configure my drives. I have also verified that the location of the keyfile is the right one and that it is able to decrypt the luks partition

I will admit I have never done usb-key based luks unlocking before. But I have done ssh-based luks unlocking before and I had similar systemd-ordering issues. So I can give you advice based on that.

  1. It is never enough to specify “Before” if you want a service to run before a different service. You also have to include “WantedBy” or “RequiredBy”. Same thing for “After” and “Wants/Requires”.

  2. Remember that you can override or add to attributes of services that you have not defined yourself. In the example below I enable ssh with network.ssh.enable = true so I never write the ssh initrd service myself. But I still add dependencies to the systemd service.

  3. If you want to find the name of the service based on the message that appears during boot then you can run journalctl -o verbose and just search for the message.

I posted my initrd config in a post completely unrelated to this. But you can still read it and maybe it will help. Running NetworkManager in initrd - #3 by zachliebl

In that config I want ssh to run before I decrypt my disks. Notice that I added the following lines:

        services."sshd".unitConfig.WantedBy = [
          "systemd-cryptsetup@argon_zfs_root_1.service"
          "systemd-cryptsetup@argon_zfs_root_2.service"
        ];
        services."sshd".unitConfig.Before = [
          "systemd-cryptsetup@argon_zfs_root_1.service"
          "systemd-cryptsetup@argon_zfs_root_2.service"
        ];

This isn’t how you do that. Firstly, in a typical systemd unit, WantedBy goes in the [Install] section, not the [Unit] section. But secondly, nixos just straight up doesn’t implement the [Install] section like this. You do boot.initrd.systemd.services.foo.wantedBy = [ "bar.service" ];.

Huh, that’s weird. When I run systemd-cryptsetup-generator manually, the resulting systemd-cryptsetup@.service instance does have RequiresMountsFor=/path/to/key, which means it should gain a Requires= on the mountpoint and order itself after. I would really expect this to be all that’s needed:

fileSystems."/" = {
  device = "/dev/mapper/foo";
  fsType = "ext4";
};

boot.initrd = {
  luks.devices.foo = {
    device = "/dev/disk/by-whatever/foo";
    keyFile = "/key/file";
  };

  systemd.mounts = [
    {
      what = "/dev/disk/by-label/KEYS";
      where = "/key";
      options = [ "ro" ];
      type = "vfat";

      # This shouldn't be necessary, but worth a try
      requiredBy = [ "systemd-cryptsetup@foo.service" ];
      before = [ "systemd-cryptsetup@foo.service" ];
    }
  ];
};

Ok, now that I think about it putting the WantedBy in the [Install] section is cleaner than the [Unit] section, I will do that from now on.

Thanks for pointing it out that I need boot.initrd prefixes. That is definitely true, and I made a typo when making that reply. My actual config has it in a boot.initrd = {}; multi-line attribute set.

Ok so I made progress. There was a problem with my disko config, it wasn’t setting boot.initrd.luks.devices.<name>.keyFile so that’s way systemd.mounts didn’t worked. I fixed this and now I can successfully boot with usb unlocking but when the usb stick isn’t present decrypting fails and I don’t get asked for my password. I think this is because systemd-cryptsetup@cryptroot sets RequiresMountsFor instead of WantsMountsFor. I tried correcting this by setting:

services."systemd-cryptsetup@cryptroot".unitConfig = {
  RequiresMountsFor = [ ];
  WantsMountsFor = [ "/key" ];
};

but this causes complete boot failure, I guess because it gets generated and not set over nix.

I tried it with a service again but to no avail. It again runs before devices are present.

Oh interesting. Yea, you would need to make that change as a drop-in.

boot.initrd.systemd.services."systemd-cryptsetup@cryptroot" = {
  overrideStrategy = "asDropin";
  unitConfig = {
    RequiresMountsFor = [ ];
    WantsMountsFor = [ "/key" ];
  };
};

Still doesn’t work, while I can boot now, it has the same behavior as not overriding systemd-cryptsetup@cryptroot at all. After timeout I get placed into emergency mode.
With this error:

[ TIME ] Timed out mounting /key
[ DEPEND ] Dependency failed for Cryptography Setup for cryptroot.
[ DEPEND ] Dependency failed for ...
[ DEPEND ] Dependency failed for ...

This is my current config:

boot.initrd.systemd = {
  enable = true;
  mounts = [
    {
      what = "/dev/disk/by-label/KEYS";
      where = "/key";
      options = "ro";
      type = "vfat";
      unitConfig = {
        DefaultDependencies = false;
        JobTimeoutSec = 10;
      };
    }
  ];
  services."systemd-cryptsetup@cryptroot" = {
    overrideStrategy = "asDropin";
    unitConfig = {
      RequiresMountsFor = [ ];
      WantsMountsFor = [ "/key" ];
    };
  };
};

Oh this is very interesting. That’s a very unusual error, which indicates that it has found the device for /key and is trying to mount it, but the mount command itself is timing out. My guess is there’s something very flakey about the USB IO in your initrd for some reason.