Setup with LUKS detached headers in LUKS partition

Help
1

You can see my configuration and installation scripts at nixos-dotfiles/deuterium at main - someonewithpc/nixos-dotfiles - Codeberg.org

I’m trying to setup NixOS in a system with a (perhaps overly) complicated partition scheme, where

  • Two USBs with a FAT32 ESP and a LUKS partition which then has a RAID1 BTRFS filesystem containing keys and headers
  • 8 SCSI disks formatted with LUKS that then contain a RAID5 BTRFS filesystem

I managed to get the USBs to mount and for the SCSI disks to wait and use the keys and headers, but the problem is that the root filesystem only supports one device, rather than devices, which means that not all cryptography setups for the disks are finished before it attempts to mount /sysroot, in stage 1, so it fails, as seen below

Boot Log
Boot Log1920×873 173 KB

I’ve tried to add

boot.initrd.systemd.targets = {
  "cryptsetup" = {
    after = ["usb.mount"];
  };
};

but i doesn’t seem to have worked.

How can I make sure the sysroot.target waits for all my disks? Or should I take some other approach entirely?

(I plan to attempt to make a PR for adding support for a filesystem to specify devices, but wanted to get this working first)

PS I’ve since found the functions to properly escape systemd unit names, just haven’t pushed those changes

2

This is an important point for me to begin with, because this isn’t possible. A systemd mount unit can only have one What= directive. You can’t have a plural devices option because we can only put one device in the first field of an fstab file entry, which is translated to a mount unit which can only have one What= directive derived from that field.

Btrfs is supposed to solve this with udev rules, so I am surprised this is breaking for you. The relevant rules, from 64-btrfs.rules:

SUBSYSTEM!="block", GOTO="btrfs_end"
ACTION=="remove", GOTO="btrfs_end"
ENV{ID_FS_TYPE}!="btrfs", GOTO="btrfs_end"
ENV{SYSTEMD_READY}=="0", GOTO="btrfs_end"

# let the kernel know about this btrfs filesystem, and check if it is complete
IMPORT{builtin}="btrfs ready $devnode"

# mark the device as not ready to be used by the system
ENV{ID_BTRFS_READY}=="0", ENV{SYSTEMD_READY}="0"

# reconsider pending devices in case when multidevice volume awaits
ENV{ID_BTRFS_READY}=="1", RUN+="/nix/store/1lbc6v5p1a3rn4rjaqnz0694xfbq8dxq-systemd-256.4/bin/udevadm trigger -s block -p ID_BTRFS_READY=0"

LABEL="btrfs_end"

With these rules, a btrfs device should not be found by systemd until all the devices have been found. These rules are meant to make it wait for that.

If this isn’t working for you, that’s a bug we need to fix.

No, I wouldn’t expect that to do anything useful. Delaying cryptsetup.target till after your usb.mount wouldn’t do anything meaningful; file systems are ordered after their specific devices, not after cryptsetup.target; and anyway usb.mount is only your USB drive file system, not the dmcrypt devices you’re looking for.

Well, first of all, it’s sysroot.mount, not sysroot.target. And there is a workaround. You can add x-systemd.requires= to the file system options, and that will cause systemd to delay the mount till after the required units. So you can do fileSystems."/".options = ["x-systemd.requires=dev-mapper-${mapperName}.device"];, but with one list element specifying each specific device. But again, and I can’t stress this enough, the btrfs udev rules are designed to make this unnecessary, so I’d rather get to the bottom of why that isn’t working for you.

3

Yes, I understand that, but what I was thinking of was adding the devices as After dependencies. That way it could even be possible for having it mount if one of the devices is missing, maybe. Currently, the sysroot.mount unit depends on the device I specified and the system.slice unit (whose purpose I’m not familiar with). Alternatively, the existing depends could maybe be added to the dependencies, no? If that wouldn’t work, there should still be some easier way to achieve my setup, I think. Having to write the usb.mount, for instance, seems like it should be handled automatically, somehow

Well, they’re not working in my case, I’m not sure why. Maybe because it’s BTRFS on LUKS? Or somehow because of the detached headers? I don’t know much about udev

Yeah, that was a brainfart, I should have it depend on the /dev/mapper devices, but I don’t know if that would work

Progress! That sort of worked, but now the volume fails to mount .-. The filesystem mount correctly in the live USB environment, but in the rescue shell, mounting it manually yields

mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/mapper/scsi, missing codepage or helper program, or other error.

I suspect I’m missing some kernel module, probably something needed for raid56 and not raid1, as the USBs mount fine. I’m investigating, but if you know, I’d welcome the help. I think I just need to add the raid456 module, but I’m working on it. This machine takes quite a while to boot

4

You need to check the dmesg after it fails.

5

I’m not that big of a noob e.e But I couldn’t find anything, so I’m working under the theory that it was because the LUKS partition was created with an offset that it wasn’t mounting correctly, as I couldn’t find a way to specify that offset under NixOS without more hacks. I don’t recall where I even saw that recommendation and couldn’t find it again, so I re-formatted the drives and it’s (still) installing (this thing is slow). I’ll mark your previous post as the solution and I’ll post again or update here if I run into more problems, but hopefully it works now

6

Ok, after a lot of banging my head against the wall, I closed the LUKS devices, reopened them, and now the BTRFS volume mounts fine, so, apparently, the issue is with LUKS, not BTRFS… I’m noticing that under lsblk, with my manual mount, the disks are displaying as

sda        disk
`-sda1     part
  `-scsi-* crypt

Whereas with the way the system is mounting them, they display as

sda     disk
|-sda1  part
`- scsi crypt

So… I was opening the disk instead of the partition. I wouldn’t expect cryptsetup to allow that, though, I have no idea why it was “working”.

Anyway, changing the device to use the partition instead worked. Now I just need to clean it up and remove the whole bunch of unnecessary stuff I added

This is probably also why the BRTFS udev rules weren’t working