A week and a bit later and I finally have holiday to try and debug this. I’ve been scratching my head over this for a further 3 hours now. I’ve since discovered that adding debug
to the kernelParams
will give better debug output, and using boot.initrd.systemd
will fail exactly the same way, but is more robust to non-root drives not mounting, since it doesn’t ask for the password ad infinitum, and is therefore nicer for debugging.
So now I have some logs! The broken drive:
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@storage.service: Will spawn child (service_enter_start): /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@storage.service: Passing 0 fds to service
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@storage.service: About to execute /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup attach storage /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 - cipher=serpent-xts-plain64,discard,no-read-workqueue,no-write-workqueue
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@storage.service: Forked /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup as 247
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@storage.service: Changed dead -> start
Dec 15 20:28:12 localhost systemd[1]: Starting Cryptography Setup for storage...
Dec 15 20:28:12 localhost systemd[247]: systemd-cryptsetup@storage.service: Executing: /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup attach storage /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 - cipher=serpent-xts-plain64,discard,no-read-workqueue,no-write-workqueue
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: run storage ← /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 type= cipher=serpent-xts-plain64
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Allocating context for crypt device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Trying to open and read device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 with direct-io.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Initialising device-mapper backend library.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: dm version [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Detected dm-ioctl version 4.47.0.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Device-mapper backend running with UDEV support enabled.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: dm status storage [ opencount noflush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Trying to load any crypt type from device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Crypto backend (OpenSSL 3.0.7 1 Nov 2022 [default]) initialized in cryptsetup library version 2.5.0.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Detected kernel Linux 6.0.10 x86_64.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Loading LUKS2 header (repair disabled).
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Acquiring read lock for device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Locking directory /run/cryptsetup will be created with default compiled-in permissions.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Opening lock resource file /run/cryptsetup/L_259:0
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Verifying lock handle for /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 READ lock taken.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Trying to read primary LUKS2 header at offset 0x0.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Opening locked device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Verifying locked device handle (bdev)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: LUKS2 header version 2 of size 16384 bytes, checksum sha256.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Checksum:c20a6837e55bd871b7b73f776c0401fdeb2d9c831bd0c07bb47332e5ea529018 (on-disk)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Checksum:c20a6837e55bd871b7b73f776c0401fdeb2d9c831bd0c07bb47332e5ea529018 (in-memory)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Trying to read secondary LUKS2 header at offset 0x4000.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Reusing open ro fd on device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: LUKS2 header version 2 of size 16384 bytes, checksum sha256.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Checksum:f894490c5d825194d72c927fc87b394e0207599687db63b8ee6957a0406094a3 (on-disk)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Checksum:f894490c5d825194d72c927fc87b394e0207599687db63b8ee6957a0406094a3 (in-memory)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Device size 2000398934016, offset 16777216.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 READ lock released.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Activating volume storage using token (any type) -1.
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: dm status storage [ opencount noflush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[247]: Token activation unsuccessful for device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08: No such file or directory
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Added key to kernel keyring as 687670047.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Set cipher serpent, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Activating volume storage [keyslot -1] using passphrase.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: dm status storage [ opencount noflush ] [16384] (*1)
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Keyslot 0 priority 1 != 2 (required), skipped.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Trying to open LUKS2 keyslot 0.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Taking global memory-hard access serialization lock.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Acquiring blocking write lock for resource memory-hard-access.
Dec 15 20:28:25 localhost systemd-cryptsetup[247]: Opening lock resource file /run/cryptsetup/LN_memory-hard-access
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: Verifying lock handle for memory-hard-access.
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: Lock handle verification failed.
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: Opening lock resource file /run/cryptsetup/LN_memory-hard-access
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: Verifying lock handle for memory-hard-access.
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: WRITE lock for resource memory-hard-access taken.
Dec 15 20:28:26 localhost systemd-cryptsetup[247]: Running keyslot key derivation.
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@storage.service: Reinstalled deserialized job systemd-cryptsetup@storage.service/start as 33
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@storage.service: Changed dead -> start
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Unlocking WRITE lock for resource memory-hard-access.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Reading keyslot area [0x8000].
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Acquiring read lock for device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Opening lock resource file /run/cryptsetup/L_259:0
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Verifying lock handle for /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 READ lock taken.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Reusing open ro fd on device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 READ lock released.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Keyslot 0 (luks2) open failed with -22.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Keyslot open failed.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Failed to activate with specified passphrase: Invalid argument
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Releasing crypt device /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08 context.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Releasing device-mapper backend.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Closing read only fd for /dev/disk/by-uuid/dd17e735-fac4-467f-b1ee-8bb214bc2b08.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Child 247 belongs to systemd-cryptsetup@storage.service.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Failed with result 'exit-code'.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Service will not restart (restart setting)
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Changed start -> failed
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Job 33 systemd-cryptsetup@storage.service/start finished, result=failed
Dec 15 20:28:28 localhost systemd[1]: Failed to start Cryptography Setup for storage.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Unit entered failed state.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Consumed 6.104s CPU time.
Dec 15 20:28:28 localhost systemd[1]: systemd-cryptsetup@storage.service: Control group is empty.
The working drive:
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@root.service: Will spawn child (service_enter_start): /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@root.service: Passing 0 fds to service
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@root.service: About to execute /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup attach root /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 - discard,no-read-workqueue,no-write-workqueue
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@root.service: Forked /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup as 251
Dec 15 20:28:12 localhost systemd[1]: systemd-cryptsetup@root.service: Changed dead -> start
Dec 15 20:28:12 localhost systemd[1]: Starting Cryptography Setup for root...
Dec 15 20:28:12 localhost systemd[251]: systemd-cryptsetup@root.service: Executing: /nix/store/8ac6kybdmbla694ikm5bihbj9v116rv2-systemd-stage-1-251.7/lib/systemd/systemd-cryptsetup attach root /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 - discard,no-read-workqueue,no-write-workqueue
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: run root ← /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 type= cipher=
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Allocating context for crypt device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Trying to open and read device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 with direct-io.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Initialising device-mapper backend library.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: dm version [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Detected dm-ioctl version 4.47.0.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Device-mapper backend running with UDEV support enabled.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: dm status root [ opencount noflush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Trying to load any crypt type from device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Crypto backend (OpenSSL 3.0.7 1 Nov 2022 [default]) initialized in cryptsetup library version 2.5.0.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Detected kernel Linux 6.0.10 x86_64.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Loading LUKS2 header (repair disabled).
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Acquiring read lock for device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Opening lock resource file /run/cryptsetup/L_259:3
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Verifying lock handle for /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 READ lock taken.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Trying to read primary LUKS2 header at offset 0x0.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Opening locked device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Verifying locked device handle (bdev)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: LUKS2 header version 2 of size 16384 bytes, checksum sha256.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Checksum:c3926ea265a6f61692f645cd07aff329faa16af969083f5dbd0b0f834da60286 (on-disk)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Checksum:c3926ea265a6f61692f645cd07aff329faa16af969083f5dbd0b0f834da60286 (in-memory)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Trying to read secondary LUKS2 header at offset 0x4000.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Reusing open ro fd on device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: LUKS2 header version 2 of size 16384 bytes, checksum sha256.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Checksum:21385b12052395b849cfb5fd84cf9ca7865baeecae3183365a6c7b30377ec3ed (on-disk)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Checksum:21385b12052395b849cfb5fd84cf9ca7865baeecae3183365a6c7b30377ec3ed (in-memory)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Device size 999665917952, offset 16777216.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 READ lock released.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Activating volume root using token (any type) -1.
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: dm status root [ opencount noflush ] [16384] (*1)
Dec 15 20:28:12 localhost systemd-cryptsetup[251]: Token activation unsuccessful for device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7: No such file or directory
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Set cipher serpent, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Activating volume root [keyslot -1] using passphrase.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: dm status root [ opencount noflush ] [16384] (*1)
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Keyslot 0 priority 1 != 2 (required), skipped.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Trying to open LUKS2 keyslot 0.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Taking global memory-hard access serialization lock.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Acquiring blocking write lock for resource memory-hard-access.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Opening lock resource file /run/cryptsetup/LN_memory-hard-access
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Verifying lock handle for memory-hard-access.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: WRITE lock for resource memory-hard-access taken.
Dec 15 20:28:25 localhost systemd-cryptsetup[251]: Running keyslot key derivation.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Unlocking WRITE lock for resource memory-hard-access.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Reading keyslot area [0x8000].
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Acquiring read lock for device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Opening lock resource file /run/cryptsetup/L_259:3
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Verifying lock handle for /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 READ lock taken.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Reusing open ro fd on device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 READ lock released.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Verifying key from keyslot 0, digest 0.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm target-version crypt [ opencount flush ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Detected dm-crypt version 1.24.0.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Loading key (64 bytes, type logon) in thread keyring.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm versions [ opencount flush ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm status root [ opencount noflush ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Calculated device size is 1952439728 sectors (RW), offset 32768.
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: DM-UUID is CRYPT-LUKS2-3c0d48f6f05143289919677a7fcddae7-root
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) created
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) incremented to 1
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) incremented to 2
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) assigned to CREATE task(0) with flags DISABLE_LIBRARY_FALLBACK (0x20)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm create root CRYPT-LUKS2-3c0d48f6f05143289919677a7fcddae7-root [ opencount flush ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm reload (254:0) [ opencount flush securedata ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: dm resume root [ opencount flush securedata ] [16384] (*1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: root: Stacking NODE_ADD (254,0) 0:0 0600 [trust_udev]
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: root: Stacking NODE_READ_AHEAD 256 (flags=1)
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) decremented to 1
Dec 15 20:28:26 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) waiting for zero
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: Udev cookie 0xd4d59de (semid 0) destroyed
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: root: Skipping NODE_ADD (254,0) 0:0 0600 [trust_udev]
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: root: Processing NODE_READ_AHEAD 256 (flags=1)
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: root (254:0): read ahead is 256
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: root: retaining kernel read ahead of 256 (requested 256)
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: Releasing crypt device /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7 context.
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: Releasing device-mapper backend.
Dec 15 20:28:27 localhost systemd-cryptsetup[251]: Closing read only fd for /dev/disk/by-uuid/3c0d48f6-f051-4328-9919-677a7fcddae7.
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Child 251 belongs to systemd-cryptsetup@root.service.
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Main process exited, code=exited, status=0/SUCCESS (success)
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Changed start -> exited
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Job 30 systemd-cryptsetup@root.service/start finished, result=done
Dec 15 20:28:27 localhost systemd[1]: Finished Cryptography Setup for root.
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Control group is empty.
Dec 15 20:28:27 localhost systemd[1]: systemd-cryptsetup@root.service: Changed dead -> exited
Dec 15 20:28:30 yui systemd[1]: systemd-cryptsetup@root.service: Installed new job systemd-cryptsetup@root.service/stop as 251
I have two take-aways from this:
- I believe the first drive that is attempted to be decrypted fails to decrypt, the second one succeeds. This is because the logs for
storage
mention opening temporary files, while the ones for root
don’t.
- This is the actual error output, nothing else seems significantly different between the drives:
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Keyslot 0 (luks2) open failed with -22.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Keyslot open failed.
Dec 15 20:28:28 localhost systemd-cryptsetup[247]: Failed to activate with specified passphrase: Invalid argument
I’ve also since reproduced this on my second system, after I finally got around to trying to upgrade that. That one is royally borked since it only has one drive.
I’ve tried manually loading the modules required for serpent-xts-plain64
after stumbling across someone with a similar error message on a random Russian forum solving it that way, and since confirmed they are included in boot.initrd.luks.cryptoModules by default.
I’ve also manually force-set the crypttab cipher, with no success.
I remain completely stumped. Does anyone have any idea what changed? Both my desktop systems can suddenly no longer boot with 22.11.