I’m running into a reproducible issue on the NixOS 25.11 installer ISO (kernel 7.0.5, cryptsetup 2.8.6) when using LUKS2 with binary keyfiles.
Environment
-
NixOS 25.11 minimal installer ISO
-
Kernel: 7.0.5 x86_64
-
cryptsetup: 2.8.6
-
OpenSSL: 3.6.2
-
Target disk: NVMe SSD (
/dev/nvme0n1) -
LUKS2 + ZFS root setup (no data involved yet)
Problem
LUKS unlock fails when using a 2048-byte binary keyfile, but succeeds with a 64-byte keyfile.
The failure is deterministic and reproducible.
Steps to reproduce
1. Create a 2048-byte keyfile
dd if=/dev/urandom of=/root/testkey.bin bs=2048 count=1
chmod 0400 /root/testkey.bin
2. Format LUKS2 device
cryptsetup luksFormat --type luks2 /dev/nvme0n1p2 /root/testkey.bin
3. Attempt to open
cryptsetup open /dev/nvme0n1p2 crypted --key-file /root/testkey.bin
Result:
No key available with this passphrase.
Expected behavior
The device should open successfully, since the same keyfile was used to create the LUKS volume.
Working case (comparison)
A 64-byte keyfile works as expected:
dd if=/dev/urandom of=/root/testkey.bin bs=64 count=1
cryptsetup luksFormat --type luks2 /dev/nvme0n1p2 /root/testkey.bin
cryptsetup open /dev/nvme0n1p2 crypted --key-file /root/testkey.bin
This succeeds consistently.
Debug evidence
LUKS header is valid:
LUKS2 header version 2
Keyslot 0 exists
PBKDF2 / Argon2 OK
cryptsetup trace shows correct file access:
st_size=2048
openat(...) = 9
So the keyfile is read correctly from disk.
Failure occurs during verification:
Digest 0 (pbkdf2) verify failed
No key available with this passphrase.
Important observation
-
64-byte keyfile → works
-
2048-byte keyfile → fails
-
Same system, same partition, same process
This is fully deterministic.
What I have ruled out
-
Disk corruption (luksDump is clean)
-
Keyfile truncation (file is read correctly)
-
RAM issues (behavior is fully deterministic)
-
Wrong device or partition
Hypothesis
This appears to be a cryptsetup keyfile handling inconsistency in this environment where:
-
keyfile size affects how key material is derived or processed
-
resulting in mismatch between luksFormat and luksOpen derivation paths
Possibly related to:
-
keyfile processing path differences in cryptsetup 2.8.x
-
OpenSSL 3.6 backend behavior
-
or implicit key truncation during enrollment vs unlock
Workaround
Using a 64-byte random keyfile works reliably:
dd if=/dev/urandom of=/root/key.bin bs=64 count=1
and using --keyfile-size 64 also appears to stabilize behavior.
Question
Is this expected behavior for cryptsetup 2.8.x, or a regression in the NixOS 25.11 ISO stack?
Has anyone else observed LUKS2 unlock failures with larger binary keyfiles?
And yes, I let some LLM help me write this post, sorry for that.
Best,
Skip