Root on LVM on LUKS no key available with this passphrase after moving from nixos rebuild to flake

Help
1

I’ve been playing around with NixOS and LUKS for the first time in my life during the past one and a half week with pretty good and enjoyable results.

I have set up my system based on mostly the following guide:

I.e. used a EFI partition on a GPT disk as boot and encrypted another partition on the same disk with LUKS, created pv, vg and lv for swap and root on top of it rebuilded nixos the standard way.

I have used systemd and useXkdConfig = true option in the console, not sure about earlySetup though and believed that during stage one the passphrase I am entering is being encoded according to my colemak keyboard layout which I have used during setup to enter and confirm the passphrase.

I’ve done many configuration changes and created 35 generations of nixos which all worked pretty well and at any reboot I was always capable to decrypt the disk with the same passphrase entered.

Then I’ve moved to the unstable channel and believe still was capable to successfully reboot and decrypt, but when trying out configuring with flakes and not absolutely sure to have goten it right a few generations reverted to the stable nixos channel. I’ve did some adjustments on XkbConfig (was using Wayland), tried to make ctrl+alt+del kill which did not work, and remapped caps lock to backspace. Then my computer froze up and I believe that some gnome apps even before that displayed garbled text on choice buttons in dialogs or elsewhere and after a reboot, no matter what I’ve tried I was not capable to get my passphrase accepted. After weeks of usage without any issues suddenly I’ve became unsure about whether it has one trailing character at its end more or less, but otherwise I am absolutely sure that I got it right.

But nevertheless I get “no key available with this passphrase” whether I boot through EFI/systemd on my disk or installation usb.

sudo cryptsetup luksDump /dev/sdb4

LUKS header information
Version: 2
Epoch: 3
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: d97bc8fa-7dbe-4825-8f79-6d2abf8ad979
Label: crypted
Subsystem: (no subsystem)
Flags: (no flags)

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

Keyslots:
0: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2id
Time cost: 14
Memory: 1048576
Threads: 4
Salt: d8 81 fe 0d 4e 83 0a b1 71 2b 4f 7e 80 e0 d2 7e
ba 48 34 2a 41 f4 bf 17 b1 af f8 7b 18 93 ec 91
AF stripes: 4000
AF hash: sha256
Area offset:32768 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
Digests:
0: pbkdf2
Hash: sha256
Iterations: 428339
Salt: 47 8a 4e 36 7c ce 5d d2 8a d0 ce 7b df 25 af a2
e7 9c de e9 6f 48 4c d5 17 d1 bf a7 7a 89 e1 b5
Digest: 04 e9 29 ae b4 11 07 b4 55 30 41 5b 51 12 ea 11
89 83 57 33 9e 37 9b e8 e0 3b 48 78 38 e0 09 8e

As far as I can tell all seems to be fine. I have a Samsung SDD which Samsung Magician is not capable to analyse for errors more thoroughly, but at least SMART doesnt show anything out of order. I had no issues with it on Windows 11 Pro 64-bit which is installed on its other half.

It does not seem very likely to me that I would have suffered corruption exactly in the LUKS header in such a way that it would not be perceivable.

I thought I had the header and keys backed up on another unencrypted filesystem, but no matter how hard I’ve tried I could not find it anywhere.

Right now I am left with whatever is available in the EFI boot:

/mnt/boot/EFI/Boot
/mnt/boot/EFI/Boot/bootx64.efi
/mnt/boot/EFI/Boot/fbx64.efi
/mnt/boot/EFI/Boot/mmx64.efi
/mnt/boot/EFI/ubuntu
/mnt/boot/EFI/ubuntu/grubx64.efi
/mnt/boot/EFI/ubuntu/shimx64.efi
/mnt/boot/EFI/ubuntu/mmx64.efi
/mnt/boot/EFI/ubuntu/BOOTX64.CSV
/mnt/boot/EFI/ubuntu/grub.cfg
/mnt/boot/EFI/systemd
/mnt/boot/EFI/systemd/systemd-bootx64.efi
/mnt/boot/EFI/Linux
/mnt/boot/EFI/nixos
/mnt/boot/EFI/nixos/02s7x0gv18z968bzjz7nk38a2lgc20l0-linux-6.6.63-bzImage.efi
/mnt/boot/EFI/nixos/h74kbq0zmzhcfy0qv13d89nszayb94z2-initrd-linux-6.6.63-initrd.efi
/mnt/boot/EFI/nixos/.extra-files
/mnt/boot/EFI/nixos/yhdx84xjcswsf7d2ckm251b3v9hizlw9-initrd-linux-6.6.63-initrd.efi
/mnt/boot/EFI/nixos/dspqjs81r9m74805l5fzmp4b9hdib0yd-initrd-linux-6.6.63-initrd.efi
/mnt/boot/EFI/nixos/3dbihkrqfdqsxa5s9pmqxx8slx2rx61x-initrd-linux-6.6.63-initrd.efi
/mnt/boot/EFI/nixos/5zm7p5kyqvzn4ib6px23wl8n73g8v8yg-linux-6.6.64-bzImage.efi
/mnt/boot/EFI/nixos/k5jxd6s7cg3da7kb8aik44cjlbbz37xf-initrd-linux-6.6.64-initrd.efi
/mnt/boot/EFI/nixos/jzl52vx9j42jgn92nynihpniamzwd31p-linux-6.6.64-bzImage.efi
/mnt/boot/EFI/nixos/nxnw0xfh6600yl7vnx6n143j4ny71c4k-initrd-linux-6.6.64-initrd.efi
/mnt/boot/EFI/nixos/977517whn6knicf17cznvrbh57a55iyx-linux-6.6.66-bzImage.efi
/mnt/boot/EFI/nixos/8kkww0a0n8hkzbd6amnrh8y2r272v6bh-initrd-linux-6.6.66-initrd.efi
/mnt/boot/EFI/nixos/rhp6qdkyvqrmpcbmdrbnqqbx6dqaj97v-initrd-linux-6.6.66-initrd.efi
/mnt/boot/System Volume Information
/mnt/boot/loader
/mnt/boot/loader/entries
/mnt/boot/loader/entries/nixos-generation-1.conf
/mnt/boot/loader/entries/nixos-generation-2.conf
/mnt/boot/loader/entries/nixos-generation-3.conf
/mnt/boot/loader/entries/nixos-generation-4.conf
/mnt/boot/loader/entries/nixos-generation-5.conf
/mnt/boot/loader/entries/nixos-generation-6.conf
/mnt/boot/loader/entries/nixos-generation-7.conf
/mnt/boot/loader/entries/nixos-generation-8.conf
/mnt/boot/loader/entries/nixos-generation-9.conf
/mnt/boot/loader/entries/nixos-generation-10.conf
/mnt/boot/loader/entries/nixos-generation-11.conf
/mnt/boot/loader/entries/nixos-generation-12.conf
/mnt/boot/loader/entries/nixos-generation-13.conf
/mnt/boot/loader/entries/nixos-generation-14.conf
/mnt/boot/loader/entries/nixos-generation-15.conf
/mnt/boot/loader/entries/nixos-generation-16.conf
/mnt/boot/loader/entries/nixos-generation-17.conf
/mnt/boot/loader/entries/nixos-generation-18.conf
/mnt/boot/loader/entries/nixos-generation-19.conf
/mnt/boot/loader/entries/nixos-generation-20.conf
/mnt/boot/loader/entries/nixos-generation-21.conf
/mnt/boot/loader/entries/nixos-generation-22.conf
/mnt/boot/loader/entries/nixos-generation-23.conf
/mnt/boot/loader/entries/nixos-generation-24.conf
/mnt/boot/loader/entries/nixos-generation-25.conf
/mnt/boot/loader/entries/nixos-generation-26.conf
/mnt/boot/loader/entries/nixos-generation-27.conf
/mnt/boot/loader/entries/nixos-generation-28.conf
/mnt/boot/loader/entries/nixos-generation-29.conf
/mnt/boot/loader/entries/nixos-generation-30.conf
/mnt/boot/loader/entries/nixos-generation-31.conf
/mnt/boot/loader/entries/nixos-generation-32.conf
/mnt/boot/loader/entries/nixos-generation-33.conf
/mnt/boot/loader/entries/nixos-generation-34.conf
/mnt/boot/loader/entries/nixos-generation-35.conf
/mnt/boot/loader/loader.conf
/mnt/boot/loader/random-seed
/mnt/boot/loader/entries.srel

but otherwise no access to my encrypted root.

I am completely unsure, whether systemd during the first stage is capable to switch the keymap to colemak layout based on the wayland xkbconfig setup. I’ve read complaints that it does not, but if it did not like I believed it would, then I cant explain how it would have been possible for me to decrypt/boot the system so many times by entering the passphrase. But if it does, I am unable to comprehend what could have happened on the flake system rebuilds that could have broken the keyboard mapping on all my generations or luks header corruption or change. SDD corruption does not seem very likely also. I’ve run 5 complete iterations of memtest even with most aggresive memory BIOS settings and no issues were detected. I did not change the LUKS key, digest, algorithms, passphrase, did not touch anything in the header. If there was some corruption then I assume that the header backup somewhere on the same device would got out of sync and LUKS would automatically complain about it but it did not.

I’ve experimented with several hypotheses what could have been wrong and wrote my own and tried out other peoples software to auto transform the passphrase into characters that would be output by a standard US mapping and then tried to enter them with colemak and even tried piping “echo -n passphrase” directly into “cryptsetup open /dev/sdb4 crypted --type luks2” but without any success.

No matter what hypotheses I try to come up with what could have happened they are not consistent with all the conflicting informations about the issue I have.

Any idea what I could do on my side to somehow be able to fix the issue and not loose almost 2 weeks of tuning which I unfortunately was not, due to silly overconfidence in the availability of 100% working bootable previous generations of NixOs and not enough time have not been able to back up yet.

I have even experimented with the Nix system and tried out running garbage collection on it a few hours before the issue appeared. Unsure whether it might have affected anything related.

The system is awesome so far, but unfortunately not as conveniently fool and bullet proof as one might hope it to be.

2

Also tried entering the passphrase after booting into the installer and not changing the default us layout to colemak to emulate initrd boot stage 1 not remapping, but it did not help at all either.

3

Even before I read the part about colemak I thought that it propably has to do with keymapping.

Why would you need software for that? You just try entering the password as if you have a american keyboard. And if that does not work you try entering the password as if you have a colemak keyboard in front of you.

can you show as the specific command of course redact the passphrase

There would be an explanation. You set the password while thinking you are using colemak but werent actually , you were decrypting the drive while thinking you are using colemak but weren’t. And know you actually have to use colemak so the letters have different locations.

You propably tested those hypothesis already, but I am just trying to be sure

4

Simply because the situation got rather fast quite complicated and software is great at providing repeatable, iteratively improveable, debuggable solutions or algorithms for specific or generalizable type of problems/issues. It is easy to fix, adjust rerun and auto test anything realized by software which without it could take tremendous time/effort/frustration levels costs.

Character remapping between what you might thought you were typing, while using a different keymap and emitting different characters might be one of them. Not something that might be reliably, efficiently and simply doable and verifiable without errors or doubt with sufficiently complex passphrases without software with merely some standard and colemak layout pictures and a piece of paper.

My syntactic (raw data) memory is tremendously bad, I am unable to rely on it for anything more complex, therefore for remembering a good, completely new, strong password intended for this purpose and long term use, that would not sabotage security by being the weakest, insufficient entropy providing vulnerability of the whole solution and therefore make use of as many types of character classes available in ASCII as possible and be over 30 characters long, yet I would be capable if not to keep remembered at all times, but at least capable to proceduraly reconstruct from rememberable semantic sub elements and their transformations and composition, and would be with sufficient frequency of use retypable and (re)learnable pretty fast and reliably, I had to rely on my semantic/procedural/algorithmic memory which is rather pretty good to remember a way to reconstruct the passphrase whenever I would need to make use of it.

To be a true security providing token, I dislike the idea of having it anywhere written down (whether on paper or computer script or memory), especially in a plain, untransformed, unobfuscated form but being able to reliable realize the described hypothetized keyboard mapping error fixing transformations upon it and verify/test them to be free of bugs in the end and automatically be capable to test the result with the speed LUKS provides, forced me into breaking all these limitations in some way during the process, if only in a sufficiently confidential and secure environment, and expose myself to significant levels of professional paranoia, yet in the end it did not matter that very much, because anyhow in no way, not even once I was capable to generate the correct passphrase that would be capable of unlocking the LUKS encryption upon my block device.

Even though I have simultaneously manually on paper performed the transformations and cross checked them with outputs generated by scripts and softwae and made the outputs fed into cryptsetup through echo -n piping as well as typed in manually, I have fed LUKS with hundreds of software generated and manually entered passphrases during a day I have devoted my time to this issue, yet not a single one of them has been capable of succeeding at decrypting my block device.

I am not sure what your asking. I have been basing my attempts upon a belief I have not been able yet to verify (but when I think about it just right now, it would be rather simple to do so) that piping echo -n “passphrase_ascii_character_string_exactly_as_typed” into cryptsetup open should do the very same job as entering the same characters into the cryptsetup read input and pressing enter. But because I havent been able to succeed, it did not enlighten me upon the answer of the important question of whether this hypothesis is indeed true.

Well I have to admit I am already after the time I spent thinking and attacking this problem from diverse vectors of doubts/hypotheses completely confused and in doubt about what was working and what was not in my setup, because like stated the informations I’ve gathered about the problem are contradictory. I certainly had colemak layout active during the installation, since I did it manually and was typing in commands into a terminal the whole time. I am not sure whether the installation environment would be so ingenuinely designed to make use of the information about my used colemak layout to derive any transormation function upon the entered passphrase to be working equivalently during a systemd phase 1 reentering if the keyboard map would indeed anyhow differ. I do not find that very probable, although not beyond being imaginable. But if it did not and I was capable to authenticate successfuly anyway that would mean that the sytemd passphrase entry must have successfully made available the colemak mapping and by merely remapping caps lock to backspace in xkbconfig and rebuilding the system via a flake I do not see any reason why my ability to successfully provide the decryption passphrase should suddenly break and even in all the previous generations of NixOs. Otherwise by some means, I am unable to comprehend or remember, the passphrase must have been made equivalent with a character string that a standard us layout would produce when used to enter the passphrase using the corresponding colemak keys to the imagined passphrase being entered. I have explored this hypothesis both on paper and in code and provided to cryptsetup both manually and automatically passphrases that would be created in such a scenario instead through the several input possibilities, yet I was not capable to succeed at all.

I have tested everything many times, entered the passphrase and any possible transformations of it I was capable to imagine might have been created with a setup broken in any way I could have imagined/hypothesized and also fed them through echo -n pipe. But in the end no success at all, it was a rather enjoyable, if though pretty frustrating journey and thought experiment and motivation for increased carefulness in the future and I’ve learned a lot by exploring this subject area, yet it would be really much more beneficial, if I would be able to arrive at some beneficial knowledge somehow that in the ideal case would allow me to decrypt my system and fix its passphrase to become with complete definitiveness whatever I would like it to be and work as expected.

I am certain that there must be errors in some of my assumptions, as I am unable to determine all the variables affecting and creating the problem and tried to simply brute force them by trying all the possibilities they generate. But nevertheless I was not able to succeed and therefore I am pretty curious where I might have made any wrong assumptions which I am not able to comprehend yet so that I might be capable to learn and grow from them.

1 Like
5

Although my description and its details might seem rather confusing, in essence the problem is very well and simply defined and reproducible.

I’ve run the latest iso installer for Gnome NixOs, selected in the installer window the layout to be US/Colemak, left the window open but switched to a terminal window in which I have realised the installation according to the instructions in the provided guide. I had comlemak mapping active in the installer and know the sequence of keyboard keys I’ve pressed to enter and confirm the passphrase. After that for almost 2 weeks I was capable with the same sequence of key presses to decrypt my LUKS volume and boot into the system until I’ve tried to do the described things with the Nix system/flake config I’ve created which unfortunately I havent been able to backup outside of the system it is encrypted in and therefore can not simply provide. I would like to be able to somehow find out what went wrong that made my passphrase stop working, whether it corresponds to some different character string than that I’ve imagined I was entering into LUKS “knowing”/believing having colemak active and therefore being able to change the passphrase and whatever got broken in my setup to be capable to regain access to the encrypted root NixOs system.

6

I’m the primary maintainer of initrd stuff in NixOS and I’m usually eager to figure problems like these out. But with all due respect, your messages are drastically too long and meandering for me to follow and draw meaningful conclusions from. I need you to be a lot more brief and to-the-point for me to provide any help.

2 Likes
7

Sorry for the verbosity and not being able to meet your expectations with my descriptions of the issue, I’ve simply tried to put what I’ve considered important for consideration as succintly down as I was able. If I am needed to provide any further necessary details, answers to any questions or somehow clarify things further, please let me know. Otherwise I am not sure what else I might further provide to improve things beside what I was already capable to.

8

I believe the best way of finding a solution would be to list out all hypothesis in maximal two sentences as bullets. And explain how you challenged each hypothesis individually and separately under each bullet point.