Luks/LVM Boot Issues

Hi, I am sure I have done something dumb but I can’t see it for looking… :man_facepalming:

I want to test some scripts before deploying them, so I ran up a virsh domain with an encrypted disk using luks/lvm. I have done this successfully before (23.11) and referred back to my notes on the process, but having completed the install and rebooted the domain promptly gave me the following warning;

EFI stub: Loaded initrd from Linux_EFI_INITRD_MEDIA_GUI device path
EFI stub: Measured initrd data into PCR 9

Screenshot_2024-10-12_06-01-00

What I have tested;

  • Tested same process/config on 23.11 and 24.05 with the same result
  • Tested same process on 24.05 without the encryption - works fine
  • Tested that I can boot from the ISO afterwards, decrypt the drive and verify that it contains /boot etc
  • Checked the log during install for errors
  • Checked that I can do an update from the media and when it reboots, I do get the systemd boot menu but all options do the same thing. I’m thinking this suggests UEFI is finding /boot and handing over to NixOS ok.
  • Searched for answers and ideas…

The build process is as follows;

[root@nixos:~]# lsblk
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0   7:0    0    1G  1 loop /nix/.ro-store
sr0    11:0    1  1.1G  0 rom  /iso
vda   253:0    0   40G  0 disk 

[root@nixos:~]# wipefs -a /dev/vda

[root@nixos:~]# parted /dev/vda -- mklabel gpt
Information: You may need to update /etc/fstab.

                                                                          
[root@nixos:~]# parted /dev/vda -- mkpart ESP fat32 1MiB 512MiB
parted /dev/vda -- set 1 esp on
parted /dev/vda -- mkpart primary 512MiB 100%
Information: You may need to update /etc/fstab.

Information: You may need to update /etc/fstab.                           

Information: You may need to update /etc/fstab.                           

                                                                          
[root@nixos:~]# cryptsetup luksFormat --type luks2 /dev/vda2

WARNING!
========
This will overwrite data on /dev/vda2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/vda2: 
Verify passphrase: 

[root@nixos:~]# cryptsetup luksOpen /dev/vda2 crypted
Enter passphrase for /dev/vda2: 

[root@nixos:~]# pvcreate /dev/mapper/crypted
vgcreate vg /dev/mapper/crypted
  Physical volume "/dev/mapper/crypted" successfully created.
  Volume group "vg" successfully created

[root@nixos:~]# lvcreate -L 8G -n swap vg
lvcreate -l '100%FREE' -n nixos vg
  Logical volume "swap" created.
  Logical volume "nixos" created.

[root@nixos:~]# mkfs.fat -F 32 -n BOOT /dev/vda1
mkswap -L swap /dev/vg/swap
mkfs.ext4 -L nixos /dev/vg/nixos
mkfs.fat 4.2 (2021-01-31)
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
LABEL=swap, UUID=614a06ba-0478-4d42-b10f-c827ca415176
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 8252416 4k blocks and 2064384 inodes
Filesystem UUID: 516821ee-d7f1-4c56-95cf-6cb51ec0bbbd
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   


[root@nixos:~]# mount /dev/disk/by-label/nixos /mnt
mkdir -p /mnt/boot
mount -o umask=0077 /dev/disk/by-label/BOOT /mnt/boot
sudo swapon /dev/vg/swap

[root@nixos:~]# lsblk --fs
NAME           FSTYPE      FSVER            LABEL                      UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
loop0          squashfs    4.0                                                                                      0   100% /nix/.ro-store
sr0            iso9660     Joliet Extension nixos-minimal-24.05-x86_64 1980-01-01-00-00-00-00                       0   100% /iso
vda                                                                                                                          
├─vda1         vfat        FAT32            BOOT                       4537-0996                                 510M     0% /mnt/boot
└─vda2         crypto_LUKS 2                                           bc6df4c9-5be9-4130-8e85-925fb564af80                  
  └─crypted    LVM2_member LVM2 001                                    0UFbqS-80oy-v6RQ-Q9UL-8kTb-ypa4-OsOsI5                
    ├─vg-swap  swap        1                swap                       614a06ba-0478-4d42-b10f-c827ca415176                  [SWAP]
    └─vg-nixos ext4        1.0              nixos                      516821ee-d7f1-4c56-95cf-6cb51ec0bbbd     29.2G     0% /mnt

[root@nixos:~]# nixos-generate-config --root /mnt
writing /mnt/etc/nixos/hardware-configuration.nix...
writing /mnt/etc/nixos/configuration.nix...
For more hardware-specific settings, see https://github.com/NixOS/nixos-hardware.

[root@nixos:~]# vi /mnt/etc/nixos/configuration.nix
*** replace uuid from vda2 above into config and then paste ***

[root@nixos:~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=99.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=137 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 96.063/111.018/137.387/18.701 ms

[root@nixos:~]# nixos-install

enter root password
reboot

The parts of the config that I add and which are the difference between a working guest and a non-working guest are; (Note lanzaboote is prepped but not setup yet)

{ config, lib, pkgs, ... }:
let
#	sources = import ./nix/sources.nix;
#	lanzaboote = import sources.lanzaboote;
in
{
	boot = {
		initrd = {
			luks = {
				devices = {
					crypted = {
						device = "/dev/disk/by-uuid/bc6df4c9-5be9-4130-8e85-925fb564af80";
						preLVM = true;
						allowDiscards = true;
					};
				};
			};
		};
#		lanzaboote = {
#			enable = true;
#			pkiBundle = "/etc/secureboot";
#		};
		loader = {
			efi = {
				canTouchEfiVariables = true;
			};
			systemd-boot = {
				consoleMode = "auto";
				editor = false;
				enable = true;
#				enable = lib.mkForce false;
			};
			timeout = 1;
		};
	};
	environment = {
		systemPackages = with pkgs; [
			lanzaboote-tool
			niv
			sbctl
		];
	};
	imports = [
		./hardware-configuration.nix
#		lanzaboote.nixosModules.lanzaboote
 	];
}

Any suggestions gratefully received… TIA

This is normal. This output is just the kernel logging a bit of info about the initrd. It’s nothing to worry about.

Usually when you get no output after this point, it means one of two things. Either the kernel is panicking very early, or the console just isn’t getting set up properly. You often need to tweak the console= kernel parameter for VMs for whatever reason.

1 Like

You can also just enable something like SSH when you install the configuration to see if the system is just booting without outputting anything on the console. If you can SSH in, it’s definitely just a console problem.

1 Like

Thanks @ElvishJerricco, you’re absolutely correct, there was nothing wrong at all, simply that I had swapped the console. For some strange reason, perhaps lack of sleep, seeing both the initial output and being able to choose from the boot menu in the virt-viewer console meant I never considered looking at ‘virsh console ’ even though I had used during the install and knew it was configured! I ran the virsh console and typed the password and the machine completed the boot process as expected. I knew I had done something dumb!!! :crazy_face:

Not sure if I can have the best of both worlds and be able to enter the password from either console. I seem to recall it was one or the other. Not important though.

Both 23.11 and 24.05 virsh domains are now running with LUKS/LVM and Secure-Boot (lanzaboote). Thanks again.

1 Like