GRUB MirroredBoot Volumes Think They're Encrypted

Help
1

After days of struggle, I’ve got NixOS installing on a seemingly redundant disk setup from a combination of bash scripts and .nix files but am having an issue with GRUB /boot/efiX folder thinking it’s encrypted/

In this Partition Creation Script I’m

  • Using mdadm to create a RAID10 out of my four nvme drives
  • Creating a luks volume on that array and formatting it with ext4 fs
  • Mounting that encrypted luks partition to root / ← this might be part of the problem
  • Creating and mounting /boot/efiX partition on each of those drives for redundant bootloader

Then in my hardware-configuration.nix I’m

  • Setting up the four redundant /boot/efiX as grub boot mirrors
  • Mounting the luks ext4 partition as root /

Technically, this installs and boots, however there’s an error at the end of the process which reappears inside the booted OS which is preventing me from doing a nix-rebuild switch.

The error in the nix rebuild is like so

building Nix...
building the system configuration...
updating GRUB 2 menu...
updating GRUB 2 menu...
updating GRUB 2 menu...
updating GRUB 2 menu...
updating GRUB 2 menu...
installing the GRUB 2 boot loader into /boot/efi0...
Installing for x86_64-efi platform.
/nix/store/k2cwk22w6v7fhmmac0ifk4si3cbn18gj-grub-2.12/sbin/grub-install: error: attempt to install to encrypted disk without cryptodisk enabled. Set `GRUB_ENABLE_CRYPTODISK=y' in file `/nix/store/k2cwk22w6v7fhmmac0ifk4si3cbn18gj-grub-2.12/etc/default/grub'.
/nix/store/ilc3scjv4fbq33pxi26n8d585wvssgzm-install-grub.pl: installation of GRUB EFI into /boot/efi0 failed: No such file or directory
Failed to install bootloader

It seems to take issue with the /boot/efiX partitions being mounted in what it thinks is an encrypted space, and so grub wants me to set enableCryptodisk=true;.

I can’t do that however because

  • That /boot/efiX mount isn’t actually encrypted (to my knowledge?)
  • According to This other thread grub encryption and luks encryption aren’t particularly compatible
  • I don’t want an encrypted bootloader, that seems to add complexity without benefit

Any idea how I can essentially coerce grub into thinking that /boot/efiX are not encrypted directories?

Contents of those boot/efiX directories, despite the warnings, show bootloader was installed…
Really I just need to suppress the warning

/boot/efi0:
total 2392
drwxr-xr-x 5 root root    4096 Dec 31  1969 .
drwxr-xr-x 7 root root    4096 Dec 18 18:48 ..
-rwxr-xr-x 1 root root    9058 Dec 18 18:48 background.png
-rwxr-xr-x 1 root root 2413132 Dec 18 18:48 converted-font.pf2
drwxr-xr-x 3 root root    4096 Dec 18 18:17 EFI
drwxr-xr-x 5 root root    4096 Dec 18 18:48 grub
drwxr-xr-x 2 root root    4096 Dec 18 18:32 kernels

Extra context, lsblk output from within booted NixOS

NAME           MAJ:MIN RM  SIZE RO TYPE   MOUNTPOINTS
nvme3n1        259:0    0  3.7T  0 disk   
├─nvme3n1p1    259:4    0  512M  0 part   /boot/efi3
└─nvme3n1p2    259:5    0  3.7T  0 part   
  └─md0          9:0    0  7.5T  0 raid10 
    └─luksraid 254:0    0  7.5T  0 crypt  /nix/store
                                          /
nvme2n1        259:1    0  3.7T  0 disk   
├─nvme2n1p1    259:7    0  512M  0 part   /boot/efi2
└─nvme2n1p2    259:8    0  3.7T  0 part   
  └─md0          9:0    0  7.5T  0 raid10 
    └─luksraid 254:0    0  7.5T  0 crypt  /nix/store
                                          /
nvme1n1        259:2    0  3.7T  0 disk   
├─nvme1n1p1    259:6    0  512M  0 part   /boot/efi1
└─nvme1n1p2    259:9    0  3.7T  0 part   
  └─md0          9:0    0  7.5T  0 raid10 
    └─luksraid 254:0    0  7.5T  0 crypt  /nix/store
                                          /
nvme0n1        259:3    0  3.7T  0 disk   
├─nvme0n1p1    259:10   0  512M  0 part   /boot/efi0
└─nvme0n1p2    259:11   0  3.7T  0 part   
  └─md0          9:0    0  7.5T  0 raid10 
    └─luksraid 254:0    0  7.5T  0 crypt  /nix/store
                                          /
2

From what I can tell it is due to /boot being on the encrypted luks volume. Grub will attempt to install files to /boot as well as whichever EFI parition(s) are specified so because /boot is on the encrypted volume you get that error message.

I have similar mirrored boot setup, using disko, in which I mount the EFI partition(s) at /bootX (and bind mount one of them to /boot) instead of like you do at /boot/efiX. This keeps all the files needed for booting outside of the encrypted volume but also requires more space on the EFI partitions to hold files needed for all NixOS generations.

Or, keeping the /boot/efiX arrangement you already have, you may be able to set boot.loader.grub.enableCryptodisk to true and have it work.

1 Like
3

I think what you’re supposed to do is ignore the high level boot.loader.efi.efiSysMountpoint option and instead set that in mirroredBoots. You also shouldn’t have to set devices at all because that’s for the MBR boot loader.

boot.loader.grub.mirroredBoots = [
  {
    path = "/boot1";
    efiSysMountPoint = "/boot1";
  }
  {
    path = "/boot2";
    efiSysMountPoint = "/boot2";
  }
  {
    path = "/boot3";
    efiSysMountPoint = "/boot3";
  }
  {
    path = "/boot4";
    efiSysMountPoint = "/boot4";
  }
];
1 Like
4

Thankyou! Got it working with a combination of both your feedback.

Key notes:

  • Putting the boot partitions at /bootX indeed takes them out of the encrypted root FS
  • Bind mounting one of them to /boot was required to satisfy grubs desire for that default
  • “devices” is required in the grub mirroredBoots definition (I tried to remove it but it threw errors)
  • enableCryptoDisk=true does not work, it lands in a grub rescue shell. See previously linked post. TLDR, LUKS & GRUB encryption don’t play nice together

For record keeping

Partition Creation Script

#### CREATE PARTITIONS ON BLANK DISKS ####
for drive in /dev/nvme{0..3}n1; do
		
	# Create GPT partition tables
	parted -s "$drive" mklabel gpt
	
	# Create 2GB EFI partition
	parted -s "$drive" mkpart EFI fat32 1MiB 2049MiB
	parted -s "$drive" set 1 boot on

 	# Create Primary partition on remainder of disk
	parted -s "$drive" mkpart primary 2049MiB 100%

 	# Write a FAT32 FS to the EFI partition
	mkfs.fat -F 32 "$drive"p1
 
done

#### CREATE LINUX MD RAID 10 ARRAY ####
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/nvme{0..3}n1p2
echo "Waiting for raid array to initialize"
while [ "$(cat /proc/mdstat | grep -c "resync = ")" -eq 0 ]; do
	sleep 3
done
echo "Array initialized"

#### PRINT RESULTS TO SCREEN ####
lsblk
cat /proc/mdstat

#### CREATE ENCRYPTED LUKS VOLUME ####
cryptsetup --verbose --verify-passphrase luksFormat /dev/md0
cryptsetup luksOpen /dev/md0 luksraid

#### PUT FILESYSTEM ON LUKS AND MOUNT IT TO MNT ROOT ####
mkfs.ext4 /dev/mapper/luksraid
mkdir /mnt
mount /dev/mapper/luksraid /mnt

#### MOUNT BOOT PARTITIONS ####
for i in {0..3}; do
	mkdir -p /mnt/boot$((i+1))
 	mount /dev/nvme"$i"n1p1 /mnt/boot$((i+1))
done

#### BIND MOUNT FIRST BOOT PARTITION TO DEFAULT /BOOT ####
mkdir /mnt/boot
mount --bind /mnt/boot1 /mnt/boot

#### PRINT MOUNTS TO SCREEN ####
df -h | grep /mnt

hardware-config.nix

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
  ];

  boot.loader = {
    grub = {
      enable = true;
      device = "nodev";
      efiSupport = true;
      mirroredBoots = [
        {
          path = "/boot1";
          devices = ["/dev/nvme0n1p1"];
        }
        {
          path = "/boot2";
          devices = ["/dev/nvme1n1p1"];
        }
        {
          path = "/boot3";
          devices = ["/dev/nvme2n1p1"];
        }
        {
          path = "/boot4";
          devices = ["/dev/nvme3n1p1"];
        }
      ];
    };
    efi = {
      canTouchEfiVariables = true;
    };
  };

  # Setup RAID
  boot.swraid = {
    enable = true;
    mdadmConf = ''
      MAILADDR nixosconfignotificat.flaccid440@passmail.net
      DEVICE /dev/nvme0n1p2 /dev/nvme1n1p2 /dev/nvme2n1p2 /dev/nvme3n1p2
      ARRAY /dev/md0 metadata=1.2 UUID=ARRAYUUID
    '';
  };

  # define encrypted root filesystem on linux md raid array
  fileSystems."/" = {
    device = "/dev/mapper/luksraid";
    fsType = "ext4";
  };

  # define redundant boot partitions
  fileSystems."/boot1" = {
    device = "/dev/nvme0n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot2" = {
    device = "/dev/nvme1n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot3" = {
    device = "/dev/nvme2n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot4" = {
    device = "/dev/nvme3n1p1";
    fsType = "vfat";
  };

  # bind mount one to /boot where grub expects there must be a folder
  fileSystems."/boot" = {
    depends = [
      "/boot1"
    ];
    device = "/boot1";
    fsType = "vfat";
    options = [
      "bind"
    ];
  };

  # Ensure necessary kernel modules are available in initrd
  boot.initrd = {
    kernelModules = [
    ];
    availableKernelModules = [
      "dm-mod"
      "xhci_pci"
      "ahci"
      "usbhid"
      "usb_storage"
      "sd_mod"
      "nvme"
      "md"
      "raid10"
      "md-mod"
    ];
    luks.devices = {
      "luksraid" = {
        device = "/dev/disk/by-id/md-uuid-ARRAYUUID";
        preLVM = false; # If LUKS is on top of LVM, set this to true
        allowDiscards = true; # Optional, enables TRIM if supported by your SSD
      };
    };
  };

  swapDevices = [];
  boot.kernelModules = ["kvm-intel"];
  boot.kernelParams = ["boot.shell_on_fail"];
  boot.extraModulePackages = [];
  networking.useDHCP = lib.mkDefault true;
  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;

  # #### NVIDIA CONFIG ####
  hardware.graphics.enable = true; # Enable OpenGL
  services.xserver.videoDrivers = ["nvidia"]; # Nvidia graphics driver
  hardware.nvidia-container-toolkit.enable = true; # Nvidia CDI support for docker/podman
  hardware.nvidia = {
    modesetting.enable = true;
    powerManagement.enable = false;
    powerManagement.finegrained = false;
    open = false;
    nvidiaSettings = true;
  };
}
2 Likes
5

Personally I really don’t like that /boot bind mount. I’m pretty sure you could just do what I said and delete the devices stuff and just use the efiSysMountPoint in the mirroredBoot entries.

Hm, tested my way. Doesn’t seem to work like I thought it should. Will investigate a little more…

6

Ok this worked as expected:

  boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.grub = {
    enable = true;
    efiSupport = true;
    mirroredBoots = [
      {
        path = "/boot1";
        efiSysMountPoint = "/boot1";
        devices = [ "nodev" ];
      }
      {
        path = "/boot2";
        efiSysMountPoint = "/boot2";
        devices = [ "nodev" ];
      }
    ];
  };

No need for toplevel boot.loader.efi.efiSysMountPoint, toplevel boot.loader.grub.device, or for a /boot bind mount.

3 Likes
7

Nice! Just tried this with a fresh install and it works great.
Side-benefit, my UEFI “bios” boot step now shows all four bootloader options :slight_smile:

Updated partitioning script

#### CREATE PARTITIONS ON BLANK DISKS ####
for drive in /dev/nvme{0..3}n1; do
		
	# Create GPT partition tables
	parted -s "$drive" mklabel gpt
	
	# Create 2GB EFI partition
	parted -s "$drive" mkpart EFI fat32 1MiB 2049MiB
	parted -s "$drive" set 1 boot on

 	# Create Primary partition on remainder of disk
	parted -s "$drive" mkpart primary 2049MiB 100%

 	# Write a FAT32 FS to the EFI partition
	mkfs.fat -F 32 "$drive"p1
 
done

#### CREATE LINUX MD RAID 10 ARRAY ####
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/nvme{0..3}n1p2
echo "Waiting for raid array to initialize"
while [ "$(cat /proc/mdstat | grep -c "resync = ")" -eq 0 ]; do
	sleep 3
done
echo "Array initialized"

#### PRINT RESULTS TO SCREEN ####
lsblk
cat /proc/mdstat

#### CREATE ENCRYPTED LUKS VOLUME ####
cryptsetup --verbose --verify-passphrase luksFormat /dev/md0
cryptsetup luksOpen /dev/md0 luksraid

#### PUT FILESYSTEM ON LUKS AND MOUNT IT TO MNT ROOT ####
mkfs.ext4 /dev/mapper/luksraid
mkdir /mnt
mount /dev/mapper/luksraid /mnt

#### MOUNT BOOT PARTITIONS ####
for i in {0..3}; do
	mkdir -p /mnt/boot$((i+1))
 	mount /dev/nvme"$i"n1p1 /mnt/boot$((i+1))
done

#### PRINT MOUNTS TO SCREEN ####
df -h | grep /mnt

Updated hardware-configuration.nix

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
  ];

  boot.loader = {
    efi.canTouchEfiVariables = true;
    grub = {
      enable = true;
      efiSupport = true;
      mirroredBoots = [
        {
          path = "/boot1";
          efiSysMountPoint = "/boot1";
          devices = ["nodev"];
        }
        {
          path = "/boot2";
          efiSysMountPoint = "/boot2";
          devices = ["nodev"];
        }
        {
          path = "/boot3";
          efiSysMountPoint = "/boot3";
          devices = ["nodev"];
        }
        {
          path = "/boot4";
          efiSysMountPoint = "/boot4";
          devices = ["nodev"];
        }
      ];
    };
  };

  # Setup RAID
  boot.swraid = {
    enable = true;
    mdadmConf = ''
      MAILADDR nixosconfignotificat.flaccid440@passmail.net
      DEVICE /dev/nvme0n1p2 /dev/nvme1n1p2 /dev/nvme2n1p2 /dev/nvme3n1p2
      ARRAY /dev/md0 metadata=1.2 UUID=ARRAYUUID
    '';
  };

  # define encrypted root filesystem on linux md raid array
  fileSystems."/" = {
    device = "/dev/mapper/luksraid";
    fsType = "ext4";
  };

  # define redundant boot partitions
  fileSystems."/boot1" = {
    device = "/dev/nvme0n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot2" = {
    device = "/dev/nvme1n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot3" = {
    device = "/dev/nvme2n1p1";
    fsType = "vfat";
  };
  fileSystems."/boot4" = {
    device = "/dev/nvme3n1p1";
    fsType = "vfat";
  };

  # Ensure necessary kernel modules are available in initrd
  boot.initrd = {
    kernelModules = [
    ];
    availableKernelModules = [
      "dm-mod"
      "xhci_pci"
      "ahci"
      "usbhid"
      "usb_storage"
      "sd_mod"
      "nvme"
      "md"
      "raid10"
      "md-mod"
    ];
    luks.devices = {
      "luksraid" = {
        device = "/dev/disk/by-id/md-uuid-ARRAYUUID";
        preLVM = false; # If LUKS is on top of LVM, set this to true
        allowDiscards = true; # Optional, enables TRIM if supported by your SSD
      };
    };
  };

  swapDevices = [];
  boot.kernelModules = ["kvm-intel"];
  boot.kernelParams = ["boot.shell_on_fail"];
  boot.extraModulePackages = [];
  networking.useDHCP = lib.mkDefault true;
  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;

  # #### NVIDIA CONFIG ####
  hardware.graphics.enable = true; # Enable OpenGL
  services.xserver.videoDrivers = ["nvidia"]; # Nvidia graphics driver
  hardware.nvidia-container-toolkit.enable = true; # Nvidia CDI support for docker/podman
  hardware.nvidia = {
    modesetting.enable = true;
    powerManagement.enable = false;
    powerManagement.finegrained = false;
    open = false;
    nvidiaSettings = true;
  };
}
2 Likes