Please Note: I am aware that with this setup if I use systemd-boot it will work, please don’t tell me to do that. I am also aware that with this setup, if I make /boot a separate partition that is not zfs and copy the kernel and initramfs accross, this will work, I am trying to fix/diagnose the actual issue though. Please don’t tell me to do that either, I will fall back to it if I have too, but I am trying to avoid that.
This is similar in functionality to having a single partition install that is ZFS. Which should be possible. I have done it with other OSes. What I think is happening is that the grub install command is not getting the zfs module preloaded to allow for the install process to recognise the filesystem.
Relevant Partition Layout
- ESP is mounted to /boot/efi, and is fat32
I know this oldschool method is not officially supported. I am doing it to solve other issues with the way grub is implimented on NixOS. My setup does actually work when doing this and not using ZFS. This was done as I had to unpick a lot of really borked behaviour from the standard nix options around grub. - /boot is not a separate partition from /.
- / is zfs
The error I am getting:
/nix/store/jw0hq5my8ymgrzx229qylrgq5g7q84fm-grub-2.12/sbin/grub-install: error: unknown filesystem.
/nix/store/b9w7m1g5kbhyrx7k04g6yzpdh9avx2z6-install-grub.pl: installation of GRUB EFI into /boot/efi failed: No such file or directory
I have confirmed the modules were compiled (but not if done successfully as I don’t know how to do that):
[root@nixos:/home/nixos]# ls /mnt/nix/store/jw0hq5my8ymgrzx229qylrgq5g7q84fm-grub-2.12/lib/grub/x86_64-efi/ | grep zfs
zfscrypt.mod
zfscrypt.module
zfsinfo.mod
zfsinfo.module
zfs.mod
zfs.module
Other testing I have done:
- Manual install with the modules forcibly preloaded with:
nixos-enter --root '/mnt'
grub-install --modules='zfs zfsinfo' --boot-directory=/boot --efi-directory=/boot/efi --removable --target=x86_64-efi --bootloader-id=GrUB --force /dev/sda
grub-install: error: unknown filesystem.
- grub-probe from chrooted environment
nixos-enter --root '/mnt'
grub-probe: error: unknown filesystem.
My grub configuration.
Please note, I know the bash scripts are strange as hell, I tried to edit them for readability, but the extraInstallCommands option has some massive limitations in the way executes a shell environment, and this particular eccentric scripting style is relatively bulletproof given how the borked shell environment that extraInstallCommands envokes. Sorry for the readability headache, you probably don’t need to read the horrible parts, but I have included it for thoroughness’ sake.
The reason for the scripts is because the nix options around the efi binary file are not correctly handled/are ignored, and are kind of broken in general… or poorly documented. As a nix noob who is extremely confident messing with grub, I cannot tell which actually tell if it’s a documentation issue or a nix code issue. Those options really suck if you need to do anything like encrypted /boot, hence me writing and testing the hell out of a shell script to replace a lot of the options that aren’t working as I expected given snippets of an explaination that is the documentation. It was easier to subvert these options then spelunk in the source code.
Also, the grub shell script that is embedded into the efi binary file is incorrect for this zfs install, and is still setup for a btrfs setup I use on another machine, but as I haven’t even gotten grub correctly installed, I have left this for future me to correct. Please ignore it, it’s not an oversight.
{ config, lib, pkgs, ... }:
{
# Prerequisite Packages
environment = {
systemPackages = with pkgs; [
efibootmgr
grub2_efi
];
};
#
boot = {
loader = {
systemd-boot.enable = false; # Need to be explicitly stated for some reason I don't understand.
efi = {
canTouchEfiVariables = false;
efiSysMountPoint = "/boot/efi";
};
timeout = 5;
grub = let
bootid = "GrUB";
modules = "zfs part_gpt sleep btrfs file true echo test probe search_label search search_fs_uuid search_fs_file ls regexp cmp";
stub = "${config.boot.loader.efi.efiSysMountPoint}/EFI/BOOT/BOOTX64.EFI";
inherit (lib) escapeShellArg;
in {
# Attempts at zfs support, also see 'modules' parameter above
zfsSupport = true;
forceInstall = true;
extraGrubInstallArgs = [ "--modules=zfs zfsinfo" ];
extraConfig = ''
modprobe zfs
modprobe zfsinfo
'';
enable = true;
device = "nodev";
fsIdentifier = "uuid";
efiSupport = true;
efiInstallAsRemovable = true;
enableCryptodisk = false;
copyKernels = true;
configurationLimit = 50;
# gfxmodeEfi = "1920x1080x32,1920x1080x24,1024x768x32,1024x768x24,auto"; # Screen layout - uefi Needs tailoring to screen size.
extraEntries = ''
menuentry "Reboot" {
reboot
}
menuentry "Shutdown" {
halt
}
menuentry "Boot to UEFI Firmware Menu" {
fwsetup
}
'';
extraInstallCommands = ''
# GRUB SCRIPT FILE OPERATIONS
# IF: grub_efi.cfg script file exists in the location it is embedded into the binary from
[ -f /boot/grub/grub_efi.cfg ] || {
# ELSE: Re/Create the script file
${pkgs.coreutils}/bin/cat << 'EOS' > /boot/grub/grub_efi.cfg
search --label eMMC_btrfs_rootfs --set=root
set prefix=($root)/@/boot/grub
normal
EOS
}
# EFI BINARY FILE OPERATIONS
# IF: Binary file exists, AND Binary file contains the clear text header for the binary segment of each of the modules that should be embedded within.
[ -f /boot/efi/EFI/BOOT/BOOTX64.EFI ] && {
[[ "$(
${pkgs.coreutils}/bin/cat /boot/efi/EFI/BOOT/BOOTX64.EFI | \
${pkgs.gnugrep}/bin/grep -Eo --binary-files=text \
"$(
${pkgs.coreutils}/bin/echo "${modules}" | \
${pkgs.coreutils}/bin/tr -s ' ' | \
${pkgs.gnused}/bin/sed 's/ $//' | \
${pkgs.gnused}/bin/sed 's/ /|/g'\
)" | \
${pkgs.gawk}/bin/awk '!seen[$0]++' | \
${pkgs.coreutils}/bin/sort | \
${pkgs.coreutils}/bin/tr '\n' ' '
)" == "$(
${pkgs.coreutils}/bin/echo "${modules}" | \
${pkgs.coreutils}/bin/tr ' ' '\n' | \
${pkgs.coreutils}/bin/sort | \
${pkgs.coreutils}/bin/tr '\n' ' '
)" ]] || {
# ELSE: Delete binary file prior to recreation
${pkgs.coreutils}/bin/rm -f /boot/efi/EFI/BOOT/BOOTX64.EFI
}
}
# IF: Binary file exists, AND Script file embedded in binary is correct
[ -f /boot/efi/EFI/BOOT/BOOTX64.EFI ] && {
[[ "$(
${pkgs.coreutils}/bin/cat /boot/efi/EFI/BOOT/BOOTX64.EFI | \
${pkgs.gnugrep}/bin/grep -Eo --binary-files=text \
"$(
${pkgs.coreutils}/bin/cat /boot/grub/grub_efi.cfg | \
${pkgs.gnused}/bin/sed '/^[[:space:]]*$/d' | \
${pkgs.coreutils}/bin/tr '\n' '|' | \
${pkgs.gnused}/bin/sed 's/|$//' | \
${pkgs.gnused}/bin/sed 's/[][\^$.?*+(){}]/\&/g'
)" | ${pkgs.coreutils}/bin/wc -l
)" == "$(
${pkgs.coreutils}/bin/cat /boot/grub/grub_efi.cfg | \
${pkgs.gnused}/bin/sed '/^[[:space:]]*$/d' | \
${pkgs.coreutils}/bin/wc -l
)" ]] || {
# ELSE: Delete binary file for recreation
${pkgs.coreutils}/bin/rm -f /boot/efi/EFI/BOOT/BOOTX64.EFI
}
}
# IF: binary file is missing. Create it
[ ! -f /boot/efi/EFI/BOOT/BOOTX64.EFI ] && {
# THEN: Create it
# Create Directory for EFI binary file in directory doesn't already exist
${pkgs.coreutils}/bin/mkdir --parents ${escapeShellArg (builtins.dirOf stub)}
# Generate efi Binary and Embed Custom grub.cfg file and require Modules
${pkgs.grub2_efi}/bin/grub-mkimage \
--prefix "" \
--format 'x86_64-efi' \
--config '/boot/grub/grub_efi.cfg' \
--output ${escapeShellArg stub} \
${modules}
}
# BOOT MENU OPERATIONS
# IF: There's a single entry in the boot menu with the correct UUID, and Filepath, AND there's single entry in the boot menu with the correct bootid
{
[[ "$(
${pkgs.efibootmgr}/bin/efibootmgr | \
${pkgs.gnugrep}/bin/grep "$( \
${pkgs.util-linux}/bin/blkid -s PARTUUID -o value $( \
${pkgs.util-linux}/bin/findmnt -n -o SOURCE /boot/efi \
) \
)" | \
${pkgs.gnugrep}/bin/grep -E 'EFI.boot.bootx64.efi' | \
${pkgs.gnused}/bin/sed -E 's/^Boot([0-9]{4})([[:alnum:]]*).*/\1\2/'
)" == "$(
${pkgs.efibootmgr}/bin/efibootmgr | \
${pkgs.gnugrep}/bin/grep -E "${bootid}" | \
${pkgs.gnused}/bin/sed -E 's/^Boot([0-9]{4})([[:alnum:]]*).*/\1\2/'
)" ]] && [[ "$(
${pkgs.efibootmgr}/bin/efibootmgr | \
${pkgs.gnugrep}/bin/grep -E "${bootid}" | \
${pkgs.coreutils}/bin/wc -l
)" == 1 ]]
} || {
# ELSE: Delete all entries matching only UUID and/or bootid
${pkgs.efibootmgr}/bin/efibootmgr | \
${pkgs.gnugrep}/bin/grep -E "${bootid}|$( \
${pkgs.util-linux}/bin/blkid -s PARTUUID -o value $( \
${pkgs.util-linux}/bin/findmnt -n -o SOURCE /boot/efi \
) \
)" | \
${pkgs.gnused}/bin/sed -E 's/^Boot([0-9]{4})([[:alnum:]]*).*/\1\2/' | \
${pkgs.findutils}/bin/xargs -I {} \
${pkgs.efibootmgr}/bin/efibootmgr -q -B -b {}
}
# Create New UEFI Boot Entry if Needed
# IF: There's a boot menu entry with the bootid
{
${pkgs.efibootmgr}/bin/efibootmgr | \
${pkgs.gnugrep}/bin/grep -qE "${bootid}"
} || {
# ELSE: Create entry
${pkgs.efibootmgr}/bin/efibootmgr -q -c -p 1 -d "$( \
${pkgs.util-linux}/bin/findmnt -n -o SOURCE /mnt/boot/efi | \
${pkgs.coreutils-full}/bin/cut -d'p' -f1\
)" -L "${bootid}" -l EFI/boot/bootx64.efi
}
'';
};
};
};
}