Zfs + systemd-boot

Hi,

I am trying to setup a zfs example, but I am not able to get it work properly.
First I create the partitions, basically:

  • no swap,
  • a partition for root, which will be enabled with zfs + encryption
  • a partition for boot with zfs (no encryption – could be encrypted?)
  • a partition for efi (fat32) with esp flag turned on

Then I create the pools:

zpool create \
    -o compatibility=grub2 \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/boot \
    -R "/mnt" \
    zboot \
    /dev/vda2

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "/mnt" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    zroot \
    /dev/vda3

First question: maybe I can avoid the compatibility with grub2, since I am using systemd-boot?
Nevertheless, it should not be an error…

Then I create the datasets and the EFI:

mkfs.fat -F 32 -n EFI /dev/vda1

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    zboot/nixos

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    -o encryption=on \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    zroot/nixos

zfs create -o mountpoint=legacy zboot/nixos/boot
zfs create -o mountpoint=legacy zroot/nixos/root
zfs create -o mountpoint=legacy zroot/nixos/home

And here already I do not know if it is fine to pass legacy and canmount=off, since for what I read they seems old options, but all the tutorial found uses them…

Then I mount, as:

mount -t zfs zroot/nixos/root /mnt/
mkdir -p /mnt/boot
mount -t zfs zboot/nixos/boot /mnt/boot
mkdir -p /mnt/boot/efi
mount /dev/vda1 /mnt/boot/efi
mkdir -p /mnt/home
mount -t zfs zroot/nixos/home /mnt/home

Then the configuration:

  boot.loader.systemd-boot.enable = true; 
  boot.loader.efi.efiSysMountPoint = "/boot/efi";
  boot.loader.efi.canTouchEfiVariables = true;
  boot.initrd.systemd.enable = true;

  boot.kernelPackages = pkgs.zfs.latestCompatibleLinuxPackages;
  boot.supportedFilesystems = [ "zfs" ];
  boot.initrd.supportedFilesystems = [ "zfs" ];

  boot.zfs.requestEncryptionCredentials = true;

  networking.hostId = "12345678";
  boot.zfs.forceImportAll = false;
  boot.zfs.forceImportRoot = false;

And the hw config:

  fileSystems."/" =
    { device = "zroot/nixos/root";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "zboot/nixos/boot";
      fsType = "zfs";
    };

  fileSystems."/boot/efi" =
    { device = "/dev/disk/by-uuid/824F-1CB0";
      fsType = "vfat";
    };

  fileSystems."/home" =
    { device = "zroot/nixos/home";
      fsType = "zfs";
    };

  swapDevices = [ ];

The current error is the following:

> nixos-install --no-root-password --root /mnt --cores 0

Traceback (most recent call last):
  File "/nix/store/9c2qb99qs0yswf0xkfrqgawxbiq580lp-systemd-boot", line 341, in <module>
    main()
  File "/nix/store/9c2qb99qs0yswf0xkfrqgawxbiq580lp-systemd-boot", line 258, in main
    subprocess.check_call(["/nix/store/8lgs0dqh9ks1164fp4g14gq7w1ihjbf0-systemd-253.5/bin/bootctl", "--esp-path=/boot/efi"] + bootctl_flags + ["install"])
  File "/nix/store/lwzzgbnj41d657lpxczk6l5f7d5zcnj1-python3-3.10.11/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/8lgs0dqh9ks1164fp4g14gq7w1ihjbf0-systemd-253.5/bin/bootctl', '--esp-path=/boot/efi', 'install']' returned non-zero exit status 1.


Can someone please explain me what I am doing wrong please?
Thank you.
Regards

I have done a test, by changing the boot partition from zfs to ext4.
Maybe systemd-boot requires non-zfs?

With the change, the auto-generated hw config is:

  fileSystems."/" =
    { device = "zroot/nixos/root";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/726c473e-1604-4158-8b90-92365e040557";
      fsType = "ext4";
    };

  fileSystems."/boot/efi" =
    { device = "/dev/disk/by-uuid/7D4E-FFD8";
      fsType = "vfat";
    };

  fileSystems."/home" =
    { device = "zroot/nixos/home";
      fsType = "zfs";
    };

  swapDevices = [ ];

With this change, nixos installs, but at reboot no password is asked, and the system does not boot…

Regards

systemd-boot should be able to load zfs mounts directly and initiate the unlock passphrase. I have this setup on my laptop. I do this with two partitions, the ESP and a zfs partition.

  boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.systemd-boot.enable = true;
  boot.supportedFilesystems = ["zfs"];

  fileSystems."/var/lib" = {
    device = "main/nixos/var/lib";
    fsType = "zfs";
  };

  fileSystems."/var/log" = {
    device = "main/nixos/var/log";
    fsType = "zfs";
  };

  fileSystems."/nix" = {
    device = "main/nixos/nix";
    fsType = "zfs";
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/8B7C-B408";
    fsType = "vfat";
  };

I use a mirrored zpool, no separate boot pool, no special config with systemd-boot issueless, as an example

export d0="/dev/nvme0n1"
export d1="/dev/nvme1n1"

wipefs $d0; sgdisk -z $d0; sgdisk -og $d0
wipefs $d1; sgdisk -z $d1; sgdisk -og $d1

sgdisk --new 1::+512M --typecode=1:EF00 --change-name=1:'ESP0' $d0
sgdisk --new 1::+512M --typecode=1:EF00 --change-name=1:'ESP1' $d1

sgdisk --new 2::+1857G --typecode=2:BF01 --change-name=2:'rpvol0' $d0
sgdisk --new 2::+1857G --typecode=2:BF01 --change-name=2:'rpvol1' $d1

partprobe
udevadm settle

mkfs.vfat -F32 -nESP1 /dev/disk/by-partlabel/ESP0
mkfs.vfat -F32 -nESP2 /dev/disk/by-partlabel/ESP1

zpool create                  \
      -o ashift=12            \
      -o autotrim=on          \
      -o acltype=posixacl     \
      -o compression=on       \
      -o dnodesize=auto       \
      -o normalization=formD  \
      -o encryption=on        \
      -o keyformat=passphrase \
      -o keylocation=prompt   \
      -o relatime=on          \
      -o xattr=sa             \
      -o mountpoint=none rootp mirror /dev/disk/by-partlabel/rpvol0 /dev/disk/by-partlabel/rpvol1

zfs create -o mountpoint=legacy -o compression=lz4 rootp/nixos
# and all others

mount -t zfs rootp/nixos /mnt
noglob mkdir -p /mnt/efiboot/{efi1,efi2}
mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP1 /mnt/efiboot/efi1
mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP2 /mnt/efiboot/efi2

# all other mounts
# generate config, if needed, install etc

then for relevant NixOS config

  boot = {
    loader = {
      systemd-boot = {
        enable = true;

        extraInstallCommands = ''
          mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP1 /efiboot/efi1
          mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP2 /efiboot/efi2
          cp -r /efiboot/efi1/* /efiboot/efi2
        '';
      }; # systemd-boot

      generationsDir.copyKernels = true;

      efi = {
        canTouchEfiVariables = true;
        efiSysMountPoint = "/efiboot/efi1";
      }; # efi
    }; # loader

    kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

    initrd = {
      kernelModules = [ "zfs" ];

      postDeviceCommands = ''
        zpool import -lf rootp
      ''; # postDeviceCommands
  }; # initrd

  supportedFilesystems = [ "zfs" ];

    zfs = {
      devNodes = "/dev/disk/by-partlabel";
    }; # zfs

  }; # boot

############################################

  fileSystems."/" =
    { device = "kwsrp/nixos";
      fsType = "zfs";
      neededForBoot = true;
    };

  fileSystems."/efiboot/efi2" =
    { device = "/dev/disk/by-label/ESP2";
      fsType = "vfat";
      options = [ "X-mount.mkdir" "iocharset=iso8859-1" ];
    };

Nothing else. No need for a separate boot pool with limited features for a grub-reachable extra stages, the kernel is in the EFI partition, in the initrd the zpool get unlocked/mounted (also useful if you have other pools encrypted with keys in some files stored for instance on a USB key).

1 Like

systemd-boot doesn’t have any file system drivers like Grub does. It only reads partitions that the UEFI knows how to read, which unsurprisingly is not going to include anything like ZFS or ext4. To have /boot on ZFS or really anything that isn’t FAT23, you’re going to have to use Grub.

Not that I’d recommend that. There’s really no point, and it’s just painful. Grub’s support for ZFS is atrocious and I wouldn’t trust it. Really, the same goes for anything Grub supports that isn’t FAT32, ext4, and maybe btrfs. It’s far easier to just have one FAT32 partition for the ESP and leave it at that. No separation between /boot and /boot/efi, and you get to use systemd-boot, which I consider much better than Grub.

This means storing your kernels and initrds on the ESP, which IMO is completely fine. systemd-boot will load your initrd and start the kernel, the kernel will start the software in the initrd, and the initrd will ask for encryption passwords and load the OS on the root partition, which can be ZFS or whatever FS you want.

3 Likes

I will second what @ElvishJerricco says. I have (slowly) come around to agreeing.

In part, my earlier reluctance to give up on a bpool, and grub compatibility options, was for zfs snapshots and replication for backup of the boot loader and kernel/initrd, including habits and expectations carried over from other OS’s. NixOS makes this largely irrelevant.

So I have been slowly migrating my machines from (tiny)EFI+bpool+rpool with grub, to EFI+rpool with systemd-boot, both for the above reasons, and as a step in preparation for enabling secure-boot.

Hi,

thank to all of you for the support.
I have done a mix of the changes you suggested, and now I am able to boot correctly:

  1. Added missing boot.initrd.kernelModules = [ "zfs" ];
  2. Used a single partition for boot/efi of type fat32
  3. Used a single zpool create command, to create the root pool (as @xte has done, instead of two commands, to simplify the configuration)
  4. Removed separate pool for home (just to simplify the configuration)
  5. Removed boot.loader.efi.efiSysMountPoint = "/boot/efi"; adn boot.zfs.requestEncryptionCredentials = true;
  6. Turned on import: boot.zfs.forceImportRoot = true

From here now I’ll try slowly to perform some changes as separate home pool, use swap, etc.
But finally I am able to have a starting point.

Thanks again to all of you.
Regards

3 Likes