Zfs + systemd-boot

Hi,

I am trying to setup a zfs example, but I am not able to get it work properly.
First I create the partitions, basically:

  • no swap,
  • a partition for root, which will be enabled with zfs + encryption
  • a partition for boot with zfs (no encryption – could be encrypted?)
  • a partition for efi (fat32) with esp flag turned on

Then I create the pools:

zpool create \
    -o compatibility=grub2 \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/boot \
    -R "/mnt" \
    zboot \
    /dev/vda2

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "/mnt" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    zroot \
    /dev/vda3

First question: maybe I can avoid the compatibility with grub2, since I am using systemd-boot?
Nevertheless, it should not be an error…

Then I create the datasets and the EFI:

mkfs.fat -F 32 -n EFI /dev/vda1

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    zboot/nixos

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    -o encryption=on \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    zroot/nixos

zfs create -o mountpoint=legacy zboot/nixos/boot
zfs create -o mountpoint=legacy zroot/nixos/root
zfs create -o mountpoint=legacy zroot/nixos/home

And here already I do not know if it is fine to pass legacy and canmount=off, since for what I read they seems old options, but all the tutorial found uses them…

Then I mount, as:

mount -t zfs zroot/nixos/root /mnt/
mkdir -p /mnt/boot
mount -t zfs zboot/nixos/boot /mnt/boot
mkdir -p /mnt/boot/efi
mount /dev/vda1 /mnt/boot/efi
mkdir -p /mnt/home
mount -t zfs zroot/nixos/home /mnt/home

Then the configuration:

  boot.loader.systemd-boot.enable = true; 
  boot.loader.efi.efiSysMountPoint = "/boot/efi";
  boot.loader.efi.canTouchEfiVariables = true;
  boot.initrd.systemd.enable = true;

  boot.kernelPackages = pkgs.zfs.latestCompatibleLinuxPackages;
  boot.supportedFilesystems = [ "zfs" ];
  boot.initrd.supportedFilesystems = [ "zfs" ];

  boot.zfs.requestEncryptionCredentials = true;

  networking.hostId = "12345678";
  boot.zfs.forceImportAll = false;
  boot.zfs.forceImportRoot = false;

And the hw config:

  fileSystems."/" =
    { device = "zroot/nixos/root";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "zboot/nixos/boot";
      fsType = "zfs";
    };

  fileSystems."/boot/efi" =
    { device = "/dev/disk/by-uuid/824F-1CB0";
      fsType = "vfat";
    };

  fileSystems."/home" =
    { device = "zroot/nixos/home";
      fsType = "zfs";
    };

  swapDevices = [ ];

The current error is the following:

> nixos-install --no-root-password --root /mnt --cores 0

Traceback (most recent call last):
  File "/nix/store/9c2qb99qs0yswf0xkfrqgawxbiq580lp-systemd-boot", line 341, in <module>
    main()
  File "/nix/store/9c2qb99qs0yswf0xkfrqgawxbiq580lp-systemd-boot", line 258, in main
    subprocess.check_call(["/nix/store/8lgs0dqh9ks1164fp4g14gq7w1ihjbf0-systemd-253.5/bin/bootctl", "--esp-path=/boot/efi"] + bootctl_flags + ["install"])
  File "/nix/store/lwzzgbnj41d657lpxczk6l5f7d5zcnj1-python3-3.10.11/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/8lgs0dqh9ks1164fp4g14gq7w1ihjbf0-systemd-253.5/bin/bootctl', '--esp-path=/boot/efi', 'install']' returned non-zero exit status 1.


Can someone please explain me what I am doing wrong please?
Thank you.
Regards

I have done a test, by changing the boot partition from zfs to ext4.
Maybe systemd-boot requires non-zfs?

With the change, the auto-generated hw config is:

  fileSystems."/" =
    { device = "zroot/nixos/root";
      fsType = "zfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/726c473e-1604-4158-8b90-92365e040557";
      fsType = "ext4";
    };

  fileSystems."/boot/efi" =
    { device = "/dev/disk/by-uuid/7D4E-FFD8";
      fsType = "vfat";
    };

  fileSystems."/home" =
    { device = "zroot/nixos/home";
      fsType = "zfs";
    };

  swapDevices = [ ];

With this change, nixos installs, but at reboot no password is asked, and the system does not boot…

Regards

systemd-boot should be able to load zfs mounts directly and initiate the unlock passphrase. I have this setup on my laptop. I do this with two partitions, the ESP and a zfs partition.

  boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.systemd-boot.enable = true;
  boot.supportedFilesystems = ["zfs"];

  fileSystems."/var/lib" = {
    device = "main/nixos/var/lib";
    fsType = "zfs";
  };

  fileSystems."/var/log" = {
    device = "main/nixos/var/log";
    fsType = "zfs";
  };

  fileSystems."/nix" = {
    device = "main/nixos/nix";
    fsType = "zfs";
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/8B7C-B408";
    fsType = "vfat";
  };

I use a mirrored zpool, no separate boot pool, no special config with systemd-boot issueless, as an example

export d0="/dev/nvme0n1"
export d1="/dev/nvme1n1"

wipefs $d0; sgdisk -z $d0; sgdisk -og $d0
wipefs $d1; sgdisk -z $d1; sgdisk -og $d1

sgdisk --new 1::+512M --typecode=1:EF00 --change-name=1:'ESP0' $d0
sgdisk --new 1::+512M --typecode=1:EF00 --change-name=1:'ESP1' $d1

sgdisk --new 2::+1857G --typecode=2:BF01 --change-name=2:'rpvol0' $d0
sgdisk --new 2::+1857G --typecode=2:BF01 --change-name=2:'rpvol1' $d1

partprobe
udevadm settle

mkfs.vfat -F32 -nESP1 /dev/disk/by-partlabel/ESP0
mkfs.vfat -F32 -nESP2 /dev/disk/by-partlabel/ESP1

zpool create                  \
      -o ashift=12            \
      -o autotrim=on          \
      -o acltype=posixacl     \
      -o compression=on       \
      -o dnodesize=auto       \
      -o normalization=formD  \
      -o encryption=on        \
      -o keyformat=passphrase \
      -o keylocation=prompt   \
      -o relatime=on          \
      -o xattr=sa             \
      -o mountpoint=none rootp mirror /dev/disk/by-partlabel/rpvol0 /dev/disk/by-partlabel/rpvol1

zfs create -o mountpoint=legacy -o compression=lz4 rootp/nixos
# and all others

mount -t zfs rootp/nixos /mnt
noglob mkdir -p /mnt/efiboot/{efi1,efi2}
mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP1 /mnt/efiboot/efi1
mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP2 /mnt/efiboot/efi2

# all other mounts
# generate config, if needed, install etc

then for relevant NixOS config

  boot = {
    loader = {
      systemd-boot = {
        enable = true;

        extraInstallCommands = ''
          mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP1 /efiboot/efi1
          mount -t vfat -o iocharset=iso8859-1 /dev/disk/by-label/ESP2 /efiboot/efi2
          cp -r /efiboot/efi1/* /efiboot/efi2
        '';
      }; # systemd-boot

      generationsDir.copyKernels = true;

      efi = {
        canTouchEfiVariables = true;
        efiSysMountPoint = "/efiboot/efi1";
      }; # efi
    }; # loader

    kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

    initrd = {
      kernelModules = [ "zfs" ];

      postDeviceCommands = ''
        zpool import -lf rootp
      ''; # postDeviceCommands
  }; # initrd

  supportedFilesystems = [ "zfs" ];

    zfs = {
      devNodes = "/dev/disk/by-partlabel";
    }; # zfs

  }; # boot

############################################

  fileSystems."/" =
    { device = "kwsrp/nixos";
      fsType = "zfs";
      neededForBoot = true;
    };

  fileSystems."/efiboot/efi2" =
    { device = "/dev/disk/by-label/ESP2";
      fsType = "vfat";
      options = [ "X-mount.mkdir" "iocharset=iso8859-1" ];
    };

Nothing else. No need for a separate boot pool with limited features for a grub-reachable extra stages, the kernel is in the EFI partition, in the initrd the zpool get unlocked/mounted (also useful if you have other pools encrypted with keys in some files stored for instance on a USB key).

2 Likes

systemd-boot doesn’t have any file system drivers like Grub does. It only reads partitions that the UEFI knows how to read, which unsurprisingly is not going to include anything like ZFS or ext4. To have /boot on ZFS or really anything that isn’t FAT23, you’re going to have to use Grub.

Not that I’d recommend that. There’s really no point, and it’s just painful. Grub’s support for ZFS is atrocious and I wouldn’t trust it. Really, the same goes for anything Grub supports that isn’t FAT32, ext4, and maybe btrfs. It’s far easier to just have one FAT32 partition for the ESP and leave it at that. No separation between /boot and /boot/efi, and you get to use systemd-boot, which I consider much better than Grub.

This means storing your kernels and initrds on the ESP, which IMO is completely fine. systemd-boot will load your initrd and start the kernel, the kernel will start the software in the initrd, and the initrd will ask for encryption passwords and load the OS on the root partition, which can be ZFS or whatever FS you want.

4 Likes

I will second what @ElvishJerricco says. I have (slowly) come around to agreeing.

In part, my earlier reluctance to give up on a bpool, and grub compatibility options, was for zfs snapshots and replication for backup of the boot loader and kernel/initrd, including habits and expectations carried over from other OS’s. NixOS makes this largely irrelevant.

So I have been slowly migrating my machines from (tiny)EFI+bpool+rpool with grub, to EFI+rpool with systemd-boot, both for the above reasons, and as a step in preparation for enabling secure-boot.

Hi,

thank to all of you for the support.
I have done a mix of the changes you suggested, and now I am able to boot correctly:

  1. Added missing boot.initrd.kernelModules = [ "zfs" ];
  2. Used a single partition for boot/efi of type fat32
  3. Used a single zpool create command, to create the root pool (as @xte has done, instead of two commands, to simplify the configuration)
  4. Removed separate pool for home (just to simplify the configuration)
  5. Removed boot.loader.efi.efiSysMountPoint = "/boot/efi"; adn boot.zfs.requestEncryptionCredentials = true;
  6. Turned on import: boot.zfs.forceImportRoot = true

From here now I’ll try slowly to perform some changes as separate home pool, use swap, etc.
But finally I am able to have a starting point.

Thanks again to all of you.
Regards

3 Likes

@xte I see there are two efi partitions, is that in case one disk fails? In that case, would you have to manually generate a new boot config with an edited efiSysMountPoint? Would it make sense to add a config specialisation for this contingency?

Related, I see further down you define fileSystems."/efiboot/efi2", is there a reason why this one is mounted instead of the other one?

I’m only now coming across these issues as I set up a zfs mirrored pool (previously only on single-drive laptops), so trying to figure out best practices.

Thanks!

Beware that forcing UTF-8 filenames and a normalization (-O normalization=formD), is not changeable after creating the pool, and has caused issues in the past when used on a pool containing a Nix store.

3 Likes

Yea, this has caused me to stop using nomalization=formD. It’s quite rare to be relevant, but sometimes it really sucks.

I see there are two efi partitions, is that in case one disk fails?

Yes, because the EFI partitions are out of the ZFS mirrored pool.

In that case, would you have to manually generate a new boot config with an edited efiSysMountPoint?

Yes if /mnt/efiboot/efi1 is not available (underlying disk missing), but otherwise a copy of the EFI is done during the nixos rebuild by the cp -r /efiboot/efi1/* /efiboot/efi2 in boot.loaded.systemd-boot.extraInstallCommands.
AFAICS, in the case where it’s /mnt/efiboot/efi2 which is not available, a nixos rebuild would not fail, the failing mount and cp would just echo an error message (since that shell script is not using set -e).
However, using rm -rf /efiboot/efi2; cp -r /efiboot/efi1 /efiboot/efi2 would be cleaner.

Would it make sense to add a config specialisation for this contingency?

See Feature Request: boot.loader.systemd-boot.mirroredBoots · Issue #152155 · NixOS/nixpkgs · GitHub
and the (draft) PR: nixos/systemd-boot: Add mirroredBoots by Gerg-L · Pull Request #246897 · NixOS/nixpkgs · GitHub which is mainly calling systemd-boot-builder.py on each EFI mountpoint.

Related, I see further down you define fileSystems."/efiboot/efi2", is there a reason why this one is mounted instead of the other one?

I guess that, in a non-degraded state, /efiboot/efi1 is found first and mounted by systemd-boot due to boot.loader.systemd-boot.efi.efiSysMountPoint = "/efiboot/efi1";
If that’s correct and if /efiboot/efi1 has a deterministic priority over /efiboot/efi2, it might work to skip defining fileSystems."/efiboot/efi1", but I would not bet on that. All mirrored EFI mountpoints are defined in the nixos test of the draft PR nixos/systemd-boot: Add mirroredBoots by Gerg-L · Pull Request #246897 · NixOS/nixpkgs · GitHub

Also, the mounts in extraInstallCommands, do not seem necessary, moreover if they are in fileSystems (but I’ve not tested).

Hi,

I too have been trying to setup my Framework Laptop so that I can achieve snapshots for my encrypted system.
I did a partition for EFI for /boot. And an encrypted zpool with datasets for /, /nix, /var and /home .
But I’m not happy with the setup. I would like to either do LUKS, or just do unencrypted and just encrypt the /home dataset and do snapshots of only /home.

Can you share how you did your setup?

(I’m not the same person, but I have opinions).

First of all, check out a comment I made on another thread about things to consider with ZFS root, including why you probably don’t want to use ZFS’s native encryption: How good is ZFS on root on NixOS? - #9 by ElvishJerricco

Otherwise, I recommend a fairly simple configuration with just an EFI partition and a root ZFS pool partition using LUKS. Dataset layout within the pool is a matter of taste, but I do something like the following usually:

NAME                                       CANMOUNT  MOUNTPOINT
jace                                       on        none
jace/cache                                 off       /
jace/cache/home                            off       /home
jace/cache/home/will                       off       /home/will
jace/cache/home/will/.cache                on        /home/will/.cache
jace/cache/tmp                             on        /tmp
jace/cache/var                             off       /var
jace/cache/var/cache                       on        /var/cache
jace/system                                on        /
jace/system/nix                            on        /nix
jace/system/nix/store                      on        /nix/store
jace/system/var                            on        /var
jace/user                                  off       /
jace/user/home                             on        /home
jace/user/home/will                        on        /home/will

I read the other thread, thanks for that.
Yeah, my reason for wanting to use zfs encrypt was because I have a TrueNAS system and wanted to do encrypted backups to TrueNAS.
What other solution is there to this?