NixOS on mirrored ssd: boot, swap, native encrypted zfs

Hello all

I’ve been trying to move to a more declarative way of handling my computers. I read a lot and tried a few things with NixOS. But now I’m stuck.

I’ve been trying to get mirrored boot working, but it keeps failing.
It does work on a single 1TB ssd.

When I try a short tutorial it fails:
Encrypted ZFS mirror with mirrored boot on NixOS # Doesn’t work, none of the drives are recognized/won’t boot.

ZFS - NixOS Wiki When I change my “working” single ssd script, it also fails. This time during installation (nixos-install). I didn’t use the sda1/sdb1, instead I used by-id. I did use ext4 for mirror-boot instead of the vfat for single-boot.

File system “/boot” is not a FAT EFI System Partition (ESP) file system.

Can anyone point me to a “better”/other tutorial.

note: Tomorrow I’ll upload my current script. I would be great if someone could check it, feedback is always appreciated. Or it might be helpful for others in the future.

Some background / My roadmap:

  • single ssd: boot, swap, native encrypted zfs (success)
  • mirrored ssd
  • thinkpad T540p: thank you Fedora, hello NixOS (gnome → sway, guake → some-terminal, … )
  • supermicro server: mirrored 21TB ssd + raidz2 68TB (eaton ups, nextcloud, mail, )
  • pi4: single ssd: kiosk mode 24/7, display 4 ipCamera images in closed network(4cam+1poeSwitch+pi4+tv)
  • saas development: elixir, phoenix, liveview, postgresql; android
  • router/firewall/wireguard/unbound-or-similar: NixOS or opnsense box. unencrypted so it boots without password prompt.

I’ve looked at a lot op personal repos/configs. The end goal is to try to do it the “flakes” way. (GitHub - divnix/devos: NixOS Template)

edit1: add error message

1 Like

this one seems to work

1 Like

You might also find some useful information over at ZFS - NixOS Wiki.

I did but couldn’t get it too boot mirrored. everything else worked.

For now I’m not using a swap partition because I don’t want to unlock 2 encryptions at boot:

  • native zfs passphrase
  • swap partition luks passphrase

My sources were:
source_1: 20200624; ZFS - NixOS Wiki
source_2: 20200626; NixOS Installation Init · GitHub
source_3: 20200626; ArchLinux install script with ZFS as the root, all sitting on LUKS encryption, using systemd-boot for UEFI (ArchLinux with ZFS Root on LUKS with UEFI) · GitHub
source_4: 20200626; Installing Ubuntu on a ZFS root, with encryption and mirroring – Saverio Miroddi – 64K RAM SYSTEM  38911 BASIC BYTES FREE
source_5: 20200626; GitHub - saveriomiroddi/zfs-installer: Former repository of my ZFS installer project
source_6: 20200403; GitHub - bhougland18/nixos_config: Nixos configuration
source_7: 20190804; Encrypted ZFS mirror with mirrored boot on NixOS # Doesn’t work, none of the drives are recognized/won’t boot.
source_8: 20190814; GitHub - johnalotoski/nixos-etc: NixOS related system configs
source_9: 20190817; GitHub - bjornfor/nixos-config: My NixOS configuration files. (This public mirror is not updated anymore.)
source_10: 20200919; Diskussion:NixOS – C3D2
source_11: 20190704; How I installed Encrypted ZFS root on NixOS · GitHub
source_12: 20191024; GitHub - a-schaefers/themelios: Bootstrap a zfs-on-root NixOS configuration in one command. Themelios automates the entire installation process using any NixOS livedisk with an internet connection and your git repo. Themelios is flexible with many configuration options and allows for unique, per-machine customization.
source_13: 20200117; nix-config/scripts/install at ea5870d33e2e1d9259260ebb79cb76840d4eeea7 · eoli3n/nix-config · GitHub
source_14: 20180624; Secure, Declarative Key Management with NixOps, Pass, and nix-plugins
source_15: 20190417; NixOps User's Guide

1 Like

I’ve recently been working on this as well. Following a disk failure in which I lost my boot partition, I decided to attempt a mirrored boot setup.

My setup is now as follows:

  • nixos 20.09
  • zfs 2.0 (boot.zfs.enableUnstable = true)
  • 2 identical drives in a mirror configuration. Each drive has a giant partition for ZFS and a 1 GB EFI boot partition
  • My zfs pool is a mirror, using zfs native encryption and contains my root and /home datasets.
  • Grub bootloader

Here are the relevant lines from my nixos configuration:

boot.loader.grub = {
    enable = true;
    zfsSupport = true;
    efiSupport = true;
    mirroredBoots = [
        devices = [ "nodev" ];
        path = "/boot1";
        devices = [ "nodev" ];
        path = "/boot2";
  boot.loader.efi.canTouchEfiVariables = true;

  boot.supportedFilesystems = [ "zfs" ];
  boot.zfs.enableUnstable = true;
  # prevents "multiple pools with same name" problem during boot
  boot.zfs.devNodes = "/dev/disk/by-partuuid";

  fileSystems."/" =
    { device = "rpool/safe/root/nixos";
      fsType = "zfs";

  fileSystems."/home" =
    { device = "rpool/safe/home";
      fsType = "zfs";

  fileSystems."/nix" =
    { device = "rpool/nix";
      fsType = "zfs";

  fileSystems."/boot1" =
    { device = "/dev/disk/by-uuid/B80C-173D";
      fsType = "vfat";

  fileSystems."/boot2" =
    { device = "/dev/disk/by-uuid/9E23-5760";
      fsType = "vfat";

  swapDevices = [ ];

This is working very well when both drives are installed. However, when I remove a drive and attempt to boot, I can’t get to a fully working system. Here’s what happens:

  1. grub works fine
  2. nixos stage 1 works fine (rpool imported)
  3. nixos stage 2 hangs for a long time trying to mount the missing /boot2
  4. eventually I get the option to enter the root password for emergency mode, or press ctrl-d to continue

If I press ctrl-d, it basically hangs again for a long time, and eventually puts me back to step 4.

If instead I enter the root password, I get a shell and can poke around. But if I try to systemctl start display-manager.service, the system hangs again, and eventually I’m back to step 4.

Is there some way for me to tell systemd that missing /boot2 is ok for now? That would be enough to make the mirrored booting worthwhile. In fact, having the system boot up as if everything is totally fine even though an entire drive is dead may not be desirable anyway.

1 Like

Try fileSystems."/boot2".options = [ "nofail" ];.


Thanks, that works well. I added "nofail" for /boot1 and boot2 to allow either to be missing.

1 Like

Interesting. I just set up a similar mirrored boot + zfs native encryption for /, but I used MD + ext4 for /boot instead of mirroredBoots + vfat.

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdf1 /dev/sdg1

What’s the advantage of using mirroredBoots? Is it needed for EFI booting? (I’m still booting in BIOS mode because I’ve found it to generally be more reliable, though I imagine I will try switching again eventually on newer hardware.)

Old thread, but since this question wasn’t answered, yes EFI booting requires a FAT/16/32 partition with the ESP flag. So an EXT4 partition can only boot via BIOS/MBR.