NixOS on mirrored ssd: boot, swap, native encrypted zfs

mackenzie · September 29, 2020, 9:31pm

Hello all

I’ve been trying to move to a more declarative way of handling my computers. I read a lot and tried a few things with NixOS. But now I’m stuck.

I’ve been trying to get mirrored boot working, but it keeps failing.
It does work on a single 1TB ssd.

When I try a short tutorial it fails:
Encrypted ZFS mirror with mirrored boot on NixOS - Elis Hirwing # Doesn’t work, none of the drives are recognized/won’t boot.

https://nixos.wiki/wiki/NixOS_on_ZFS#With_RAID When I change my “working” single ssd script, it also fails. This time during installation (nixos-install). I didn’t use the sda1/sdb1, instead I used by-id. I did use ext4 for mirror-boot instead of the vfat for single-boot.

error:
File system “/boot” is not a FAT EFI System Partition (ESP) file system.

Can anyone point me to a “better”/other tutorial.

note: Tomorrow I’ll upload my current script. I would be great if someone could check it, feedback is always appreciated. Or it might be helpful for others in the future.

Some background / My roadmap:

single ssd: boot, swap, native encrypted zfs (success)

mirrored ssd

thinkpad T540p: thank you Fedora, hello NixOS (gnome → sway, guake → some-terminal, … )

supermicro server: mirrored 21TB ssd + raidz2 68TB (eaton ups, nextcloud, mail, )

pi4: single ssd: kiosk mode 24/7, display 4 ipCamera images in closed network(4cam+1poeSwitch+pi4+tv)

saas development: elixir, phoenix, liveview, postgresql; android

router/firewall/wireguard/unbound-or-similar: NixOS or opnsense box. unencrypted so it boots without password prompt.

I’ve looked at a lot op personal repos/configs. The end goal is to try to do it the “flakes” way. (https://github.com/nrdxp/nixflk)

edit1: add error message

mackenzie · October 5, 2020, 10:30am

this one seems to work

0x4A6F · October 6, 2020, 8:32pm

You might also find some useful information over at ZFS - NixOS Wiki.

mackenzie · October 7, 2020, 8:39am

I did but couldn’t get it too boot mirrored. everything else worked.

For now I’m not using a swap partition because I don’t want to unlock 2 encryptions at boot:

native zfs passphrase
swap partition luks passphrase

My sources were:
source_1: 20200624; ZFS - NixOS Wiki
source_2: 20200626; https://gist.github.com/mannkind/07b21461061e599e1372b2bf8c46a337
source_3: 20200626; ArchLinux install script with ZFS as the root, all sitting on LUKS encryption, using systemd-boot for UEFI (ArchLinux with ZFS Root on LUKS with UEFI) · GitHub
source_4: 20200626; Installing Ubuntu on a ZFS root, with encryption and mirroring – Saverio Miroddi – 64K RAM SYSTEM 38911 BASIC BYTES FREE
source_5: 20200626; GitHub - saveriomiroddi/zfs-installer: Former repository of my ZFS installer project
source_6: 20200403; GitHub - bhougland18/nixos_config: Nixos configuration
source_7: 20190804; https://elis.nu/blog/2019/08/encrypted-zfs-mirror-with-mirrored-boot-on-nixos/ # Doesn’t work, none of the drives are recognized/won’t boot.
source_8: 20190814; GitHub - johnalotoski/nixos-etc: NixOS related system configs
source_9: 20190817; GitHub - bjornfor/nixos-config: My NixOS configuration files. (This public mirror is not updated anymore.)
source_10: 20200919; https://wiki.c3d2.de/Diskussion:NixOS
source_11: 20190704; How I installed Encrypted ZFS root on NixOS · GitHub
source_12: 20191024; GitHub - a-schaefers/themelios: Bootstrap a zfs-on-root NixOS configuration in one command. Themelios automates the entire installation process using any NixOS livedisk with an internet connection and your git repo. Themelios is flexible with many configuration options and allows for unique, per-machine customization.
source_13: 20200117; https://github.com/eoli3n/nix-config/blob/ea5870d33e2e1d9259260ebb79cb76840d4eeea7/scripts/install
source_14: 20180624; Secure, Declarative Key Management with NixOps, Pass, and nix-plugins
source_15: 20190417; NixOps User's Guide

rbrewer123 · December 31, 2020, 6:44pm

I’ve recently been working on this as well. Following a disk failure in which I lost my boot partition, I decided to attempt a mirrored boot setup.

My setup is now as follows:

nixos 20.09
zfs 2.0 (boot.zfs.enableUnstable = true)
2 identical drives in a mirror configuration. Each drive has a giant partition for ZFS and a 1 GB EFI boot partition
My zfs pool is a mirror, using zfs native encryption and contains my root and /home datasets.
Grub bootloader

Here are the relevant lines from my nixos configuration:

boot.loader.grub = {
    enable = true;
    zfsSupport = true;
    efiSupport = true;
    mirroredBoots = [
      {
        devices = [ "nodev" ];
        path = "/boot1";
      }
      {
        devices = [ "nodev" ];
        path = "/boot2";
      }
    ];
  };
 
  boot.loader.efi.canTouchEfiVariables = true;

  boot.supportedFilesystems = [ "zfs" ];
  boot.zfs.enableUnstable = true;
  
  # prevents "multiple pools with same name" problem during boot
  boot.zfs.devNodes = "/dev/disk/by-partuuid";

  fileSystems."/" =
    { device = "rpool/safe/root/nixos";
      fsType = "zfs";
    };

  fileSystems."/home" =
    { device = "rpool/safe/home";
      fsType = "zfs";
    };

  fileSystems."/nix" =
    { device = "rpool/nix";
      fsType = "zfs";
    };

  fileSystems."/boot1" =
    { device = "/dev/disk/by-uuid/B80C-173D";
      fsType = "vfat";
    };

  fileSystems."/boot2" =
    { device = "/dev/disk/by-uuid/9E23-5760";
      fsType = "vfat";
    };

  swapDevices = [ ];

This is working very well when both drives are installed. However, when I remove a drive and attempt to boot, I can’t get to a fully working system. Here’s what happens:

grub works fine
nixos stage 1 works fine (rpool imported)
nixos stage 2 hangs for a long time trying to mount the missing /boot2
eventually I get the option to enter the root password for emergency mode, or press ctrl-d to continue

If I press ctrl-d, it basically hangs again for a long time, and eventually puts me back to step 4.

If instead I enter the root password, I get a shell and can poke around. But if I try to systemctl start display-manager.service, the system hangs again, and eventually I’m back to step 4.

Is there some way for me to tell systemd that missing /boot2 is ok for now? That would be enough to make the mirrored booting worthwhile. In fact, having the system boot up as if everything is totally fine even though an entire drive is dead may not be desirable anyway.

bjornfor · January 1, 2021, 9:56am

Try fileSystems."/boot2".options = [ "nofail" ];.

rbrewer123 · January 3, 2021, 5:47pm

Thanks, that works well. I added "nofail" for /boot1 and boot2 to allow either to be missing.

aij · February 28, 2021, 10:08pm

Interesting. I just set up a similar mirrored boot + zfs native encryption for /, but I used MD + ext4 for /boot instead of mirroredBoots + vfat.

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdf1 /dev/sdg1

What’s the advantage of using mirroredBoots? Is it needed for EFI booting? (I’m still booting in BIOS mode because I’ve found it to generally be more reliable, though I imagine I will try switching again eventually on newer hardware.)

bgibson · July 12, 2021, 6:30pm

Old thread, but since this question wasn’t answered, yes EFI booting requires a FAT/16/32 partition with the ESP flag. So an EXT4 partition can only boot via BIOS/MBR.