Install NixOS with Btrfs, LVM, on LUKS using TPM2 to start up, with support for suspend to disk

Hello all. I’m making this thread so I can possibly help others trying to do the same sort of configuration. Usually I just use the calamares installer and call it a day, but this time I don’t want to make any compromises. I tried to piece together what I need to create this from the NixOS and Arch Linux wikis.

To start, I will explain each step I took on my live USB.

I first deleted all existing partitions and created new ones with gparted.

[nixos@nixos:~]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0   2.3G  1 loop /nix/.ro-store
sda           8:0    1   7.3G  0 disk 
├─sda1        8:1    1   7.2G  0 part 
│ └─ventoy  254:0    0   2.4G  1 dm   /iso
└─sda2        8:2    1    32M  0 part 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0   512M  0 part 
└─nvme0n1p2 259:2    0   931G  0 part

I then did the following as root.

# creating the encrypted partition
cryptsetup luksFormat /dev/nvme0n1p2
cryptsetup open /dev/nvme0n1p2 cryptlvm

# LVM setup
pvcreate /dev/mapper/cryptlvm
vgcreate vg /dev/mapper/cryptlvm

# I have 32GB of RAM, so my swap should be just as big
lvcreate -L 32G -n swap vg
lvcreate -l 100%FREE -n root vg

mkfs.fat -F 32 /dev/nvme0n1p1

mkdir -p /mnt
mkswap /dev/mapper/vg-swap

mkfs.btrfs /dev/mapper/vg-root
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix

umount /mnt

mount -o compress=zstd,subvol=root /dev/mapper/vg-root /mnt
mkdir /mnt/{home,nix}
mount -o compress=zstd,subvol=home /dev/mapper/vg-root /mnt/home
mount -o compress=zstd,noatime,subvol=nix /dev/mapper/vg-root /mnt/nix
mkdir /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot

Now that seemed like everything I had to do on the command line, and the rest would have to go into my Nix config. Thus I tried to do nixos-generate-config --root /mnt. Here is the result of that. What’s wrong here? Did I miss some steps?

For one thing, I don’t see the place LUKS is supposed to unlock my hard drive.

configuration.nix:

# Edit this configuration file to define what should be installed on
# your system. Help is available in the configuration.nix(5) man page, on
# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).

{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  # networking.hostName = "nixos"; # Define your hostname.
  # Pick only one of the below networking options.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.
  # networking.networkmanager.enable = true;  # Easiest to use and most distros use this by default.

  # Set your time zone.
  # time.timeZone = "Europe/Amsterdam";

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Select internationalisation properties.
  # i18n.defaultLocale = "en_US.UTF-8";
  # console = {
  #   font = "Lat2-Terminus16";
  #   keyMap = "us";
  #   useXkbConfig = true; # use xkb.options in tty.
  # };

  # Enable the X11 windowing system.
  services.xserver.enable = true;


  # Enable the GNOME Desktop Environment.
  services.xserver.displayManager.gdm.enable = true;
  services.xserver.desktopManager.gnome.enable = true;
  

  # Configure keymap in X11
  # services.xserver.xkb.layout = "us";
  # services.xserver.xkb.options = "eurosign:e,caps:escape";

  # Enable CUPS to print documents.
  # services.printing.enable = true;

  # Enable sound.
  # hardware.pulseaudio.enable = true;
  # OR
  # services.pipewire = {
  #   enable = true;
  #   pulse.enable = true;
  # };

  # Enable touchpad support (enabled default in most desktopManager).
  # services.libinput.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  # users.users.alice = {
  #   isNormalUser = true;
  #   extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  #   packages = with pkgs; [
  #     tree
  #   ];
  # };

  # programs.firefox.enable = true;

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  # environment.systemPackages = with pkgs; [
  #   vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
  #   wget
  # ];

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = {
  #   enable = true;
  #   enableSSHSupport = true;
  # };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  # services.openssh.enable = true;

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  # Copy the NixOS configuration file and link it from the resulting system
  # (/run/current-system/configuration.nix). This is useful in case you
  # accidentally delete configuration.nix.
  # system.copySystemConfiguration = true;

  # This option defines the first version of NixOS you have installed on this particular machine,
  # and is used to maintain compatibility with application data (e.g. databases) created on older NixOS versions.
  #
  # Most users should NEVER change this value after the initial install, for any reason,
  # even if you've upgraded your system to a new NixOS release.
  #
  # This value does NOT affect the Nixpkgs version your packages and OS are pulled from,
  # so changing it will NOT upgrade your system - see https://nixos.org/manual/nixos/stable/#sec-upgrading for how
  # to actually do that.
  #
  # This value being lower than the current NixOS release does NOT mean your system is
  # out of date, out of support, or vulnerable.
  #
  # Do NOT change this value unless you have manually inspected all the changes it would make to your configuration,
  # and migrated your data accordingly.
  #
  # For more information, see `man configuration.nix` or https://nixos.org/manual/nixos/stable/options#opt-system.stateVersion .
  system.stateVersion = "24.11"; # Did you read the comment?

}

hardware-configuration.nix:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "nvme" "xhci_pci" "thunderbolt" "usb_storage" "usbhid" "sd_mod" ];
  boot.initrd.kernelModules = [ "dm-snapshot" ];
  boot.kernelModules = [ "kvm-amd" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/7ea32639-ffba-4283-af2f-880b95f8e856";
      fsType = "btrfs";
      options = [ "subvol=root" ];
    };

  fileSystems."/home" =
    { device = "/dev/disk/by-uuid/7ea32639-ffba-4283-af2f-880b95f8e856";
      fsType = "btrfs";
      options = [ "subvol=home" ];
    };

  fileSystems."/nix" =
    { device = "/dev/disk/by-uuid/7ea32639-ffba-4283-af2f-880b95f8e856";
      fsType = "btrfs";
      options = [ "subvol=nix" ];
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/253D-AA2B";
      fsType = "vfat";
      options = [ "fmask=0022" "dmask=0022" ];
    };

  swapDevices = [ ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;
  # networking.interfaces.wlp1s0.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}
1 Like

For the tpm part you’ll need :

boot.initrd.systemd.enableTpm2 = true;
boot.initrd.luks.devices.root = {
device = “/dev/disk/by-partuuid/$myPARTUUID”;
};
security.tpm2.enable = true;

use systemd-enroll utility to register tpm keys in a luks slot

1 Like

I made the changes you suggested to the initrd.

Leaving this for others, I made a mistake when making the boot partition.

File system "/dev/nvme0n1p1" has wrong type for an EFI System Partition (ESP).

To attempt to fix:

umount /mnt/boot
cfdisk /dev/nvme0n1 # change to ESP partition, I just had fat32.
mkfs.fat -F 32 /dev/nvme0n1p1
mount /dev/nvme0n1p1 /mnt/boot

Make sure you change the UUID of the boot partition!

Running nixos-install now yields:

[root@nixos:/mnt]# nixos-install
copying channel...
building the configuration in /mnt/etc/nixos/configuration.nix...
/nix/store/ndlg17m34nawicnrny2kchra1mzygvww-nixos-system-nixos-24.11.713895.666e1b3f09c2
installing the boot loader...
setting up /etc...
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/loader/entries".
Created "/boot/EFI/Linux".
Copied "/nix/store/xv7q10lk4lxfax7naj3b63aj2pyjv9gb-systemd-256.10/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/nix/store/xv7q10lk4lxfax7naj3b63aj2pyjv9gb-systemd-256.10/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
⚠️ Mount point '/boot' which backs the random seed file is world accessible, which is a security hole! ⚠️
⚠️ Random seed file '/boot/loader/.#bootctlrandom-seed29b32b775ec3aec8' is world accessible, which is a security hole! ⚠️
Random seed file /boot/loader/random-seed successfully written (32 bytes).
Created EFI boot entry "Linux Boot Manager".
setting up /etc...
setting up /etc...
setting root password...
New password: 
Password change has been aborted.
passwd: Permission denied
passwd: password unchanged
Setting a root password failed with the above printed error.
You can set the root password manually by executing `nixos-enter --root '/mnt'` and then running `passwd` in the shell of the new system.

I figured I couldn’t ignore these warnings, so what to do now?

According to here (Nixos-install with custom flake results in /boot being world accessible - #6 by NovaViper), I need to set permissions differently.

  fileSystems."/boot" = {
      device = "${bootPart}";
      fsType = "vfat";
-      options = [ "fmask=0022" "dmask=0022" ];
+     options = [ "umask=0077" ]";
    };

This however didn’t make the warning go away. Reading further it says I can just ignore the warning.

Looks like booting up worked without any issue. I just had to log in and add a password to my user account.


One problem I’m having is that vesktop screen sharing doesn’t work. It seems like it’s falling back to X11-based screensharing because I can capture any xwayland window.

Seems like there might be a bug where nixos-generate-config isn’t picking up your LUKS device, so you’ll need to set boot.initrd.luks.devices manually, as @noximus237 showed.

When you mount the ESP, you need to do so with mount -o umask=0077 ..., or else it gets the default which is 0022, which is then translated by linux into the pairing of fmask and dmask settings. Then, nixos-generate-config knows that, because of the warning you’re getting, the fmask and dmask options need to be preserved in the generated hardware-configuration.nix, so it just copies in whatever value you mounted it with. All this is to say, nixos-generate-config generated the wrong thing, and you’re now getting warnings, because you need to have it mounted with -o umask=0077 in the first place.

So, boot.initrd.systemd.enableTpm2 has been renamed to boot.initrd.systemd.tpm2.enable, and it’s enabled by default. But, it’s only relevant if you use boot.initrd.systemd.enable = true;, which @wildwestrom has not done yet. But yes, you’ll need boot.initrd.systemd.enable = true; if you intend to unlock the disk using the TPM2.

Also security.tpm2.enable is something very different and not relevant for just unlocking the root disk.

1 Like

I will also add, it is possible to do TPM2 unlocking securely, but it’s difficult and you’ll probably get it wrong if you don’t know what you’re doing. Here’s an article about some of the ways it can go wrong

So I did in fact enable boot.initrd.systemd.enable = true in my configuration. Also, thank you for the article. I do want to make sure I get this right, so I’ll read carefully so I’m not open to the various exploits out there.

I also tested suspend to disk with systemctl hibernate and it seemed to work without any issue. It halted, the screen turned off, and when I hit the power button again in the morning and after putting in my LUKS password it started right up where I left off.

Have you considered using zfs instead to reduce your headaches?

I don’t think btrfs caused me any headaches, but feel free to tell me about the advantages of zfs.

For those who would like to see it, here’s my config (it’s not well organized at all)

It’s not that btrfs is bad (even tho very recently, I migrated back to zfs since btrfs got corrupted…), but you are trying to get a very complex filesystem setup when there are options to do it faster and easier while having the same features

when I said headaches, I meant setup headache. I don’t think that zfs has much of an advantage for desktop systems at least, but it is fully featured and simple to setup.

You may find this zfs setup script useful

Ah, so you’re saying I could skip luks, lvm, and btrfs and let zfs take care of it all?

Yep, exactly

These are extra characters so I could send this message :smiley:

One thing you might consider doing is using disko.
It handles partitioning, but also almost all of your stack. There are examples (that double as tests) of luks + btrfs + subvolumes, luks + lvm, so it’s trivial to work out the whole stack (done that, sans tpm2). hibernate goes as a one-liner (at least using swap partition on said lvm). It also doubles as the filesystem part of hardware-config.nix. No idea if it handles tpm though. But, if you want to follow @gytis-ivaskevicius advice and pursue zfs, it’s supported.

The good parts: you don’t have to create partitions etc by yourself, disko handles luks + lvm + btrfs pretty well.

The bad parts: you need to use flakes (hey, that’s less scary than it sounds). And it doesn’t handle changes in partitions well. It’ll probably work if you change the description, reconfigure partitions, luks, lvm & btrfs yourself and just stick with using it as filesystem config reference, it should work. But normally it can only wipe your drive & recreate partitions.

As I’ve said above, I have a setup like this, the only difference with yours is that I use password-based luks (sorry, what’s the point of using luks if it decrypts automatically). I’m not ready to share it (have to get rid of some secrets), but I might be able to help if you need it.

2 Likes

First thought:
Oh right! disko project exists, why arent I’m using it?

Second thought:
Oh right! It seems to require a large config file that I need to learn

Anyways, I will look into it as well. Probably about time

The documentation is lacking IMHO.
There’s no autogenerated options configuration, most probably because the structure itself is recursive (hey, you can have lvm within luks or the other way round, or even lvm within luks within lvm if you’re insane enough).
But the source and examples are relatively readable for me and, assuming you have a mental image of what you want to achieve, it’s rather simple to get this into configuration.

And there’s also nothing really new you put in that config. The script made by OP for creating luks container, LVs and stuff would translate directly into that, the only issue would be where to put what.

My setup (3 partitions, luks on one of them containing lvm with swap and btrfs with 9 subvolumes) is exactly 113 lines excluding comments and empty ones. I could probably make it shorter and less repeatable with a lambda that’d generate stuff, but I’m to lazy for that now.