Best way to add LUKS and LVM after install is done, with advanced setup?

ElvishJerricco · August 17, 2022, 1:42pm

@TLATER yea, that’s really difficult even with a proper secure boot chain, which we just aren’t ready for yet. There’s some promising stuff in that direction though, like bootspec.

nyx · August 18, 2022, 7:48am

I followed the installation manual, using a terminal window in the Gnome environment from the official Gnome install-ISO.
When came the step for running nixos-install, it failed.

Here is the disk layout:

NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0          7:0    0   2.2G  1 loop  /nix/.ro-store
sda            8:0    0 119.2G  0 disk  
├─sda1         8:1    0     1G  0 part  /mnt/boot
└─sda2         8:2    0 118.2G  0 part  
  └─ssd      254:0    0 118.2G  0 crypt 
    ├─J-Swap 254:2    0    12G  0 lvm   [SWAP]
    └─J-Root 254:3    0    30G  0 lvm   /mnt
sdb            8:16   0 931.5G  0 disk  
└─sdb1         8:17   0 931.5G  0 part  
  └─hdd      254:1    0 931.5G  0 crypt 
    └─J-Home 254:4    0   100G  0 lvm   /mnt/home
sdc            8:32   0   3.7G  0 disk  
├─sdc1         8:33   0   2.2G  0 part  /iso
└─sdc2         8:34   0     3M  0 part

Here is the configuration:

{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.initrd.luks.devices = {
    ssd = "/dev/disk/by-uuid/3e6a10fb-8862-4c27-91de-fb378a03edb1";
    hdd = "/dev/disk/by-uuid/7d39ec29-d775-4471-a949-dce61670ce50";
  };
  fileSystems."/".device = pkgs.lib.mkForce "/dev/mapper/J-Root";
  fileSystems."/home".device = "/dev/mapper/J-Home";
  networking.hostName = "junior"; # Define your hostname.
  networking.networkmanager.enable = true;  # Easiest to use and most distros use this by default.
  time.timeZone = "Europe/Paris";
  i18n.defaultLocale = "fr_FR.UTF-8";
  console = {
    font = "Lat2-Terminus16";
    keyMap = "fr";
    useXkbConfig = true; # use xkbOptions in tty.
  };
  services.xserver.enable = true;
  services.xserver.displayManager.gdm.enable = true;
  services.xserver.desktopManager.gnome.enable = true;
  environment.gnome.excludePackages = [ pkgs.gnome.totem ];
  services.gnome.tracker-miners.enable = false;
  services.gnome.tracker.enable = false;
  services.xserver.layout = "fr";
  services.xserver.videoDrivers = [ "modesetting" ];
  services.xserver.useGlamor = true;
  hardware.opengl.driSupport32Bit = true;
  services.printing.enable = true;
  sound.enable = true;
  services.xserver.libinput.enable = true;
  users.users = {
    yves = {
      isNormalUser = true;
      extraGroups = [ "networkmanager" "wheel" ]; # Enable ‘sudo’ for the user.
      packages = with pkgs; [
        podman
      ];
    };
    iris = {
      isNormalUser = true;
      extraGroups = [ "networkmanager" "wheel" ]; # Enable ‘sudo’ for the user.
      packages = with pkgs; [
        fanficfare
      ];
    };
  };
  nixpkgs.config.allowUnfree = true;
  environment.systemPackages = with pkgs; [
    firefox
    libreoffice-fresh
    nextcloud-client
    openssh
    sshfs
    thunderbird
    vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
  ];
  services.flatpak.enable = true;
  services.openssh.enable = true;
  networking.firewall.allowedTCPPorts = [ 22 ];
  system.stateVersion = "22.05"; # Did you read the comment?
}

Here is the output of nixos-install:

[root@nixos:~]# nixos-install 
copying channel...
building the configuration in /mnt/etc/nixos/configuration.nix...
warning: dumping very large path (> 256 MiB); this may run out of memory
/run/current-system/sw/bin/nixos-install: line 178:  5500 Killed                  nix-build --out-link "$outLink" --store "$mountPoint" "${extraBuildFlags[@]}" --extra-substituters "$sub" '<nixpkgs/nixos>' -A system -I "nixos-config=$NIXOS_CONFIG" "${verbosity[@]}"

When run in verbose mode, here are the lines before the warning:

[…]
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/applications/virtualization/podman/default.nix'
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/development/go-modules/generic/default.nix'
evaluating file '/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/lib/systemd-lib.nix'
evaluating file '/dev/sdb1'
warning: dumping very large path (> 256 MiB); this may run out of memory
^Cerror: interrupted by the user

TLATER · August 18, 2022, 1:38pm

This means that nix is reading the entire contents of /dev/sdb1. Evaluating it as a nix file, no less.

Your config has no reference to it, anything in hardware-configuration.nix?

nyx · August 18, 2022, 3:49pm

Here is the hardware-configuration.nix file, minus the comments:

{ config, lib, pkgs, modulesPath, ... }:
{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];
  boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "sd_mod" "rtsx_usb_sdmmc" ];
  boot.initrd.kernelModules = [ "dm-snapshot" ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];
  fileSystems."/" =
    { device = "/dev/disk/by-uuid/a78f1831-84b1-4790-80f5-acdadc35011a";
      fsType = "ext4";
    };
  fileSystems."/home" =
    { device = "/dev/disk/by-uuid/b1aa3b2e-36ed-4689-9a31-03362e56d2f3";
      fsType = "ext4";
    };
  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/2890-0666";
      fsType = "vfat";
    };
  swapDevices =
    [ { device = "/dev/disk/by-uuid/5499069c-ddf8-45c3-9bcd-16da2064ed92"; }
    ];
  networking.useDHCP = lib.mkDefault true;
  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

nyx · August 18, 2022, 5:41pm

I’ve been thinking: I could get /dev/sdb out of the picture entirely (removing the drive if necessary), and install with only /dev/sda.

Am I right to assume, that to reach my target, these would be the operations to run on the running NixOS?

luksOpen /dev/sdb1 into “hdd”;
re-add /dev/mapper/hdd to the VG (“J”);
re-create /dev/mapper/J-Home and mount it to a temporary place (e.g. /mnt/H);
mv /home/* /mnt/H/
umount J-Home from /mnt/H and remount it on /home;
re-add /dev/sdb1 to the list of LUKS devices in configuration.nix, and maybe set again the exact device (the LV) of /home;
run nixos switch (IIRC…).
=> At this point, I expect NixOS to detect the new configuration, and generate a correct hardware-configuration.nix file, allowing me to reboot to target.

ElvishJerricco · August 18, 2022, 6:41pm

You’re missing the broader problem. Somewhere a mistake has been made, and the code in your Nix config is treating your raw drive as a file containing Nix code. This should not happen. The presence of the drive is not the reason this happened.

nyx · August 18, 2022, 8:56pm

First, I really want to thank everyone! I did not expect such friendly help so fast, by so many people

I am not missing the broader problem, just trying to circumvent it.

Indeed, I am currently running an official LiveCD, for which I checked the sha256 checksum. I consider it untainted. After it booted, I only did that (as root):

created a new GPT layout on sda and sdb (as described above);
ran cryptsetup on sda2 and sdb1, opened them and made them PVs;
created a VG out of the 2 PVs, then LVs for Root, Home, and Swap;
mounted the volumes under /mnt;
generated a first configuration as instructed by the official documentation;
tweaked configuration.nix based on the documentation for misc. packages;
ran nixos-install.

As you can see, sdb1 appears nowhere in the configuration, yet it is loaded as if it were a nix file

Without feeling overconfident, I think I can safely say that the mistake is not on my side (since nobody seems to see any problem with the configuration I posted); and I don’t blame anyone because as a programmer I know bugs happen (and PEBCAK too, still in a corner of my mind ), and besides it is voluntary work in open-source developers’ free time.
But I feel like I have no hold on the current issue, and I need to move forward…

TLATER · August 19, 2022, 2:03am

Yep, I think by the lack of replies you might note that all of us are a bit stumped (or at least I am), and I think some of the others in this thread are literally the world’s foremost experts on nix. You’ve hit one of the weirdest issues I’ve seen.

I’m incredibly frustrated by this, because nix is awesome and I hope that anyone who tries it can get a feel for that, but this is a terrible experience for you and I have neither seen it nor any idea how this can even happen!

There was a user recently with a similar problem where a many-GB file suddenly appeared in their evaluation, wonder if you’re having the same problem. Maybe it is a bug. Did you start from an empty /mnt? Probably…

Anyway, I think there are two potential paths for figuring out what’s going on without getting too technical:

Reduce the configuration file to the bare minimum, and see if it still breaks

i.e., system.stateVersion, the boot loader and / file system

Remove /dev/sdb without changing your configuration at all and see what breaks

In the former case, if this fixes it, you can add back options bit by bit and narrow down which one causes it. Once you know what causes it, figuring out where that read is coming from is easier.

In the latter, whatever error occurs might be more specific than what you’re seeing now.

I’d definitely go down the route of debugging this a bit more, rather than trying to find a “workaround”. This is NixOS - your system is built declaratively, whatever weird problem this is will crop up again even if you find some horrifying workaround in the mean time, unless something is truly horribly corrupted.

Other things to try:

nix-channel --update
- And in fact, nix-channel --list, most weird problems I’ve seen are caused by broken channel configuration. No idea how you’d end up with that though.
- In a similar vein, nix-env -q
nix-collect-garbage -d in case something weird ended up in the store somehow
[nix-store --verify](https://nixos.org/manual/nix/stable/command-ref/nix-store.html#operation---verify) in case something weird ended up in the store but is part of the running image

You could also try some interactive evaluation with nix repl, and when you’re desperate enough, there’s always strace. But hopefully that won’t be necessary.

uep · August 19, 2022, 2:44am

Very strange. Is there any chance that some other file has wound up as a symlink to /dev/sdb?

Try something like find / -type l -ls |& egrep '/sdb$' just in case?

nyx · August 19, 2022, 7:04am

I ran find / -lname '*/sdb*' -ls 2>/dev/null and got nothing unusual:

/run/udev/links/disk\\x2fby-partuuid\\x2fdf759de7-d8a1-4b51-a91e-8e4c1c29ded0/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-id\\x2fwwn-0x5000c500a8e2abb6-part1/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-id\\x2fata-ST1000LM035-1RK172_WDE739WF-part1/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-path\\x2fpci-0000:00:17.0-ata-2.0-part1/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-path\\x2fpci-0000:00:17.0-ata-2-part1/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-uuid\\x2f7d39ec29-d775-4471-a949-dce61670ce50/b8:17 -> 0:/dev/sdb1
/run/udev/links/disk\\x2fby-id\\x2fwwn-0x5000c500a8e2abb6/b8:16 -> 0:/dev/sdb
/run/udev/links/disk\\x2fby-path\\x2fpci-0000:00:17.0-ata-2.0/b8:16 -> 0:/dev/sdb
/run/udev/links/disk\\x2fby-path\\x2fpci-0000:00:17.0-ata-2/b8:16 -> 0:/dev/sdb
/run/udev/links/disk\\x2fby-id\\x2fata-ST1000LM035-1RK172_WDE739WF/b8:16 -> 0:/dev/sdb
/dev/disk/by-uuid/7d39ec29-d775-4471-a949-dce61670ce50 -> ../../sdb1
/dev/disk/by-partuuid/df759de7-d8a1-4b51-a91e-8e4c1c29ded0 -> ../../sdb1
/dev/disk/by-path/pci-0000:00:17.0-ata-2-part1 -> ../../sdb1
/dev/disk/by-path/pci-0000:00:17.0-ata-2.0-part1 -> ../../sdb1
/dev/disk/by-path/pci-0000:00:17.0-ata-2 -> ../../sdb
/dev/disk/by-path/pci-0000:00:17.0-ata-2.0 -> ../../sdb
/dev/disk/by-id/wwn-0x5000c500a8e2abb6-part1 -> ../../sdb1
/dev/disk/by-id/ata-ST1000LM035-1RK172_WDE739WF-part1 -> ../../sdb1
/dev/disk/by-id/ata-ST1000LM035-1RK172_WDE739WF -> ../../sdb
/dev/disk/by-id/wwn-0x5000c500a8e2abb6 -> ../../sdb
/dev/block/8:17 -> ../sdb1
/dev/block/8:16 -> ../sdb
/sys/class/block/sdb -> ../../devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb
/sys/class/block/sdb1 -> ../../devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb1
/sys/devices/virtual/block/dm-1/slaves/sdb1 -> ../../../../pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb1
/sys/dev/block/8:16 -> ../../devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb
/sys/dev/block/8:17 -> ../../devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb1
/sys/block/sdb -> ../devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb

nyx · August 19, 2022, 7:12am

After reducing the configuration to:

{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.initrd.luks.devices = {
    ssd = "/dev/disk/by-uuid/3e6a10fb-8862-4c27-91de-fb378a03edb1";
    hdd = "/dev/disk/by-uuid/7d39ec29-d775-4471-a949-dce61670ce50";
  };
  system.stateVersion = "22.05"; # Did you read the comment?
}

I got the same result.
I’ll try TLATER’s house-cleaning suggestions today.

nyx · August 19, 2022, 7:22am

Interesting fact: Even though it wouldn’t have worked (because the VG would have missed one PV), I tried disabling the 2nd LUKS line, then launched the install again.
This time, I saw:

[…]
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/tools/package-management/nix/default.nix'
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/tools/package-management/nix/common.nix'
evaluating file '/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/lib/systemd-lib.nix'
evaluating file '/dev/sda2'
warning: dumping very large path (> 256 MiB); this may run out of memory
^Cerror: interrupted by the user

Now it is sda2 instead of sdb1. So the issue seems to be linked to LUKS somehow.

Sandro · August 19, 2022, 7:26am

You are using boot.initrd.luks.devices wrong. After the name there should be another devices. NixOS Search

nyx · August 19, 2022, 7:52am

Ha ha! I knew I couldn’t completely discard PEBCAK I’ll fix that right away.

Atemu · August 19, 2022, 5:18pm

Why would that try to dump a disk though and not simply error out? I’d consider that a bug.

uep · August 19, 2022, 11:00pm

nix has some ‘fun’ string interpolation magic for paths, which I was suspicious of but couldn’t spot where it would be coming in to play. Well done Sandro.

ElvishJerricco · August 19, 2022, 11:34pm

I don’t think this is due to string interpolation, because there was no string interpolation. What seems to be happening IMO is when you set boot.initrd.luks.devices.ssd = "/something/that/looks/like/a/path";, the NixOS module system sees that path as a module, not a configuration definition, because the type of boot.initrd.luks.devices is attrsOf (submodule ...). So it tries to read it as Nix.

I’m really glad someone noticed the problem because I was dumbfounded

uep · August 20, 2022, 1:54am

yah … that sounds a lot like string interpolation, which is what I was looking for and not seeing. It turns out to not technically be because of the feature with that name, but actually something … nastier.

TLATER · August 20, 2022, 3:19am

Not to derail this too much, but this thread is an excellent example of situations in which strict typing clearly helps even the most experienced developers.

Except Sandro, apparently

nyx · August 21, 2022, 2:51pm

Thank you all again. Now NixOS is running fine, and I am discovering it.
I have lots of questions, but that’ll go in other discussion threads, and I’ll do some searches on my own first.