I installed NixOS yesterday for the first time to get the feel of it, and I intend to use it permanently on this laptop.
I know my way with Archlinux, Debian… but I have never used Nix (or Guix or anything similar). I have never used LUKS either.
The laptop contains nothing else: no other OS, no personal data for now, nothing of value. It has disks sda (smallish SSD) and sdb (big HDD).
What I want to achieve
I want encryption to protect my data against common theft, not to resist secret-services investigations, so I intend:
to encrypt personal data on the HDD,
not to encrypt OS data and cache data on the SSD (but if that helps I would accept to encrypt everything).
I will use part of the SSD as cache for slower data (LVM cache; and fscache for caching NFS), which is something I have already done in Archlinux.
In theory (since I have never used LUKS), I envision something like that:
LV “Fscache” ext4 on PVA, mounted on /var/cache/fscache
LV “Home” ext4 on PVB, LVM-cached on PVA, and mounted on /home
My problem
In NixOS, I see that none of these are handled as they are in “procedural distros” like Archlinux:
the boot-loader options (to ensure LUKS and LVM support, with the right partitions in the right places)
the fstab file,
the partitions…
So what is the easiest way to switch my current vanilla NixOS on ext4 on SSD, to the full setup I described above?
I can reinstall from scratch using the official Gnome Installer on USB, if needed.
I would caution you that anything you intend to store encrypted should not be cached or swapped on unencrypted disks. So I would recommend using LUKS on all disks. If you’re concerned about having to enter multiple passwords, NixOS will attempt to reuse the same password on all LUKS disks, so you should only have to enter it once.
As for switching to this setup, I believe there are ways to convert your disk formats in-place without reinstalling the OS, but since you don’t have any meaningful data on them, I wouldn’t bother. It’s much easier to just jump back in the installer and do it from scratch. With just LVM and LUKS in the picture, NixOS should be able to automatically detect the disk topology when you run nixos-generate-config and setup hardware-configuration.nix correctly for you (though I can’t say for sure whether it handles LVM cache correctly).
Oh, also, you may regret making your ESP as small as 512M. NixOS keeps old generations of your OS around in the boot menu until you ask it to delete them, and this means you can end up with many kernel/initramfs files on your ESP. And when it fills up, it’s not the smartest about handlling that and you end up having to do some manual cleanup work.
Thank you ElvishJerrico for the hints and advice.
When I installed, at no point did I get the opportunity to run commands such as nixos-generate-config, so I assume you suggest I use the install media available through “Download (64bit)” instead of “Download (GNOME, 64bit)” that I used previously.
Also, I’ll take 1GiB for the ESP
You did, you just had to open a terminal and use it
The GUIded installer is a recent addition and not only that most long time users are not even aware of it yet, those that are aware, propose to not use it due to it being unpolished and buggy at some places.
Well worth a read if you want to understand the limitations of disk encryption on Linux systems today But yes, for the time being @ElvishJerricco is completely correct. There’s no practical way to do anything more sophisticated yet.
@TLATER yea, that’s really difficult even with a proper secure boot chain, which we just aren’t ready for yet. There’s some promising stuff in that direction though, like bootspec.
I followed the installation manual, using a terminal window in the Gnome environment from the official Gnome install-ISO.
When came the step for running nixos-install, it failed.
Here is the disk layout:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 2.2G 1 loop /nix/.ro-store
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 1G 0 part /mnt/boot
└─sda2 8:2 0 118.2G 0 part
└─ssd 254:0 0 118.2G 0 crypt
├─J-Swap 254:2 0 12G 0 lvm [SWAP]
└─J-Root 254:3 0 30G 0 lvm /mnt
sdb 8:16 0 931.5G 0 disk
└─sdb1 8:17 0 931.5G 0 part
└─hdd 254:1 0 931.5G 0 crypt
└─J-Home 254:4 0 100G 0 lvm /mnt/home
sdc 8:32 0 3.7G 0 disk
├─sdc1 8:33 0 2.2G 0 part /iso
└─sdc2 8:34 0 3M 0 part
Here is the configuration:
{ config, pkgs, ... }:
{
imports =
[ # Include the results of the hardware scan.
./hardware-configuration.nix
];
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
boot.initrd.luks.devices = {
ssd = "/dev/disk/by-uuid/3e6a10fb-8862-4c27-91de-fb378a03edb1";
hdd = "/dev/disk/by-uuid/7d39ec29-d775-4471-a949-dce61670ce50";
};
fileSystems."/".device = pkgs.lib.mkForce "/dev/mapper/J-Root";
fileSystems."/home".device = "/dev/mapper/J-Home";
networking.hostName = "junior"; # Define your hostname.
networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
time.timeZone = "Europe/Paris";
i18n.defaultLocale = "fr_FR.UTF-8";
console = {
font = "Lat2-Terminus16";
keyMap = "fr";
useXkbConfig = true; # use xkbOptions in tty.
};
services.xserver.enable = true;
services.xserver.displayManager.gdm.enable = true;
services.xserver.desktopManager.gnome.enable = true;
environment.gnome.excludePackages = [ pkgs.gnome.totem ];
services.gnome.tracker-miners.enable = false;
services.gnome.tracker.enable = false;
services.xserver.layout = "fr";
services.xserver.videoDrivers = [ "modesetting" ];
services.xserver.useGlamor = true;
hardware.opengl.driSupport32Bit = true;
services.printing.enable = true;
sound.enable = true;
services.xserver.libinput.enable = true;
users.users = {
yves = {
isNormalUser = true;
extraGroups = [ "networkmanager" "wheel" ]; # Enable ‘sudo’ for the user.
packages = with pkgs; [
podman
];
};
iris = {
isNormalUser = true;
extraGroups = [ "networkmanager" "wheel" ]; # Enable ‘sudo’ for the user.
packages = with pkgs; [
fanficfare
];
};
};
nixpkgs.config.allowUnfree = true;
environment.systemPackages = with pkgs; [
firefox
libreoffice-fresh
nextcloud-client
openssh
sshfs
thunderbird
vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
];
services.flatpak.enable = true;
services.openssh.enable = true;
networking.firewall.allowedTCPPorts = [ 22 ];
system.stateVersion = "22.05"; # Did you read the comment?
}
Here is the output of nixos-install:
[root@nixos:~]# nixos-install
copying channel...
building the configuration in /mnt/etc/nixos/configuration.nix...
warning: dumping very large path (> 256 MiB); this may run out of memory
/run/current-system/sw/bin/nixos-install: line 178: 5500 Killed nix-build --out-link "$outLink" --store "$mountPoint" "${extraBuildFlags[@]}" --extra-substituters "$sub" '<nixpkgs/nixos>' -A system -I "nixos-config=$NIXOS_CONFIG" "${verbosity[@]}"
When run in verbose mode, here are the lines before the warning:
[…]
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/applications/virtualization/podman/default.nix'
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/development/go-modules/generic/default.nix'
evaluating file '/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/lib/systemd-lib.nix'
evaluating file '/dev/sdb1'
warning: dumping very large path (> 256 MiB); this may run out of memory
^Cerror: interrupted by the user
I’ve been thinking: I could get /dev/sdb out of the picture entirely (removing the drive if necessary), and install with only /dev/sda.
Am I right to assume, that to reach my target, these would be the operations to run on the running NixOS?
luksOpen /dev/sdb1 into “hdd”;
re-add /dev/mapper/hdd to the VG (“J”);
re-create /dev/mapper/J-Home and mount it to a temporary place (e.g. /mnt/H);
mv /home/* /mnt/H/
umount J-Home from /mnt/H and remount it on /home;
re-add /dev/sdb1 to the list of LUKS devices in configuration.nix, and maybe set again the exact device (the LV) of /home;
run nixos switch (IIRC…).
=> At this point, I expect NixOS to detect the new configuration, and generate a correct hardware-configuration.nix file, allowing me to reboot to target.
You’re missing the broader problem. Somewhere a mistake has been made, and the code in your Nix config is treating your raw drive as a file containing Nix code. This should not happen. The presence of the drive is not the reason this happened.
First, I really want to thank everyone! I did not expect such friendly help so fast, by so many people
I am not missing the broader problem, just trying to circumvent it.
Indeed, I am currently running an official LiveCD, for which I checked the sha256 checksum. I consider it untainted. After it booted, I only did that (as root):
created a new GPT layout on sda and sdb (as described above);
ran cryptsetup on sda2 and sdb1, opened them and made them PVs;
created a VG out of the 2 PVs, then LVs for Root, Home, and Swap;
mounted the volumes under /mnt;
generated a first configuration as instructed by the official documentation;
tweaked configuration.nix based on the documentation for misc. packages;
ran nixos-install.
As you can see, sdb1 appears nowhere in the configuration, yet it is loaded as if it were a nix file
Without feeling overconfident, I think I can safely say that the mistake is not on my side (since nobody seems to see any problem with the configuration I posted); and I don’t blame anyone because as a programmer I know bugs happen (and PEBCAK too, still in a corner of my mind ), and besides it is voluntary work in open-source developers’ free time.
But I feel like I have no hold on the current issue, and I need to move forward…
Yep, I think by the lack of replies you might note that all of us are a bit stumped (or at least I am), and I think some of the others in this thread are literally the world’s foremost experts on nix. You’ve hit one of the weirdest issues I’ve seen.
I’m incredibly frustrated by this, because nix is awesome and I hope that anyone who tries it can get a feel for that, but this is a terrible experience for you and I have neither seen it nor any idea how this can even happen!
There was a user recently with a similar problem where a many-GB file suddenly appeared in their evaluation, wonder if you’re having the same problem. Maybe it is a bug. Did you start from an empty /mnt? Probably…
Anyway, I think there are two potential paths for figuring out what’s going on without getting too technical:
Reduce the configuration file to the bare minimum, and see if it still breaks
i.e., system.stateVersion, the boot loader and / file system
Remove /dev/sdb without changing your configuration at all and see what breaks
In the former case, if this fixes it, you can add back options bit by bit and narrow down which one causes it. Once you know what causes it, figuring out where that read is coming from is easier.
In the latter, whatever error occurs might be more specific than what you’re seeing now.
I’d definitely go down the route of debugging this a bit more, rather than trying to find a “workaround”. This is NixOS - your system is built declaratively, whatever weird problem this is will crop up again even if you find some horrifying workaround in the mean time, unless something is truly horribly corrupted.
Other things to try:
nix-channel --update
And in fact, nix-channel --list, most weird problems I’ve seen are caused by broken channel configuration. No idea how you’d end up with that though.
In a similar vein, nix-env -q
nix-collect-garbage -d in case something weird ended up in the store somehow
[nix-store --verify](https://nixos.org/manual/nix/stable/command-ref/nix-store.html#operation---verify) in case something weird ended up in the store but is part of the running image
You could also try some interactive evaluation with nix repl, and when you’re desperate enough, there’s always strace. But hopefully that won’t be necessary.
Interesting fact: Even though it wouldn’t have worked (because the VG would have missed one PV), I tried disabling the 2nd LUKS line, then launched the install again.
This time, I saw:
[…]
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/tools/package-management/nix/default.nix'
evaluating file '/nix/store/xjf4mj0ckp4hi47b55qmgmz46qmpzpn4-nixos-22.05.2307.72f492e275f/nixos/pkgs/tools/package-management/nix/common.nix'
evaluating file '/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/lib/systemd-lib.nix'
evaluating file '/dev/sda2'
warning: dumping very large path (> 256 MiB); this may run out of memory
^Cerror: interrupted by the user
Now it is sda2 instead of sdb1. So the issue seems to be linked to LUKS somehow.
), and besides it is voluntary work in open-source developers’ free time.