I’m running nix+homemanager on arch for a couple of months (and loving it) and now I explore a “transition path” to nixos.
My situation:
I want an easy & convenient way to install over and over from scratch: an installer USB drive.
I want the installer to be readily prepared with all needed customizations (eg git). In particular, secrets should be included: the drive should be (LUKS?) encrypted so I have to do nothing but enter a password to get access to a ready-to-go environment.
Ideally, build the installer to a disk image in a file: this enables quick build and fast dev iteration with qemu.
My question:
Do you know pointers how to get the encryption going? (I already went through the minimal live CD code).
I do have to ask… why? Feel free to tell me I’ve got it all wrong, because of course I don’t know your situation. But I think the desire to reinstall the OS basically goes away almost completely once you start using NixOS. When things get catastrophically messed up, you can trivially rollback without reinstalling, even if it doesn’t boot. Or if that doesn’t work, you can trivially load up a stock ISO and re-run nixos-install, because it’s idempotent and it results in a config update instead of a reinstall. If it’s more about wanting to try a different setup from scratch, I think this largely goes away too. NixOS guarantees that when you boot a generation, you get that generation, and no remnants from a previous one. So just changing the config and doing nixos-rebuild is basically as good as reinstalling and trying the new config from scratch.
Doing this is mainly an exercise in getting more familiar with setting up a nixos installation. I’m aware that I should start with a basic installation and evolve and extend from there, instead of going for such a special case from the start. But wrapping my head around this is a good way to grasp concepts right.
And you made a valuable point that I did not think through before, going for Nixos for the first time
with serious intentions–and not just toying around: namely that you could boot the generic installer and fix the system with nixos-install, tree-creating the same system state that the previous nixos-install call created.
Still I wonder: how does the trick with the genetic installer and nixos-install work when your system disk is encrypted? (I’m using a laptop, so encryption is a must.)
Just a note, I’m getting the impression you haven’t done an actual NixOS install yet - Outside of using something like disko, you more or less need to do initial disk partitioning, formatting, luksFormat, luksOpen, and mounting, by-hand, for the install.
And similarly, if you need to do some sort of recovery (usually not necessary because of the ability to boot an old generation from your bootloader menu), you would boot the installer media, and luksOpen + mount (/, /boot, /nix, etc) by hand. From there you can fixup your configuration and re-issue nixos-install.
Beyond that info, some thoughts:
pre-LUKS-ing the squashfs is fraught with peril. You’d more or less have to provide the LUKS key as a build input, leaking it into your builder store. You don’t want secrets raw as any build input.
You don’t save much, you’re still going to have to luksOpen, and mount partitions, before you can do recovery/fixup/re-nixos-install. (edit2: you’d have to double luksOpen, for the installer, and then for your laptop disk too)
Again, considering that you don’t want secrets in your raw nix config, or as build inputs, I’m not sure there’s a good way to even achieve shipping secrets in the ISO, even leaving LUKS aside.
Personally, I have a custom ISO that has Tailscale and SSHD enabled by default, with my SSH key trusted. This means I can boot the ISO and then do the unlock/mount/install remotely. Or I can boot it, authenticate to tailscale, and then install it even-more-remotely.
Ultimately, every way I come at answering this question – whether its LUKS-ing the squashfs, or embedding secrets into the image – you’re dancing around a bootstrap/root-of-encryption problem.
Edit: the only times I’ve ever needed to do “installer media recovery” in NixOS came from compounding mistakes - breaking the bootloader from Windows or another Linux install, or by deploying a bad mount configuration and simultaneously overly-aggressively purging old stable generations.
Because of the “pure” nature of the Nix language, there is a direct mapping between the code that makes up a NixOS system, and the reified end result. That is, there is no reason to build a USB drive when you can reliably install the same system again and again merely by evaluating the code.
I want to elaborate on this point a little. You don’t need a custom iso to install a custom system. I can reinstall my system using the standard installer boot image, or an older one I happen to have handy still on my ventoy usb stick from the last time I needed to install a machine. As per Nix store sqlite db corruption - #6 by uep, I can even reinstall my machine from the running machine. With disko and other additional components, there are fewer and fewer manual steps needed from a blank disk. The result is the same; as a demonstration, running another nixos-rebuild on the system right after first boot is a no-op because the result is the same.
I don’t want to tell you that you shouldn’t build a custom iso, though, if you want. Good reasons for this mostly involve avoiding repeated package downloads from cache; you can pre-seed the store on the iso with the packages you use (or a full system closure of your target, even), and they’ll be used by the installer before going out to the network (as in the linked thread above). Good for repeated builds, installing a fleet of machines, or an automated hardware test, or something that should be basically entirely hands-off.
If that system closure uses agenix, sops-nix, or similar to include encrypted secrets, then they’ll be included too, and copied to the target store just like any other — but this is true regardless of whether you booted a custom installer, or added a local network cache, or built those packages again. For any of these cases, you’ll need to put the relevant decryption key in place (ssh host key, typically) for the pre-encrypted contents to be usable, of course.
But the distinction between these cases is really just where it’s best to start your journey. You don’t need to focus on the installer as the first part of nixos to learn, but if that grabs your interest, go for it.
Basically I want to be able to go back and forth between arch and nixos with ease, as a fallback when the first steps on nixos get too complicated. (I need a running reliable machine.)
Both should have minimal system config and the rest done via HM.
Once that’s stable on Nixos, I could clean up: split Nico’s/HM properly, move sops-nix out of HM, flakes maybe, etc
If that’s your primary goal, things are much simpler.
It’s not that hard to set up dual boot, especially with zfs (or lvm) so you can reclaim and reuse the space easily once you no longer need one.
You can also get pretty far with a VM (for everything other than very hardware-specific details) or even just reconfiguring the running system on the liveCD(USB) installer. Get your basic config right, desktop environment, apps collection, networking, whatever else. Get used to using nixos. Then use that as the basis of your on-hardware config.
Again, nothing wrong with wanting to build this, other than it’s more work than seems necessary in a general use case.
I do the same thing, using a generic iso. For access to git etc, I ssh to the running installer image from another machine, with agent-forwarding. If I need the host key, that comes from backups, along with user data.
If you don’t have a second machine handy, a prepared installer seems more appealing. But I’d still suggest that working backups for user data and a general installer (that you don’t have to regenerate and re-write to usb regularly) is a better next step.
#!/usr/bin/env sh
set -e
MY_BOOTSTRAP_FILE="$(pwd)/.my-bootstrap.tar.gz.gpg"
if [ ! -f "${MY_BOOTSTRAP_FILE}" ]
then
(
cd ~
export GPG_TTY=$(tty)
tar --create --dereference -f - THE_KEYS | gpg --symmetric --output "${MY_BOOTSTRAP_FILE}" -
)
fi
nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=installer/iso.nix