Custom encrypted installer

Help
1

Hi,

I’m running nix+homemanager on arch for a couple of months (and loving it) and now I explore a “transition path” to nixos.

My situation:

I want an easy & convenient way to install over and over from scratch: an installer USB drive.

I want the installer to be readily prepared with all needed customizations (eg git). In particular, secrets should be included: the drive should be (LUKS?) encrypted so I have to do nothing but enter a password to get access to a ready-to-go environment.

Ideally, build the installer to a disk image in a file: this enables quick build and fast dev iteration with qemu.

My question:

Do you know pointers how to get the encryption going? (I already went through the minimal live CD code).

Thanks!!
:slight_smile:

2

I do have to ask… why? Feel free to tell me I’ve got it all wrong, because of course I don’t know your situation. But I think the desire to reinstall the OS basically goes away almost completely once you start using NixOS. When things get catastrophically messed up, you can trivially rollback without reinstalling, even if it doesn’t boot. Or if that doesn’t work, you can trivially load up a stock ISO and re-run nixos-install, because it’s idempotent and it results in a config update instead of a reinstall. If it’s more about wanting to try a different setup from scratch, I think this largely goes away too. NixOS guarantees that when you boot a generation, you get that generation, and no remnants from a previous one. So just changing the config and doing nixos-rebuild is basically as good as reinstalling and trying the new config from scratch.

3

Hi, and thanks for your time and thought!

Doing this is mainly an exercise in getting more familiar with setting up a nixos installation. I’m aware that I should start with a basic installation and evolve and extend from there, instead of going for such a special case from the start. But wrapping my head around this is a good way to grasp concepts right.

And you made a valuable point that I did not think through before, going for Nixos for the first time
with serious intentions–and not just toying around: namely that you could boot the generic installer and fix the system with nixos-install, tree-creating the same system state that the previous nixos-install call created.

4

This also implies that I have to deploy secrets (the sops-nix master key, in particular) only once, preventing repeated manual overhead :slight_smile:

1 Like
5

Still I wonder: how does the trick with the genetic installer and nixos-install work when your system disk is encrypted? (I’m using a laptop, so encryption is a must.)

6

Just a note, I’m getting the impression you haven’t done an actual NixOS install yet - Outside of using something like disko, you more or less need to do initial disk partitioning, formatting, luksFormat, luksOpen, and mounting, by-hand, for the install.

And similarly, if you need to do some sort of recovery (usually not necessary because of the ability to boot an old generation from your bootloader menu), you would boot the installer media, and luksOpen + mount (/, /boot, /nix, etc) by hand. From there you can fixup your configuration and re-issue nixos-install.

Beyond that info, some thoughts:

  1. pre-LUKS-ing the squashfs is fraught with peril. You’d more or less have to provide the LUKS key as a build input, leaking it into your builder store. You don’t want secrets raw as any build input.
  2. You don’t save much, you’re still going to have to luksOpen, and mount partitions, before you can do recovery/fixup/re-nixos-install. (edit2: you’d have to double luksOpen, for the installer, and then for your laptop disk too)
  3. Again, considering that you don’t want secrets in your raw nix config, or as build inputs, I’m not sure there’s a good way to even achieve shipping secrets in the ISO, even leaving LUKS aside.

Personally, I have a custom ISO that has Tailscale and SSHD enabled by default, with my SSH key trusted. This means I can boot the ISO and then do the unlock/mount/install remotely. Or I can boot it, authenticate to tailscale, and then install it even-more-remotely.

Ultimately, every way I come at answering this question – whether its LUKS-ing the squashfs, or embedding secrets into the image – you’re dancing around a bootstrap/root-of-encryption problem.

Edit: the only times I’ve ever needed to do “installer media recovery” in NixOS came from compounding mistakes - breaking the bootloader from Windows or another Linux install, or by deploying a bad mount configuration and simultaneously overly-aggressively purging old stable generations.

2 Likes
7

Because of the “pure” nature of the Nix language, there is a direct mapping between the code that makes up a NixOS system, and the reified end result. That is, there is no reason to build a USB drive when you can reliably install the same system again and again merely by evaluating the code.

8

Probably going to have to do in a few parts

  1. Build installer iso with the nix store populated
  2. Add to the installer iso an encrypted image of secrets. The encrypted image you’d have to prepare separate
  3. Boot the installer iso per usual.
  4. Mount the image
  5. Do the usual.
  6. Right before nixos-install: move the secrets to the install target

Probably could script some of that got gotta be careful to keep secrets out of the store.

For custom installer I’ve done something like:

Which is a colmena node with a config including everything I want to be in the store already.

Then building the iso:

1 Like
9

I want to elaborate on this point a little. You don’t need a custom iso to install a custom system. I can reinstall my system using the standard installer boot image, or an older one I happen to have handy still on my ventoy usb stick from the last time I needed to install a machine. As per Nix store sqlite db corruption - #6 by uep, I can even reinstall my machine from the running machine. With disko and other additional components, there are fewer and fewer manual steps needed from a blank disk. The result is the same; as a demonstration, running another nixos-rebuild on the system right after first boot is a no-op because the result is the same.

I don’t want to tell you that you shouldn’t build a custom iso, though, if you want. Good reasons for this mostly involve avoiding repeated package downloads from cache; you can pre-seed the store on the iso with the packages you use (or a full system closure of your target, even), and they’ll be used by the installer before going out to the network (as in the linked thread above). Good for repeated builds, installing a fleet of machines, or an automated hardware test, or something that should be basically entirely hands-off.

If that system closure uses agenix, sops-nix, or similar to include encrypted secrets, then they’ll be included too, and copied to the target store just like any other — but this is true regardless of whether you booted a custom installer, or added a local network cache, or built those packages again. For any of these cases, you’ll need to put the relevant decryption key in place (ssh host key, typically) for the pre-encrypted contents to be usable, of course.

But the distinction between these cases is really just where it’s best to start your journey. You don’t need to focus on the installer as the first part of nixos to learn, but if that grabs your interest, go for it.

2 Likes
10

Just want to link these two points for reference :wink:

2 Likes
11

Hey folks, thanks for all your thoughts!

You’re building up that feeling I had from the start, which also drove me to opening this thread: a custom installer is not needed.

Before I created a workaround:

  • use disko for the (encrypted) drives,
  • add git and gpg (via HM)
  • copy over gpg encrypted tar containing secrets (via HM, generated by a simple script)
  • add bootstrap script (writeshellapplication) to decrypt+untar

That one I can install to loopback mounted file and use with qemu or install to flash drive of course.

What bothers me about this workaround is:

  • I need to specify the drive => cannot simply burn file to flash since then on boot the drive is not found
  • somewhat painful gpg+bootstrap dance where a simple password would suffice?

My hope is that those can be solved building on the minimal installer.

You don’t save much, you’re still going to have to luksOpen, and mount partitions, before

My goal was to git clone my repo and disko mount the stuff (need to give the password as a file though).

12

Basically I want to be able to go back and forth between arch and nixos with ease, as a fallback when the first steps on nixos get too complicated. (I need a running reliable machine.)

Both should have minimal system config and the rest done via HM.

Once that’s stable on Nixos, I could clean up: split Nico’s/HM properly, move sops-nix out of HM, flakes maybe, etc

13

If that’s your primary goal, things are much simpler.

It’s not that hard to set up dual boot, especially with zfs (or lvm) so you can reclaim and reuse the space easily once you no longer need one.

You can also get pretty far with a VM (for everything other than very hardware-specific details) or even just reconfiguring the running system on the liveCD(USB) installer. Get your basic config right, desktop environment, apps collection, networking, whatever else. Get used to using nixos. Then use that as the basis of your on-hardware config.

14

Yes, I think using a VM is the most convenient option without a custom installer. :+1:

The installer should also serve as an easy fallback/repair-mechanism should some of the hardware fail. Eg to switch to a new laptop:

  • Boot from stick
  • git clone (git, ssh keys for gitea, etc all in the encrypted stick)
  • re-generate hardware config
  • nixos-install (also stating the laptop enc password)
  • copy over sops-nix key (it’s in the encrypted stick)
  • reboot
15

Again, nothing wrong with wanting to build this, other than it’s more work than seems necessary in a general use case.

I do the same thing, using a generic iso. For access to git etc, I ssh to the running installer image from another machine, with agent-forwarding. If I need the host key, that comes from backups, along with user data.

If you don’t have a second machine handy, a prepared installer seems more appealing. But I’d still suggest that working backups for user data and a general installer (that you don’t have to regenerate and re-write to usb regularly) is a better next step.

16

agent-forwarding to the installer is nice! :heart_eyes: Also, backups are in place.

I would have another device available, but wanted to make this “educational stretch” and find out if/how easy it’s possible.

Seems not as straight-forward as I hoped, hence I’ll better consider other transition paths (e.g. the suggested VM).




Anyways: Thanks everyone for the warm welcome and all your considerations. Takes me one step further into the world of NixOS, I guess :slight_smile:

1 Like
17

Hi.

To wrap this up somewhat, here’s the current approximation to what I had in mind in the beginning.

It’s a modified minimal installer that will automatically

  • ask you the encryption key of an archive of SSH keys for git
  • unpack it
  • clone the repo with the NixOS conf.

After this bootstrapping, off you go with installing a new machine :slight_smile:

iso.nix

{ config, pkgs, ... }:
with pkgs;
let
  # ...

  myBootstrapFile = ".my-bootstrap.tar.gz.gpg";
  myBootstrap = (writeShellApplication {
    name = "myBootstrap";
    runtimeInputs = [
      gnupg
      gnutar
      git
    ];
    text = ''
      set -e
      set -o pipefail

      cd ~

      echo Enter bootstrap password:
      gpg --decrypt ${myBootstrapFile} | tar xf -

      git clone ...
    '';
  });
in
{
  # ...

  imports = [
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
    "${(import ../nix/sources.nix).home-manager}/nixos"
  ];

  # password prompt
  programs.gnupg.agent = {
    enable = true;
    pinentryFlavor = "tty";
  };

  home-manager.users.nixos = {
    home.stateVersion = "23.11";

    home.packages = [ git ];

    programs.bash.enable = true;
    programs.bash.profileExtra = "${myBootstrap}/bin/myBootstrap";

    home.file."${myBootstrapFile}".source = ./. + "/../${myBootstrapFile}";

    programs.ssh = # ...
  };
}

build-installer.sh

#!/usr/bin/env sh

set -e

MY_BOOTSTRAP_FILE="$(pwd)/.my-bootstrap.tar.gz.gpg"

if [ ! -f "${MY_BOOTSTRAP_FILE}" ]
then
    (
        cd ~
        export GPG_TTY=$(tty)
        tar --create --dereference -f - THE_KEYS | gpg --symmetric --output "${MY_BOOTSTRAP_FILE}" -
    )
fi

nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=installer/iso.nix

Cheers. :slight_smile:

1 Like