Custom NixOS Installer - Plug ➜ Install ➜ Play - How to achieve this?

Help
1

Hello NixOS folks! :wave:

I’m stuck and need your help… I’d like to know if :

  • It’s possible to edit the defaults of the NixOS installer? For me to package all my required apps along their configurations, as well as desktop environment configuration.
  • Can it be packaged into an ISO (not a live OS, one that you can install on the target machine)

Screenshot from 2025-03-15 13-38-20
Screenshot from 2025-03-15 13-38-201367×631 46.7 KB

I was thinking that the installer (e.g. Calamares) would be used for the basic setup, including user, password, partitioning, as it is today. And the next step would automatically read your user-specific configuration (e.g. multi-file configuration.nix) and apply it. Afterwards, restart, login and tadaa :tada:

Right after the installation, you’ll get your customized NixOS. No need to run extra commands, or fetching your configurations somewhere else.

I’ve already built several ISO (vm images + standard .iso) and tested them with QEMU, but haven’t found a way to automatically apply my custom configuration which I import in the ISO file (e.g. iso.nix).

I’ve been using nixos-generate and nix-build :

  • nix-shell -p nixos-generators --run "nixos-generate --format iso --configuration ./iso.nix -o result"
  • nix-shell -p nixos-generators --run "nixos-generate --format vm --configuration ./iso.nix -o result"
  • nix-build '<nixpkgs/nixos>' -A vm -I nixpkgs=channel:nixos-24.11 -I nixos-config=./iso.nix

I’m currently running :

  • NixOS
  • with Gnome
  • on ext4 + LUKS
  • with the configuration (split into multiple files) which is the one I’d like to package

Related threads that I’ve already checked, but haven’t understood all the steps yet (which ones are automatic/manual) :

Thanks in advance for your inputs :pray:

PS: Cross-posted on Reddit

4 Likes
2

Hi :wave:

I don’t know if this will answer your questions but when reading your requirements I thought of my configuration, which is basically a copy of this one with some minor modifications.

I’ll walk you in on how I’m installing a system (which is already in my config files):

  1. Run the build-iso.sh Bash script from a Nix-enabled host: it builds a console or Gnome-based ISO of NixOS installer with some modifications
  2. Write the ISO on an USB stick
  3. Stick the USB in the target system and boot from it
  4. Somehow load my GPG key in the live ISO keyring (conveniently, it happens to be stored in an encrypted partition of my USB stick)
  5. Connect to the internet
  6. Run the install-system command, which is a custom Bash script builtin with the ISO: simply put, it uses disko to format my disk, installs NixOS and activates my Home-Manager configuration

This configuration is pretty awesome and I would never thank enough Wimpy for creating it in the first place. It’s been a delight to use since nearly a year!

Hoping to be of any help in your quest… If you have questions, I’ll gladly answer them :slightly_smiling_face:

EDIT: removed references to Ventoy (see aside discussion below on that matter)

2 Likes
3

Please don’t recommend ventoy for nixos. It breaks according to effectively random variables (hashes) contained within the ISO. Yes, I know it often works. I also know it often breaks.

1 Like
4

Indeed I never had any issue with Ventoy, neither with NixOS nor with any other kind of OS for that matter…

What do you recommend instead, then?

1 Like
5

cp foobar.iso /dev/sdX works just fine.

6

Flash the ISO directly to the drive, as it was intended to be used.

7

(as long as you sync after)

1 Like
8

Yeah, no, that won’t do it… I don’t want the ISO to take the whole partition. I have other ISOs there…

To rephrase, what would you recommend to store several ISOs in a single partition of a given USB stick? AFAIK, only Ventoy allows that.

2 Likes
9

I’m sorry I don’t have a better solution for that. NixOS does not reliably support Ventoy, and that’s kind of all there is to it.

10

For the record, the reason Ventoy doesn’t reliably work with NixOS is because of the unspeakable crimes against boot loaders and initramfs that Ventoy commits, including injecting binaries (that are prebuilt and checked into the source repo, btw) and hijacking udev rules. It’s grotesque, and there’s a reason I don’t want to put in the effort it would take to fix this.

5 Likes
11

I get what you’re saying. Thanks for taking the time to explain the why.

I did remove references to Ventoy from my original reply. That wasn’t mandatory anyway, it’s just part of my process and I shared it as is. It will work the same (well, probably even better given what your saying) with dd or cp to a whole USB partition.

Sorry for the little hijack on this side topic @Grraahaam :sweat_smile:

1 Like
12

Weird. I know Ventoy is janky, but I’ve genuinely never encountered a problem with it.

I’ve installed nearly every one of my NixOS (and other) systems from a Ventoy drive.

Do you have any references as to how/why it can be an issue with NixOS? I’d love to learn more about it (and, if it’s going to be an issue for me in the future, find a workaround).

For my use case, I have a 500GB USB drive (technically a M.2 SATA enclosure) that I dump ISOs onto, since it’s logistically more effective and vastly more performant (to ‘flash’, boot, and run) than the jars of assorted thumbdrives it succeeded.

1 Like
13

I’m a little fuzzy on the details, since it’s been over a year since I looked into it. But the basics are that Ventoy works by basically re-implementing the ISO’s bootloader, and then injecting its extra files into the initrd of the system. These files include a small embedded OS’s worth of binaries (these are the prebuilt ones that are checked into the source repo), and some udev rules. Those udev rules trigger when the USB drive is found and synthesize a virtual block device from the underlying ISO file. You’d think it would use a loopback block device for that but no, that would require mounting the file system and that would be too chaotic, apparently. So instead they parse the FAT table to identify which blocks on disk contain the ISO and create a dm-linear device that maps to those blocks. And that’s how initrd is tricked into seeing its ISO as a block device.

The reason this is unreliable is largely because of the udev rules. This is the part I’m fuzzy on. The way it injects the udev rules was something completely insane, and in a file system containing a Nix store, the odds of it injecting them into the right place were not 100%. I think this was exacerbated in my unfinished PR switching the ISO to systemd initrd, because systemd initrd doesn’t shove all the binaries into one directory, so there’s a lot more directories for Ventoy to get tripped up by.

12 Likes
14

Thanks a lot for your inputs @nicolas-goudry, and your detailed step-by-step process (which I may run sooner or later) ! :pray:

I really like the vision of those kind of repos, which also focus on security with disk encryption by default. But it’s sometimes easy to quickly get overwhelmed by the amount of file relations, and the lack of some sort of flow diagram to visually understand what’s going on (not a complain, that’s already great and helpful with the doc).

I guess what I had in mind was easier, I was thinking of decoupling the nix’s user configuration from the nix’s hardware one, to build something which would turn into a machine-agnostic installation.

What I’m trying to find is a way to apply a custom configuration (hardware excluded), in the easiest way possible. Making it possible to grab that config, apply it somewhere else, and expect the same environment, no hassle, no collisions. That’s why I looked into Nix/NixOS the first place :slight_smile:

Or even better, while following the idea of decoupling user/system config from the hardware, it could be great, in the GUI installer (Calamares) or through CLI, to be able to point to a repo (e.g. nix module or flake) or a local file from the installation media. This way, everyone could easily (re)install their configuration (or other ones). It might already be possible tho, still discovering a lot in this community.

Anyway, thanks for sharing and offering your help! I’ll test a few things on my side, and come back to you if ever I’m stuck down the road :pray:

Cheers :v:

PS: No worries for the post hijacking (below discussion), it was worth learning too.

15

Hi again @nicolas-goudry! :nerd_face:

Now that I’ve spent some time analyzing your (well-organized) repo, I’m having some questions :

  1. Could you explain how do you manage your keys used for SOPS? Do you have any backups of the hosts’ keys (i.e. ed25519) + age key? Per hosts?

  2. What’s your process to change passwords/secrets (i.e. users, root)? Do you edit your secrets.yaml files and run the below. If I run passwd and change the user’s password (imperatively), will it be reset on next boot, by reading the hashedPasswordFile? Same goes for wireless network passwords, will they persist (if using impermanence, are they going to be removed)?

  1. Why disable IPv6?

Saw some other thread about it : Disabling IPv6 - `enableIPv6` and `kernelParams` - #6 by nopro404

  1. What’s the purpose of registering flake inputs?
  1. As for now, what do you mean by the below note, since it seems that your HM configs are OK, aren’t they?
  1. Off topic, but why warp is outside the lib.optionals isLinux condition, since it seems to be a gnome-specific application?

Thanks for your time and for sharing your config! :pray:

16

Hey, I’ll try to answer your questions one by one below. Let me know if something is unclear!

Also I have to mention that I’m currently rewriting the documentation in a separate branch, starring a detailed explanation of the installation process with a diagram.

I’m also undergoing a refactor inspired by Wimpy’s units of configuration for apps, features and services which can be consumed by hosts. I’m not very fond of their implementation since it explicitly lists in the app/feature/service on which host it is enabled. I am leaning toward using a dynamic module collection which can then be enabled by hosts. It’s very much a work in progress, I just started today.

Sure, that’s actually documented in the new documentation but I’ll explain it here anyway.

I have a PGP key for my user which I store in two USB keys encrypted with LUKS. I only carry a subkey on my laptop. I followed this blog post to setup everything.

This PGP key is used alongside hosts’ age keys to encrypt all my secrets (which may not fit a multi-user installation). With the current setup, all (public) keys are required to encrypt secrets, but only one private key is required to decrypt. This allows hosts to decrypt secrets on their own.

I do not backup host keys (neither the SSH nor the derived age keys) as they can easily be regenerated.

To change a password, I first run the following command to generate the password hash:

mkpasswd -m SHA-512

Then I edit the sops secret file with:

sops hosts/common/users/nicolas/secrets.yaml

And I just switch my config.

This is working because my PGP public key is in my keyring and the hosts’ entries (age) in .sops.yaml are the actual public keys, contrary to the users’ entries (pgp) which merely references a key (they are only an identifier, not the key material).

This is controlled by the users.mutableUsers option, which is set to false in my configuration. According to the documentation:

If set to false , the contents of the user and group files will simply be replaced on system activation. This also holds for the user passwords; all changed passwords will be reset according to the users.users configuration on activation.

Therefore, you can indeed change the passwords using passwd but they will be reset on next boot according to the configuration.

Wireless network passwords are written to /run/secrets/wifi on each activation. Indeed when using impermanence they will be removed but recreated on activation anyway, so it’s pretty transparent. Without impermanence they will just stay there until they are recreated.

I disable it because I don’t care about nor use it… But it’s pretty much only out of habit.

To be transparent, I don’t know. This is legacy code kept from when my config was based on nix-starter-configs

Yeah, it’s just that I never really completed the documentation and left some sections in TODO although the implementation is there… That’s also why I’m rewriting, updating and completing the documentation in a separate branch :slight_smile:

It’s indeed a gnome app but it’s not gnome-specific. I was about to point you to the derivation showing that all platforms are supported and saw that the package is actually marked as broken on darwin :sweat_smile:

Btw, I didn’t work a lot on the darwin configuration yet so anything related to it can (and actually should) be ignored.

Hope all of this helps!

1 Like
17

Hey @nicolas-goudry !

Thanks for the detailed explanations about all my questions!

Your current and ongoing documentation are great source of learning! You do explain most of the steps, and that’s useful for people like me trying to understand the entire NixOS installation lifecycle!

Since I’m curious, this thread might be the place where I’ll ask you some questions about your config (or what you’ve found out there), until I feel in control of my machine. I hope you don’t mind, take your time, it may also help others coming here :slight_smile:

So, here’s some more concerns I may have and maybe future ideas :

  1. When using Disko for partitioning your disk, could it be useful to label the LUKS partition for example? In order to make the below more generic and reproducible? From :

To something like :

luks.devices."nixos" = {
  device = "/dev/disk/by-label/nixos";
};

Like @hmajid2301 is doing here : disko.devices.disk.main.content.partitions.luks.content.content.extraArgs = ["-L" "nixos" "-f"];

  1. In your disks.nix, this key disko.devices.disk.main.device = "/dev/<device-name>", could maybe take advantage of disko’s --arg option to dynamically provide the device name (haven’t seen what to pass as arg but I guess --arg "disko.devices.disk.main.device" "/dev/<device-name>" could be it?). It could help you to keep the disko config file generic (for your multiple hosts). I may be wrong…
  1. Is it better to let install-system.sh prompt the user for the LUKS encryption password (/tmp/data.passwordFile) or letting disko prompt for it dynamically (when no passwordFile key is given, it automatically prompts you to provide one). Because in either way, the install script will prompt the user if the passwordFile key exists in the disks.nix file.

PS: Funny to see that you’re also using Excalidraw for your diagrams. That’s a great workflow schema you did here, documentation is underrated!

1 Like
18

Thank you very much for your kind words. I’m glad to be of any help!

I definitely don’t mind and am more than happy to share what I learnt so that others can strive in using NixOS!

Actually the host you pointed to is not using disko, that’s why I manually set the configuration for the disks. This is because it’s a Windows dual boot which existed before I started this configuration, so I can’t use disko at all for this host and never bothered writing a disks.sh script to prepare the disk (also, it’s not impermanent).

However, labeling the disk wouldn’t hurt for discovery purposes, I guess… But it wouldn’t change anything to the disko setup, if you look at the g-aero host, which is using disko, there’s no fileSystems option at all.

Hmmm that’s indeed very interesting… I did have a look at disko’s code and you’re definitely right.
In the disko script, we can see here that the two values following --arg or --argstr are added to the nix_args array along with the flag and later on are sent to nixBuild — a function which runs nom-build (from nix-output-monitor) if available or else nix-build — which builds cli.nix. cli.nix then loads the disk format configuration, passing it the lib argument and all other args (including custom ones set through --arg and --argstr) if it’s a function.

Thanks a lot for pointing this out, I’ll try to make it work tonight and let you know how it goes!

I wasn’t even aware that disko would prompt for the encryption password without the passwordFile option set on LUKS content configuration… Less code = happy me!

Also, I saw that disko is now providing disko-install which basically seems to do what I’m doing in my script: prepare disks and install NixOS. I’ll also give it a shot!

Thanks a lot for your invaluable insights. I would never have looked back at this disko configuration on my own… Glad you did!

P.S. : yeah, I’ve got far too much love for Excalidraw :revolving_hearts:

1 Like
19

@Grraahaam Just a quick update: I did implement disko’s arguments to provide the disk path, along with the luks password prompting from disko, but didn’t have time to test it yet. Here is the implementation.

I’ll test it tomorrow and get back to you!

EDIT:

In the end, I had to make two small fixes and I can confirm that everything you stated is true!

1. Encryption passphrase
I removed the disk encryption passphrase prompt from the script and removed the passwordFile attribute from the disko config. Everything is still working fine, it’s just that disko’s doing the prompting now.

2. Disko extra arguments
I implemented two in g-aero’s config: diskDevice and diskName. diskDevice is required while diskName is optional.
In the script, I added a new flag --disk which should point to the disk name output from lsblk. If the value is not provided or invalid, the script looks up all disks names, ids and paths to present them to the user for choosing the right one. It looks something like that:

Available disks  ID                                                                   PATH
/dev/nvme0n1     /dev/disk/by-id/nvme-eui.000000000000001000080d030057150a            /dev/disk/by-path/pci-0000:3d:00.0-nvme-1
/dev/nvme0n1     /dev/disk/by-id/nvme-KXG50PNV2T04_NVMe_KIOXIA_2048GB_X9OA210NKKTL_1  /dev/disk/by-path/pci-0000:3d:00.0-nvme-1
/dev/nvme0n1     /dev/disk/by-id/nvme-KXG50PNV2T04_NVMe_KIOXIA_2048GB_X9OA210NKKTL    /dev/disk/by-path/pci-0000:3d:00.0-nvme-1

I’m not 100% satisfied with this but in the end, it works.

I chose the « easy » road for the script user to select the disk name and then the script maps it to the disk path and uses that value to pass the diskDevice argument to disko (script below is shortened for readability):

disks=$(lsblk -d --noheading --output NAME --paths | sort)
disks_id=$(find /dev/disk/by-id -maxdepth 1 -type l -printf '%p %l\n' | sed 's|\.\./\.\.|/dev|' | sort -k2,2)
disks_path=$(find /dev/disk/by-path -maxdepth 1 -type l -printf '%p %l\n' | sed 's|\.\./\.\.|/dev|' | sort -k2,2)

if [[ "${target_disk}" =~ ^($(echo "${disks}" | tr '\n' '|'))$ ]]; then
  target_disk=$(echo "${disks_path}" | grep -E "${target_disk}$" | head -n1 | cut -d' ' -f1)
fi

run_disko "${clone_dir}/hosts/${target_host}/disks.nix" "disko" "${target_disk}" "nixos"

run_disko() {
  local config="${1}"
  local mode="${2}"
  local disk="${3}"
  local diskName="${4}"
  local disko_args=()

  if [ -n "${disk}" ]; then
    disko_args+=("--argstr" "diskDevice" "${disk}")
  fi

  if [ -n "${diskName}" ]; then
    disko_args+=("--argstr" "diskName" "${diskName}")
  fi

  sudo disko --mode "${mode}" "${disko_args[@]}" "${config}"
}

I’m somehow puzzled with this feature because on one side it makes the disko config basically portable between hosts but on the other side it adds one more flag to the script and the host configuration is less declarative in the end.
I mean, it’s a great feature that you made me discover, but I think I will implement it differently (like generic disko configs imported from hosts with only the diskDevice and diskName set).

Anyway, thanks again for sharing!

1 Like