Nixos hetzner image

tcurdt · February 9, 2025, 2:57pm

Has anyone ever built a NixOS image that can be uploaded to hcloud?

What I am currently doing with Talos is:

download https://github.com/siderolabs/talos/releases/download/v1.9.3/metal-amd64.raw.zst
re-compress it with xz
upload it using hcloud-upload-image

    hcloud-upload-image -v upload \
        --server-type cpx21 \
        --image-path metal-amd64.raw.xz \
        --compression xz \
        --labels "talos=v1.9.3,type=cpx21"

And I now would like to have an image like that for NixOS.

I have cobbled together a unattended install iso that has worked great for some bare metal installs.

And it feels like I am at least half way there.

But I am currently lost in the machinery for the raw version

Has anyone looked into this before?

cheers,
Torsten

tcurdt · February 10, 2025, 12:41pm

Someone else running NixOS on Hetzner cloud? What approach do you use?
Infecting a Debian machine isn’t the very ideal way.

tcurdt · February 11, 2025, 1:00am

What’s causing the don't know how to build these paths?
Any pointers what might be missing here?

disk-image> Installing the system
disk-image> warning: the group 'nixbld' specified in 'build-users-group' does not exist
disk-image> warning: the group 'nixbld' specified in 'build-users-group' does not exist
disk-image> don't know how to build these paths:
disk-image>   /nix/store/chwpfw7ihh7xj1ip1qx1g3grmiqk6n95-nixos-system-nixos-25.05.20250208.a79cfe0
disk-image> error: build of '/nix/store/chwpfw7ihh7xj1ip1qx1g3grmiqk6n95-nixos-system-nixos-25.05.20250208.a79cfe0' failed
disk-image> [    5.223603] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100

MatthieuB · February 11, 2025, 9:15pm

Hi, I did an install there once, I first looked if I could upload a qcow2 image but didn’t find a way, so I simply uploaded the ISO with the NixOS graphical installer, following this doc.

tcurdt · February 12, 2025, 9:52pm

Thanks for the pointer. But a manual install is exactly what I want to avoid.

It seems nixpkgs has some definitions for cloud images (aws, azure,…)

github.com

NixOS/nixpkgs/blob/32a564c6b98610d7d750e22f0416e2889633a342/nixos/modules/virtualisation/azure-image.nix

{
  config,
  lib,
  pkgs,
  ...
}:

with lib;
let
  cfg = config.virtualisation.azureImage;
in
{
  imports = [
    ./azure-common.nix
    ./disk-size-option.nix
    ../image/file-options.nix
    (lib.mkRenamedOptionModuleWith {
      sinceRelease = 2411;
      from = [
        "virtualisation"

This file has been truncated. show original

I thought I just give them a try - but I have no idea where to find them.

I assume they should be available somewhere via hydra? But I didn’t find them here

https://hydra.nixos.org/project/nixpkgs

…or in the next step - how to customize them and build them via my flake.

MatthieuB · February 12, 2025, 10:34pm

Yeah I also wanted to avoid manual install, on other providers (exoscale, infomaniak public cloud) I could use make-disk-image to create an “ready to boot” qcow2 image (with cloud-init enabled), but never used those cloud images definitions. Hope you’ll find a way.

edit: I think you could use make-disk-image to build a raw image and upload it with this tool

tcurdt · February 13, 2025, 12:05am

It seems like getting make-disk-image to build and then customized would be the first 2 steps for me.

But I haven’t found much more than

github.com

NixOS/nixpkgs/blob/3c3abaa64db44e50b571e3832579408cf0905ffb/nixos/lib/make-disk-image.nix

/*
  Technical details

  `make-disk-image` has a bit of magic to minimize the amount of work to do in a virtual machine.

  It relies on the [LKL (Linux Kernel Library) project](https://github.com/lkl/linux) which provides Linux kernel as userspace library.

  The Nix-store only image only need to run LKL tools to produce an image and will never spawn a virtual machine, whereas full images will always require a virtual machine, but also use LKL.

  ### Image preparation phase

  Image preparation phase will produce the initial image layout in a folder:

  - devise a root folder based on `$PWD`
  - prepare the contents by copying and restoring ACLs in this root folder
  - load in the Nix store database all additional paths computed by `pkgs.closureInfo` in a temporary Nix store
  - run `nixos-install` in a temporary folder
  - transfer from the temporary store the additional paths registered to the installed NixOS
  - compute the size of the disk image based on the apparent size of the root folder
  - partition the disk image using the corresponding script according to the partition table type

This file has been truncated. show original

That’s the tool I mentioned in the original post
It works great with Talos and I would love to have an image for NixoOS, too.

MatthieuB · February 13, 2025, 12:28am

Here is an example flake.nix, you can build the image with nix build .#qcow2

{
  description = "A NixOS base image for QEMU using cloud-init";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-24.11";
  };

  outputs = { self, nixpkgs }:
  let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages.${system};
    inherit (pkgs) lib;

    baseModule = { lib, config, pkgs, ...}: {
      nixpkgs.hostPlatform = "x86_64-linux";
      imports = [
        "${nixpkgs}/nixos/modules/profiles/qemu-guest.nix" 
        # "${nixpkgs}/nixos/modules/virtualisation/azure-image.nix"
      ];

      boot.loader.grub.device = "/dev/sda";

      fileSystems."/" = {
        label = "nixos";
        fsType = "ext4";
      };

      networking.useDHCP = false;

      services = {
        cloud-init = {
          enable = true;
          network.enable = true;
          config = ''
            system_info:
              distro: nixos
              network:
                renderers: [ 'networkd' ]
            datasource_list: [ "Exoscale" ]
            cloud_init_modules:
              - migrator
              - seed_random
              - growpart
              - resizefs
            cloud_config_modules:
              - disk_setup
              - mounts
            cloud_final_modules: []
          '';
        };

        qemuGuest.enable = true;

        openssh = {
          enable = true;
          settings.PasswordAuthentication = false;
          settings.KbdInteractiveAuthentication = false;
        };
      };

      users.users.root.openssh.authorizedKeys.keys = [
        # ...
      ];
    };

    nixos = nixpkgs.lib.nixosSystem {
      modules = [ baseModule ];
    };

    make-disk-image = import "${nixpkgs}/nixos/lib/make-disk-image.nix";

  in {
    inherit pkgs;
    qcow2 = make-disk-image {
      inherit pkgs lib;
      config = nixos.config;
      name = "nixos-cloudinit";
      format = "qcow2-compressed";
      copyChannel = false;
      additionalSpace = "10G";
    };
  };
}

tcurdt · February 19, 2025, 12:10am

Thanks a lot, @MatthieuB

Unfortunately my nix language skills are failing me to integrate that
Some aspects of this language are just puzzling. I think I will never become a fan of that aspect of nixOS.

  outputs =
    inputs:
    let
      # pkgs = nixpkgs.legacyPackages.${system};
      # inherit (pkgs) lib;
      make-disk-image = import "${inputs.nixpkgs}/nixos/lib/make-disk-image.nix";
    in
    {
      nixosConfigurations = {

        nixos-x86 = inputs.nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules = [
            # inputs.disko.nixosModules.disko
            ./configuration.nix
            # ./make-disk-image.nix
          ];
          # disko.devices.disk.main.device = "/dev/sda";
        };

        disk-x86 = make-disk-image {
          inherit pkgs lib; # where should these come from?
          config = inputs.self.nixosConfigurations.nixos-x86.config;
          name = "nixos-cloud-x86";
          format = "qcow2-compressed";
          copyChannel = false;
          additionalSpace = "10G";
        };

      };
    };

Anyone pointers on how to fix this?

573 · February 19, 2025, 7:06am

baseModule in ./configuration.nix and nix build .#disk-x86 did not work ?

raboof · February 19, 2025, 9:58am

I’m running a few machines on Hetzner cloud that I update ‘on-the-fly’ ‘from outside’ with nixos-rebuild --target-host ... switch

The initial deployment goes like this:

Create the nix definition of the machine (importing the qemu-guest.nix and using disko for the partition layout, and in my case agenix for secrets)
Create a machine with the default Ubuntu image
Add the IP to DNS
Get the machine’s public key with ssh-keyscan <IP> and add it to the local agenix config so the secrets become available to the machine
Initial deploy with nixos-anywhere --copy-host-keys root@<IP>

After this subsequent updates can be pushed with nixos-rebuild --target-host ... switch

I agree it’s a bit of a weird bootstrap step, but nixos-anywhere at least hides it fairly nicely. Apparently hcloud-upload-image does something somewhat similar (though it’s nice it creates the machine for you and uses the rescue system). Of course there’s additional trade-offs here, for example you might prefer to bring your own host key instead of relying on the one provisioned by Hetzner.

tcurdt · February 19, 2025, 10:43pm

./make-disk-image.nix is basically where I moved the baseModule to.

github.com

tcurdt/nixos-unattended-install-iso/blob/79b46b2b928ffb449ca5d9a393015854b00135e3/make-disk-image.nix

{
  # config,
  pkgs,
  # lib,
  # modulesPath,
  # targetSystem,
  ...
}:
let
in
{

  # why is this needed? the platform should come from the nixos config
  nixpkgs.hostPlatform = "x86_64-linux";

  imports = [
    "${pkgs}/nixos/modules/profiles/qemu-guest.nix"
    # "${nixpkgs}/nixos/modules/virtualisation/azure-image.nix"
  ];

This file has been truncated. show original

…but that gives me an infinite recursion.

I also tried with my old ./configuration.nix but that gives me

attribute 'config' in selection path 'config.system.build.toplevel' not found

And TBH I a little lost when it comes to reasoning about these errors.

I am also lost on the need for

pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux;

As I said - I feel a little in over my head.
But I really would love an image.
So I keep trying.

tcurdt · February 19, 2025, 10:46pm

But the big difference is - it bootstraps the server only once for the image.
I really want to get to the point where I can use terraform/opentofu to create my servers.

After that it’s pretty much just running the flake but…

raboof · February 20, 2025, 9:17am

I’m not sure I understand what you mean by ‘bootstraps’ here - that it boots into the rescue system instead of the Ubuntu image? or something else?

How do you manage secrets?

(that sounds neat indeed, though for my use cases I don’t think the added complexity weighs up to the advantages yet)

aanderse · February 20, 2025, 11:56am

i didn’t bother with any custom image for hetzner cloud to do this, i just pick an ubuntu image and run nixos-infect with terraform, wait a few minutes, and then i have a nixos machine which is ready to use

edwinistrator · February 20, 2025, 12:14pm

Hi,

I’ve just done this using nixos-generators.

Essentially what you do is:

Import the nixos-generator module for the desired format into the NixOS configuration. Note that hcloud-upload-image only supports raw disk images, optionally compressed with bzip2 or xz. In my config this looks something like this:

{
  lib,
  modulesPath,
  flake,
  ...
}: {
  imports = [
    (modulesPath + "/profiles/qemu-guest.nix") # Hetzner uses QEMU/KVM
    flake.inputs.nixos-generators.nixosModules.raw-efi
  ];
  # Required for CAX (ARM) servers
  boot.initrd.kernelModules = ["virtio_gpu"];
  boot.kernelParams = ["console=tty"];
}

(I pass specialArgs.flake = self; to nixosSystem. self is one of the arguments always passed to a Flake’s output function. )

Access the attribute that has the name of config.formatAttr at config.system.build; in my case this is config.system.build.raw (and not raw-efi!):

{flake, ...}: {
    hetzner-disk-image = let
      inherit (flake.nixosConfigurations.hetzner) config;
    in
      config.system.build.${config.formatAttr};
}

I then uploaded the resulting image using hcloud-upload-image.

I compressed it with xz before, but as it turns out, hcloud-upload-image just pipes the image through xz on the remote server when uploading. So this only saves bandwidth during the upload, but not any space on the Hetzner snapshot.

Do note that I use the image to directly create a single autoscaled Kubernetes node connected to my homelab cluster; and not to have a generic NixOS installer.
But I guess you could achieve that by:

creating an image for a minimal working NixOS configurations
creating a server with that image (duh)
upload the system closure (config.system.build.toplevel) for the actual configuration to it
activate it

You could of course still use nixos-anywhere, but it would kexec into a live image and reformat the disk first.