A cloudinit image for use in proxmox

Development
1

Hey folks,

I’ve spent the last few nights fiddling with this and thought I’d share it in case someone else needs it.

Basically, I want to use nixos in our proxmox cluster. We’re used to provisioning VMs with terraform and cloudinit, but proxmox makes it a little difficult to use the cloudinit runcmd tthat’s the obvious path for nixos-infect.
Besides that, I don’t know when to stop, so I thought “Why even use the debian cloudinit vm, that seems like unnecessary extra steps” - so I built a nixos cloudinit VM.

A lot of the modules don’t make a whole lot of sense, so I removed them. The only thing we’re really doing is adding ssh keys to the predefined user and setting a static IP and gateway, so most other parts are disabled.
You’d push a nixos config that doesn’t have cloudinit on it anyways I guess.

The flake is under a hundred lines :smiling_face_with_three_hearts:

It’s somewhat opinionated though, so read through it before using it, but if you’re in this deep I assume you’d do that anyways.

3 Likes
2

Hello @voydus

Thanks a lot for sharing. I’m trying to run this flake and it stops up with multiple errors.

error: flake 'path:/home/ops/pm-cloud-init' does not provide attribute 'apps.x86_64-linux.default', 'defaultApp.x86_64-linux', 'packages.x86_64-linux.default' or 'defaultPackage.x86_64-linux'

I’m new to nixos and I’m probably missing something obvious. Could you please help?

Thanks

3

Hi, thanks for sharing.

@mrborkander to build the image you need to run nix build .#image (or set a path if you are not in the directory of flake.nix)

1 Like
4

Could I use it for unattended Nix install?

1 Like
5

@voydus

I’m very new to Nixos, and I’m trying to get this whole cloud-init image to work with Proxmox, but I can’t figure out where the configuration.nix file is for this template, as it’s not in the normal /etc/nixos/. Is it possible to use this as a barebones template and then after I’ve setup the VM, change the configuration file and switch to that configuration file?

What I’m trying to do is use the whole cloud-init structure to have a barebones nixos that I will then modify the configuration.nix & hardware-configuration.nix on after creation (scp it up since I now have an available username and ssh key thanks to cloud-init) to automate installing nixos with a bunch of stuff I want setup.

Anyway, the issue I’m having is I don’t know where to find the configuration.nix file on this cloud-init installation. I have a suspicion that there isn’t one, and this image is setup specifically “just” for what was setup in the flake, and I don’t know enough on how to modify the flake to do this (or if this is even possible)

Can anyone help out? Is what I want to do possible?

NOTES - If it helps anyone else (@MatthieuB ?), here’s the steps I did to build this image and get it running on proxmox:

# Build flake image
nix --experimental-features 'nix-command flakes' build .#image

# cd to the 'results' folder (that's where the image gets created)
cd results

# Get shell with 'qemu-img'
nix-shell -p qemu

# Convert the 'qcow2' to 'img'
qemu-img convert nixos.qcow2 -O raw nixos.img

Next you have to somehow get that image over to a folder in Proxmox (I just used Filezilla to grab it via ssh)

To create a template from the image (these commands run on Proxmox where you’ve uploaded the img file):

qm create 9001 --memory 2048 --core 2 --name nixos-23.11-kvm --net0 virtio,bridge=vmbr0
qm importdisk 9001 nixos.img local-lvm
qm set 9001 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9001-disk-0
qm set 9001 --ide2 local-lvm:cloudinit
qm set 9001 --boot c --bootdisk scsi0
qm set 9001 --serial0 socket --vga serial0
qm set 9001 --ipconfig0 ip=dhcp
qm template 9001

If I set the username when I create from template to ops (the name setup in the above flake) I get it to work fine. Trying other usernames didn’t seem to work.

However, in the above, I can’t get to the VM via the Console option in Proxmox, I think the flake needs to be modified somehow to add a serial socket connection or something. However, as long as you’ve setup an ssh key, you can access it via ssh fine.

6

… and,… I figured it all out!

Just because there isn’t a configuration.nix doesn’t mean you can’t create one and then build everything anyways. The above flake didn’t work as is and I had to add the following to be able to build a configuration.nix file:

      # Enable experimental features we'll need
      nix.settings.experimental-features = [
        "nix-command"
        "flakes"
      ];

I also needed to modify this (notice copyChannel = true):

...
  in {
    inherit pkgs;
    image = make-disk-image {
      inherit pkgs lib;
      config = nixos.config;
      name = "nixos-cloudinit";
      format = "qcow2-compressed";
      copyChannel = true;
    };
  };

If you don’t do that, you’ll have to do some extra work to get the channel going on the created image on first start. I’m not sure if it’s better to do that or put it in the image.

When you’ve create the proxmox template and you’re in the running VM, just setup your configuration.nix however you want and run the following, you’ll need to specify where the configuration.nix file is:

sudo nixos-rebuild switch -I nixos-config=/etc/nixos/configuration.nix

This should allow you to completely automate installing Nixos and configuring it in Proxmox.

1 Like
7

Hi,

I am looking into achieving exactly this, provision nixos VMs to a proxmox cluster. However I cannot get my head around the process.

I could build a nixos template based on the flake from @voydus and import to proxmox.
I also have a configuration.nix for the VM I would like to deploy.

Then how to “inject” the configuration.nix into a newly created VM instance? Is this supposed to be done by the Terraform provider (e.g. Telmate)?

If someone could take some time and explain the process in more details I would really appreciate.

Thanks in advance

8

Hey @jerhat I’ll be doing the same shortly actually, deploying NixOS onto my Proxmox cluster. Maybe we could team up and exchange notes? Send me a private message maybe…?

In the meantime, nixos-rebuild can be used remotely via ssh. That is what I might try probably. Some info on that is to be found on the unofficial wiki:

https://nixos.wiki/wiki/Nixos-rebuild

Edit: for terraform there’d be multiple way of doing that. One way I could think of would be to use the local-exec provisioner to run the nixos-rebuild via ssh.

9

@voydus could you clarify if the installation process in your image starts automatically and fully unattended?
I don’t see evidence of that in your flake.

10

I’ve been provisioning the VM via Terraform with just pretty simple cloudinit, namely a user and SSH keys. After that, I used deploy-rs to deploy the actual config.
Additional, we have not actually used this a lot in production since we’re a volunteer-drive org which can’t really afford the learning curve of nixos, even though a some of us really like it…

With all of that said, I wasn’t expecting this much interest in the topic. Somebody should probably do a proper writeup. I’d love to do it, but I’m pretty busy right now so I can’t promise anything.

Edit:
The startup procedure basically looks like this:
In proxmox, you configure the cloudinit data, which will be made available to the VM as a mountable block device with some special identifier (I don’t know the details here). On boot, the cloudinit service is started, discovers the volume and does the setup according to the data.
For me, that is creating the user and setting up the web key. On our debian machines, it also runs an initial apt upgrade etc.
Usually, cloudinit runs on every boot (but does some things differently depending on whether it’s first boot or not)
For my nixos stuff, I deploy a config that does not include cloudinit, so it’s basically just used for bootstrapping.

Hope this helps