How to bootstrap data volumes for NixOS machines on AWS 🤔

Hello guys!

I’m finally starting to see how to use NixOS-based VMs on AWS for work (:star_struck:) and I already have so many cool ideas, but I’m having a hard time finding out how to bootstrap the machines properly..

To start I’m toying around with wanting a VM with 2 data volumes (200G /nix & 200G /home, this is a dev machine for now) and deployed with deploy-rs from a mac.

I tried using the official nixos NixOS images for EC2 (NixOS Amazon Images / AMIs) but when I try to deploy to it I get an error because the additional data volumes are still unformated and the existing /nix isn’t the one I want..

I have disko in the mix but I’m not sure how to use it remotely.

I’m thinking about setting up bootstrap-related scripting in the OS activation script to initialize the data volumes.
Or I’m also potentially thinking about not using the NixOS AMIs and basically take any base image and use nixos-anywhere to set it up.. :thinking:

Or am I just shooting myself in the foot by wanting to have a seperate /nix (I could just use a bigger root volume).

This whole thought process feels like I’m holding it wrong (disko/deploy-rs) and I’m missing something that would make the bootstrapping simpler.

Could anyone help me on this?

i don’t use aws anymore, but i did (and still do, with another provider) this, normally from a mounted recovery iso or similar. you should be able to do it on anything hosted on ec2.

what feels wrong? spin the vm up, attach the appropriate ebs volume(s), peep the disk paths (pretty sure aws still does /dev/vd*, ideally you use something in /dev/disk/by-* after working out which one points to the volume you want to use), then nixos-anywhere it with the disko config pointing to those disks in the format you want - nixos-anywhere (uses|can use) disko.

assuming flakes, this becomes nix run github:nix-community/nixos-anywhere -- --flake .#my-cool-vm user@ip.address.goes.here and awayyyyyyyyyyyyyy we go. i’d imagine the command for non-flakes is similarly shaped, and that’d be the only difference.

if you run nixos-anywhere --help you can see the disko modes it offers:

$ nix run github:nix-community/nixos-anywhere -- --help
...
* --disko-mode disko|mount|format
  set the disko mode to format, mount or destroy. Default is disko.
  disko: first unmount and destroy all filesystems on the disks we want to format, then run the create and mount mode

which map to the disko modes:

$ nix run github:nix-community/disko -- --help
...
* -m, --mode mode
  set the mode, either destroy, format, mount, unmount, format,mount or destroy,format,mount
    destroy: unmount filesystems and destroy partition tables of the selected disks
    format: create partition tables, zpools, lvms, raids and filesystems if they don't exist yet
    mount: mount the partitions at the specified root-mountpoint
    format,mount: run format and mount in sequence
    destroy,format,mount: run all three modes in sequence. Previously known as --mode disko

assuming you have your 200gb nix volume at /dev/vdb and your 200gb home volume at /dev/vdc (with your root at /dev/vda), you can configure disko somewhat like this (apologies if not quite right/idiomatic, pillaged it from the readme and edited it in the discourse reply box):

{
  disko.devices = {
    disk = {
      root = {
        device = "/dev/vda";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              type = "EF00";
              size = "500M";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "umask=0077" ];
              };
            };
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "xfs";
                mountpoint = "/";
              };
            };
          };
        };
      };
      
      nix = {
        device = "/dev/vdb";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            nix = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "xfs";
                mountpoint = "/nix";
              };
            };
          };
        };
      };
      
      home = {
        device = "/dev/vdc";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            home = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "xfs";
                mountpoint = "/home";
              };
            };
          };
        };
      };
    };
  };
}

not really. i guess you could cart that ebs volume around to another instance for a laugh because it’s entirely separate, otherwise inside an ec2 instance it’s more or less the same anyway assuming you’re using the same storage class. it would also kinda make snapshotting the store a bit easier using aws’s native tools if you do want a clone of it.

Another option (not in the “extreme reproducibility”) is whitequark/nixos-bite. Where you can use cloud-init to convert a machine to nixos, where you can later apply your config. It keeps the SSH keys from the original OS, allowing you to access them after converting to NixOS.

So for example, you could use terraform to provision the machine(s) with cloud-init, and then use nixos to change its configuration. It doesn’t seem to be tested on AWS though

Echoing arcayr who covered it pretty well: nixos-anywhere plus disko targetting one of the premade EC2 AMIs is a great way to do the initial format+install.

Once you have a first test that works, a great next step is using something like terraform opentofu to codify the EC2 spinup and collapse all the AWS console clicking into one script, so you can go from scratch to a finished install with a single command.

Thanks everyone!

I went with the nixos-anywhere route with any bootstrap AMI (using the latest AmazonLinux for now), it’s working like a charm :heart:

I’m honestly very stumped here. the nixos ami will execute user-data if it’s a script, so whatever particular pin or config you want to switch into can be done on boot. Why do you want the nix store off the root volume? If you want to add extra volumes for data and what not, and you can inject this information into user-data (or read it via imds if you are feeling fancy) and format them on boot. Doing nixos-anywhere + some other distro seems like a very round-about the houses way of booting.