Building a Nix OS configuration with parameters (via Terraform)

Help
1

Hey!

I am currently in the process of migrating an Ansible configuration to Terraform + Nix. My current roadblock is that it seems quite hard to supply variables to the system configuration at build time. I will first describe the methods I have tried so far and my considerations about them.

First try

I used terraform-nixos by tweag as one of the steps in my terraform configuration. I wanted to make sure that subsequent runs of terraform would still be able to push the NixOS configuration so I wanted to supply the deploy public key to nix. There is a PR open for this on the repo, so I tried using their code with my project:

resource "aws_key_pair" "deployer" {
  key_name   = "${var.customer}-${var.stage}-concrexit-deployer-key"
  public_key = chomp(tls_private_key.ssh.public_key_openssh)
}

resource "tls_private_key" "ssh" {
  algorithm = "RSA"
  rsa_bits  = "4096"
}

module "deploy_nixos" {
  source = "../deploy_nixos"

  nixos_config = "${path.module}/../../nixos/configuration.nix"
  hermetic = true
  arguments = {
    "deploy_public_key" = chomp(tls_private_key.ssh.public_key_openssh)
  }

  target_user = "root"
  target_host = aws_eip.eip.public_ip

  ssh_private_key = tls_private_key.ssh.private_key_pem

  triggers = {
    // Also re-deploy whenever the VM is re-created
    instance_id = aws_instance.concrexit.id
  }
}

This works, but it seemed that this method sometimes rebuilds packages even though I have just built them. I also wanted to make this a two stage Terraform project, where the first stage only sets up the infrastructure and updates to the contents of the instance can be done quicker afterwards. Because my second stage is just Nix, I figured I wouldn’t need to use terraform-nixos (which seems not very actively developed, and a bit hacky to begin with). So my plan was to use standard Nix tools instead.

Second try

My idea now was to use a classic configuration.nix, but generated by the first terraform stage with the deploy public key (and maybe other variables). I now used two local_file blocks in my terraform config:

resource "local_file" "private_key" {
  filename = "id_rsa"
  file_permission = "0600"
  content = tls_private_key.ssh.private_key_pem
}

resource "local_file" "configuration_nix" {
  filename = "configuration.nix"
  content = <<EOF
{ config, pkgs, ... }:
{
    imports = [ ../../nixos/concrexit.nix ../../nixos/swapfile.nix ];
    nixpkgs.overlays = [ (builtins.getFlake (toString ../../..)).overlay ];

    swapfile = {
      enable = true;
      size = "2GiB";
    };

    # After the first deploy the generated root key should still be usable
    users.users.root.openssh.authorizedKeys.keys = [ "${chomp(tls_private_key.ssh.public_key_openssh)} terraform" ];
}
  EOF
}

Then I wanted to push this configuration to the server, I tried the following command:

NIX_SSHOPTS='-i id_rsa' nix run nixpkgs#nixos-rebuild -- switch -I nixos-config=configuration.nix --target-host root@<public ip>

Unfortunately nixos-rebuild doesn’t seem to understand that I have a remote builder configured and tries to build on my macOS machine instead, which fails with an assertion because it isn’t Linux. Even adding the --build-host option doesn’t seem to help with this, I don’t understand why not.

Third try

I had found that when building a flake, Nix seems to understand that a .x86_64-linux. flake needs to be build on a machine that supports it. So my next idea was to add my NixOS configuration to the flake I already have and try to push that with nixos-rebuild instead. I added an import to my generated settings, such that my flake configuration looked like this:

nixosConfigurations."staging.thalia.nu" = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules =
    [ ./infra/nixos/concrexit.nix
      ./infra/nixos/swapfile.nix
      ./infra/stages/staging/settings.nix
      { 
       nixpkgs.overlays = [ self.overlay ];

       swapfile = {
         enable = true;
         size = "2GiB";
       };
     }
  ];
};

The problem with this is that my settings.nix cannot be imported because it’s generated and not committed to the repo, so the flake cannot see it.

How do I do this properly?

I am quite alright with just trying stuff until it works. But I’m also wondering if I’m thinking in the wrong direction and there’s just and easy way for how to do this that I’m missing. I’d like to hear how other people are doing this stuff, such that I can make a proper configuration.

If you want to see my full configuration, it’s open source and can be found here: GitHub - svthalia/concrexit at add-terraform. I welcome any critique

3 Likes