I have a github actions workflow that uses nixos-anywhere’s terraform module to create a new server that runs nixos-anywhere. While I can successfully run terraform apply locally, it fails inside CI:
Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: External Program Execution Failed
│
│ with module.deploy-worker.module.system-build.data.external.nix-build,
│ on .terraform/modules/deploy-worker/terraform/nix-build/main.tf line 7, in data "external" "nix-build":
│ 7: program = ["${path.module}/nix-build.sh"]
│
│ The data source received an unexpected error while attempting to execute
│ the program.
│
│ Program: .terraform/modules/deploy-worker/terraform/nix-build/nix-build.sh
│ Error Message:
copying path
│ '/nix/store/6f34rina5i0vrdd87kflkypnh9wnmc0q-source' from
│ 'https://cache.nixos.org'...
│
...
│ building
│ '/nix/store/lf1n6fmf5db25cd11jysjgifyfmzhnc6-linux-6.12.35-modules-shrunk.drv'...
│ building
│ '/nix/store/xkc3w2k7qqpyawadsfrlkk0i2d8khi20-python3.11-playwright-1.52.0.drv'...
│ building
│ '/nix/store/4vwkhnxx0lmx8ka3350s1pawisplsf4b-python3.11-unidecode-1.3.8.drv'...
│ building
│ '/nix/store/gpzjg0p7hiz3mcrimbgf0f3wwg1w7zcv-reload-container.drv'...
│ building
│ '/nix/store/rxid4lr3d9z9syfr9r1f8wlqh1d51qyg-stage-1-init.sh.drv'...
│ building '/nix/store/ih9h0hxblg0jrnvvnc1bflnsd688ix7a-su.pam.drv'...
│ error: path
│ '/nix/store/3a2j2d5bvqidfqsxcik51gkxl8nq9mr9-linux-6.12.35-modules-shrunk/lib'
│ is not in the Nix store
│
│ State: exit status 1
╵
::error::Terraform exited with code 1.
Error: Process completed with exit code 1.
I don’t understand why that path isn’t in the nix store. What’s notable is that the terraform plan of the local build differs from the one in the CI.
local:
Terraform will perform the following actions:
# hcloud_server.daily_worker will be created
+ resource "hcloud_server" "daily_worker" {
+ allow_deprecated_images = false
+ backup_window = (known after apply)
+ backups = false
+ datacenter = (known after apply)
+ delete_protection = false
+ firewall_ids = [
+ 2226584,
]
+ id = (known after apply)
+ ignore_remote_firewall_ids = false
+ image = "ubuntu-24.04"
+ ipv4_address = (known after apply)
+ ipv6_address = (known after apply)
+ ipv6_network = (known after apply)
+ keep_disk = false
+ location = "fsn1"
+ name = (known after apply)
+ primary_disk_size = (known after apply)
+ rebuild_protection = false
+ server_type = "cpx11"
+ shutdown_before_deletion = false
+ ssh_keys = [
+ "29969149",
]
+ status = (known after apply)
+ network {
+ alias_ips = (known after apply)
+ ip = (known after apply)
+ mac_address = (known after apply)
+ network_id = 11183025
}
}
# module.deploy-worker.module.install.null_resource.nixos-remote will be created
+ resource "null_resource" "nixos-remote" {
+ id = (known after apply)
+ triggers = {
+ "instance_id" = (known after apply)
}
}
# module.deploy-worker.module.nixos-rebuild[0].null_resource.nixos-rebuild will be created
+ resource "null_resource" "nixos-rebuild" {
+ id = (known after apply)
+ triggers = {
+ "store_path" = "/nix/store/a8yqsdshfsnnjyicjgykksp1j8z6pq9s-nixos-system-nixos-25.11.20250629.a48741b"
}
}
Plan: 3 to add, 0 to change, 0 to destroy.
CI:
Terraform planned the following actions, but then encountered a problem:
# hcloud_server.daily_worker will be created
+ resource "hcloud_server" "daily_worker" {
+ allow_deprecated_images = false
+ backup_window = (known after apply)
+ backups = false
+ datacenter = (known after apply)
+ delete_protection = false
+ firewall_ids = [
+ 2226584,
]
+ id = (known after apply)
+ ignore_remote_firewall_ids = false
+ image = "ubuntu-24.04"
+ ipv4_address = (known after apply)
+ ipv6_address = (known after apply)
+ ipv6_network = (known after apply)
+ keep_disk = false
+ location = "fsn1"
+ name = (known after apply)
+ primary_disk_size = (known after apply)
+ rebuild_protection = false
+ server_type = "cpx11"
+ shutdown_before_deletion = false
+ ssh_keys = [
+ "29969149",
]
+ status = (known after apply)
+ network {
+ alias_ips = (known after apply)
+ ip = (known after apply)
+ mac_address = (known after apply)
+ network_id = 11183025
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
My workflow is:
# .github/workflows/deploy.yml
name: Spawn and Deploy Daily Worker Server
on:
workflow_dispatch:
inputs:
confirm:
description: "Type 'DEPLOY' to confirm infrastructure deployment"
required: true
repository_dispatch:
types: [deploy-infra]
jobs:
deploy-daily-worker:
if: github.event_name == 'repository_dispatch' || github.event.inputs.confirm == 'DEPLOY'
runs-on: ubuntu-latest
env:
HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.HETZNER_S3_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.HETZNER_S3_ACCESS_SECRET_KEY }}
TF_VAR_agenix_key: ${{ secrets.AGENIX_KEY }}
steps:
- name: Checkout repository code
uses: actions/checkout@v4
- name: Set up SSH with multiple deploy keys
env:
SSH_BACKEND_KEY: ${{ secrets.SSH_DEPLOY_BACKEND_KEY }}
SSH_SCRAPER_KEY: ${{ secrets.SSH_DEPLOY_SCRAPER_KEY }}
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$SSH_BACKEND_KEY" > ~/.ssh/id_backend
echo "$SSH_SCRAPER_KEY" > ~/.ssh/id_scraper
chmod 600 ~/.ssh/id_backend ~/.ssh/id_scraper
cat <<EOF > ~/.ssh/config
Host github.com-backend
HostName github.com
User git
IdentityFile ~/.ssh/id_backend
IdentitiesOnly yes
Host github.com-scraper
HostName github.com
User git
IdentityFile ~/.ssh/id_scraper
IdentitiesOnly yes
EOF
ssh-keyscan github.com >> ~/.ssh/known_hosts
- name: Add github.com to known_hosts
run: ssh-keyscan github.com >> ~/.ssh/known_hosts
- name: Install Nix and configure cache
uses: cachix/install-nix-action@v25
with:
extra_nix_config: |
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
substituters = https://cache.nixos.org/
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Terraform Init, Destroy, and Apply
run: |
set -e
echo "Initializing Terraform..."
terraform init
echo "Applying Terraform plan to create new worker server..."
terraform apply -auto-approve
Would appreciate any help in debugging this.