Nixos-anywhere terraform module fails in github CI with Cachix because of 'is not in the Nix store' error

I have a github actions workflow that uses nixos-anywhere’s terraform module to create a new server that runs nixos-anywhere. While I can successfully run terraform apply locally, it fails inside CI:

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: External Program Execution Failed
│ 
│   with module.deploy-worker.module.system-build.data.external.nix-build,
│   on .terraform/modules/deploy-worker/terraform/nix-build/main.tf line 7, in data "external" "nix-build":
│    7:   program = ["${path.module}/nix-build.sh"]
│ 
│ The data source received an unexpected error while attempting to execute
│ the program.
│ 
│ Program: .terraform/modules/deploy-worker/terraform/nix-build/nix-build.sh
│ Error Message: 

copying path
│ '/nix/store/6f34rina5i0vrdd87kflkypnh9wnmc0q-source' from
│ 'https://cache.nixos.org'...
│ 
...

│ building
│ '/nix/store/lf1n6fmf5db25cd11jysjgifyfmzhnc6-linux-6.12.35-modules-shrunk.drv'...
│ building
│ '/nix/store/xkc3w2k7qqpyawadsfrlkk0i2d8khi20-python3.11-playwright-1.52.0.drv'...
│ building
│ '/nix/store/4vwkhnxx0lmx8ka3350s1pawisplsf4b-python3.11-unidecode-1.3.8.drv'...
│ building
│ '/nix/store/gpzjg0p7hiz3mcrimbgf0f3wwg1w7zcv-reload-container.drv'...
│ building
│ '/nix/store/rxid4lr3d9z9syfr9r1f8wlqh1d51qyg-stage-1-init.sh.drv'...
│ building '/nix/store/ih9h0hxblg0jrnvvnc1bflnsd688ix7a-su.pam.drv'...
│ error: path
│ '/nix/store/3a2j2d5bvqidfqsxcik51gkxl8nq9mr9-linux-6.12.35-modules-shrunk/lib'
│ is not in the Nix store
│ 
│ State: exit status 1
╵
::error::Terraform exited with code 1.
Error: Process completed with exit code 1.

I don’t understand why that path isn’t in the nix store. What’s notable is that the terraform plan of the local build differs from the one in the CI.

local:

Terraform will perform the following actions:

  # hcloud_server.daily_worker will be created
  + resource "hcloud_server" "daily_worker" {
      + allow_deprecated_images    = false
      + backup_window              = (known after apply)
      + backups                    = false
      + datacenter                 = (known after apply)
      + delete_protection          = false
      + firewall_ids               = [
          + 2226584,
        ]
      + id                         = (known after apply)
      + ignore_remote_firewall_ids = false
      + image                      = "ubuntu-24.04"
      + ipv4_address               = (known after apply)
      + ipv6_address               = (known after apply)
      + ipv6_network               = (known after apply)
      + keep_disk                  = false
      + location                   = "fsn1"
      + name                       = (known after apply)
      + primary_disk_size          = (known after apply)
      + rebuild_protection         = false
      + server_type                = "cpx11"
      + shutdown_before_deletion   = false
      + ssh_keys                   = [
          + "29969149",
        ]
      + status                     = (known after apply)

      + network {
          + alias_ips   = (known after apply)
          + ip          = (known after apply)
          + mac_address = (known after apply)
          + network_id  = 11183025
        }
    }

  # module.deploy-worker.module.install.null_resource.nixos-remote will be created
  + resource "null_resource" "nixos-remote" {
      + id       = (known after apply)
      + triggers = {
          + "instance_id" = (known after apply)
        }
    }

  # module.deploy-worker.module.nixos-rebuild[0].null_resource.nixos-rebuild will be created
  + resource "null_resource" "nixos-rebuild" {
      + id       = (known after apply)
      + triggers = {
          + "store_path" = "/nix/store/a8yqsdshfsnnjyicjgykksp1j8z6pq9s-nixos-system-nixos-25.11.20250629.a48741b"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

CI:

Terraform planned the following actions, but then encountered a problem:

  # hcloud_server.daily_worker will be created
  + resource "hcloud_server" "daily_worker" {
      + allow_deprecated_images    = false
      + backup_window              = (known after apply)
      + backups                    = false
      + datacenter                 = (known after apply)
      + delete_protection          = false
      + firewall_ids               = [
          + 2226584,
        ]
      + id                         = (known after apply)
      + ignore_remote_firewall_ids = false
      + image                      = "ubuntu-24.04"
      + ipv4_address               = (known after apply)
      + ipv6_address               = (known after apply)
      + ipv6_network               = (known after apply)
      + keep_disk                  = false
      + location                   = "fsn1"
      + name                       = (known after apply)
      + primary_disk_size          = (known after apply)
      + rebuild_protection         = false
      + server_type                = "cpx11"
      + shutdown_before_deletion   = false
      + ssh_keys                   = [
          + "29969149",
        ]
      + status                     = (known after apply)

      + network {
          + alias_ips   = (known after apply)
          + ip          = (known after apply)
          + mac_address = (known after apply)
          + network_id  = 11183025
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

My workflow is:

# .github/workflows/deploy.yml

name: Spawn and Deploy Daily Worker Server

on:
  workflow_dispatch:
    inputs:
      confirm:
        description: "Type 'DEPLOY' to confirm infrastructure deployment"
        required: true

  repository_dispatch:
    types: [deploy-infra]

jobs:
  deploy-daily-worker:
    if: github.event_name == 'repository_dispatch' || github.event.inputs.confirm == 'DEPLOY'

    runs-on: ubuntu-latest
    env:
      HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}

      AWS_ACCESS_KEY_ID:     ${{ secrets.HETZNER_S3_ACCESS_KEY }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.HETZNER_S3_ACCESS_SECRET_KEY }}

      TF_VAR_agenix_key: ${{ secrets.AGENIX_KEY }}

    steps:
      - name: Checkout repository code
        uses: actions/checkout@v4

      - name: Set up SSH with multiple deploy keys
        env:
          SSH_BACKEND_KEY: ${{ secrets.SSH_DEPLOY_BACKEND_KEY }}
          SSH_SCRAPER_KEY: ${{ secrets.SSH_DEPLOY_SCRAPER_KEY }}
        run: |
          mkdir -p ~/.ssh
          chmod 700 ~/.ssh

          echo "$SSH_BACKEND_KEY" > ~/.ssh/id_backend
          echo "$SSH_SCRAPER_KEY" > ~/.ssh/id_scraper
          chmod 600 ~/.ssh/id_backend ~/.ssh/id_scraper

          cat <<EOF > ~/.ssh/config
          Host github.com-backend
            HostName github.com
            User git
            IdentityFile ~/.ssh/id_backend
            IdentitiesOnly yes

          Host github.com-scraper
            HostName github.com
            User git
            IdentityFile ~/.ssh/id_scraper
            IdentitiesOnly yes
          EOF
          
          ssh-keyscan github.com >> ~/.ssh/known_hosts

      - name: Add github.com to known_hosts
        run: ssh-keyscan github.com >> ~/.ssh/known_hosts

      - name: Install Nix and configure cache
        uses: cachix/install-nix-action@v25
        with:
          extra_nix_config: |
            access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
            substituters = https://cache.nixos.org/
            trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
            
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init, Destroy, and Apply
        run: |
          set -e
          
          echo "Initializing Terraform..."
          terraform init
          
          echo "Applying Terraform plan to create new worker server..."
          terraform apply -auto-approve

Would appreciate any help in debugging this.

Using DeterminateSystems’ nix installer and cache actions fixed this for me.

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@v10

      - name: Configure Nix Cache
        uses: DeterminateSystems/magic-nix-cache-action@v6