Derivations built on remote machines not copied to binary cache

Help
1

I have an x86_64-linux Hydra server which uses a post-build-hook to upload built derivations to an S3 bucket. It knows about an x86_64-darwin remote builder which is uses to build derivations for Mac users. I am trying to diagnose why my Hydra server is not uploading those derivations built by the x86_64-darwin machine.

Data points:

  • The Hydra server is running Ubuntu 20.04.6 LTS, Nix 2.13, and Hydra 2022-09-08
  • The post-build-hook definitely works, because I can find in the S3 bucket recent x86_64-linux derivations which were only built by Hydra.
  • EDIT: I tried using nix copy --all --to s3://... to send the Hydra server’s nix store to S3, and this pushed unsigned derivations that were built by the macOS remote builder.

Hypothesis: the post-build-hook runs correctly for locally-built derivations but doesn’t run for derivations built remotely. (If it ran and failed, I’d expect Nix to abort the build loop.) I can’t find anything on the GH issue tracker or the release notes for Nix 2.14…2.16 which suggests an obvious way out. The fact that macOS-built derivations were not signed with the key listed in secret-key-files makes me think that signing and running the post-build-hook might only happen for derivations built locally.

Is there some other data I can add to help diagnose this? Maybe a log or something I can trace through to see if post-build-hook is being (correctly) invoked for remote derivations?

2

Copying my update from GitHub:

I had misconfigured things. The correct way to do this for Hydra is to set store_uri in the Hydra config, which will make Hydra upload built derivations to a remote store. If you set secret-key= in the store URI, it will also sign them.

It’s then arguable that the current behaviour is correct: that each machine should run the post-build-hook only on derivations that it itself built.