Hydra won't sign packages being uploaded to an s3 cache

Development Hydra
1

I can’t get hydra to ever sign any of the packages that it puts up on the s3 cache. I manually tried building a package on the hydra machine and it signed it with the key used in secret-key-files and then I was able to nix copy that to my personal machine that has the hydra public key as a trusted public key. I also made sure to make hydra-queue-runner the owner of the key. Here are the permissions:

/var/lib/hydra/keys]# ls -la
total 12
drwxr-xr-x  2 root               root  4096 Apr 13 20:48 .
drwxr-x--- 10 hydra              hydra 4096 Apr 13 18:22 ..
-r--r-----  1 hydra-queue-runner hydra  102 Apr 13 18:23 hydra-cache-secret

Here is my hydra nixos config:

{ modulesPath, ... }: {
  imports = [ "${modulesPath}/virtualisation/amazon-image.nix" ];
  ec2 = { hvm = true; };

  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  nix.settings.trusted-users = [ "hydra" "root" "@wheel" ];

  nix.extraOptions = ''
    secret-key-files = /var/lib/hydra/keys/hydra-cache-secret
  '';

  networking.hostName = "ci";
  networking.firewall.enable = false;

  services = {
    hydra = {
      enable = true;
      hydraURL = "http://localhost:3000";
      notificationSender = "hydra@localhost";
      useSubstitutes = true;
      extraConfig = ''
        store_uri = s3://nix-cache?region=us-east-2&secret-key=/var/lib/hydra/keys/hydra-cache-secret&write-nar-listing=1&ls-compression=br&log-compression=br
        binary_cache_secret_key = /var/lib/hydra/keys/hydra-cache-secret
        upload_logs_to_binary_cache = true
      '';
    };
  };
}
2

So I dug a little deeper and it turns out that in the nix store of the hydra server the build outputs are being signed by the right key. But on the s3 bucket it is not getting signed at all.