Populating a binary cache on S3

1

Hi folks,

I’m trying to set up a binary cache in an S3 bucket. (I can’t use cachix for security reasons.)

I got all the configuration, signing etc figured out but I’m stuck on actually populating the cache. Using nix copy on the derivation uploads just derivations, not the actual outputs. Using nix copy on the output uploads just the runtime dependencies.

I’ve even tried doing this:

nix-store --query --requisites --include-outputs “$derivation” | xargs nix copy --no-recursive --to “$cache_url”

That seems to work—lots of stuff gets uploaded—but when I try to build using only my cache and not cache.nixos.org, it downloads a lot of stuff from the cache and then starts trying bootstrap gcc from source. Apparently something crucial isn’t making it into the cache.

Does anybody know how to do this properly?

Thanks,

Colin

1 Like
2

Well, here’s a different approach: how does cache.nixos.org get populated? I’ve been looking for that code, but I can’t find it.

3

cache.nixos.org is populate via hydra (configuration: https://github.com/NixOS/nixos-org-configurations/blob/63cb1725f4d8ddebf44c2789c005b673dad93836/delft/hydra.nix#L37).

Hydra copies the sources in addition to all the binaries that are being built. I am not aware of any of the nix cli options that might have the same effect.

Just a moonshot: You specified the signing key on the machines where you tried to use the cache? How did you specify the cache on your build command line? Posting a bit of verbose output might be helpful.

4

I got all the configuration, signing etc figured out but I’m stuck on actually populating the cache.

We’re doing this exact thing at work. Here’s the commands we use from our CI:

$ nix-build
$ nix copy \
    --to  "s3://our-nix-cache-bucket-name?profile=our-profile-name&endpoint=our.endpoint.example.com" \
    --option narinfo-cache-positive-ttl 0 \
    $(nix-store --query --requisites --include-outputs $(nix-store --query --deriver ./result))

when I try to build using only my cache and not cache.nixos.org, it downloads a lot of stuff from the cache and then starts trying bootstrap gcc from source.

When I try to build our local project from scratch for the first time, I believe some things are downloaded from cache.nixos.org and not our work nix cache. Although everything is downloaded from some cache, so we don’t have to build anything locally.

As @andir said, you probably want to run nix-build with either the --debug or --verbose flags. You get quite a lot of output, but it is usually very helpful in debugging cache-related problems.

9 Likes
5

You folks are awesome!

After poring through astounding amounts of debug output, I realized my mistake: the derivation I was building to deploy wasn’t quite the same as the derivation I was using to populate the cache. In particular, the python3 package was functionally equivalent, but had different inputs. So with that fixed, I can build entirely from the private cache.

Thank you!

Colin

2 Likes