Niks3 1.0: A S3-backed Nix binary cache with GC

Announcements
1

I have tested niks3 for a while now at https://cache.clan.lol and recently also at https://cache.numtide.com
and now decided to make first release.

The idea behind niks3 is to allow clients to upload nix store paths to S3-compatible storage while tracking uploads on a server for later garbage collection.
This makes it easy to achieve high availability for reading from the binary cache, since this is a solved problem among S3 providers.
With so many providers to choose from, it also becomes quite cheap to host larger caches.

Additionally, niks3 opens the door to features that are missing in the current binary cache ecosystem.
For example, there is an open pull request to add OIDC support,
which would allow providing upload access to binary caches from CIs without handing out long-lived tokens.

31 Likes
2

This sounds cool! I would like to setup a binary cache for riscv64 nixpkgs-25.11 stable. would this be possible to achieve with niks3? what would i need for this?

3

you will need a machine running the niks3 server: Setup Guide · Mic92/niks3 Wiki · GitHub
and a s3-compatible storage S3 Provider Comparison · Mic92/niks3 Wiki · GitHub

Buildbot-nix also has niks3 integration: buildbot-nix/nixosModules/niks3.nix at af5a582396fa643e640b77674143cee1ac633f95 · nix-community/buildbot-nix · GitHub

nix-fast-build probably should also receive niks3 support…

Otherwise you can simple do niks3 push ./result after nix build

5 Likes
4

nix-fast-build now also has support.

1 Like
5

Additionally, niks3 opens the door to features that are missing in the current binary cache ecosystem.
For example, there is an open pull request to add OIDC support

Fwiw S3 supports this today:

6

Yeah. That’s nice. Only bummer is that one still needs to put the signing key into CI, whereas niks3 keeps this in the server. OIDC support was now merged. Docs are here: Home · Mic92/niks3 Wiki · GitHub

Checkout the full release notes here: Release v1.1.0 · Mic92/niks3 · GitHub

1 Like
7

Very cool to have built-in oidc support, I had to hack together a vault server and configure attic to use its ECDSA signing key to get short lived tokens working. I would’ve loved to skip all of that.

Does it also provide similar de-duplication and filtering features (to only cache things not present in a set of upstream caches, for example)?

8

How does this compare to Attic besides the difference in OIDC support?

9

The S3 bucket is directly compatible with Nix, which makes it easy to make the cache highly available without having a lot of servers on your own. More details are in the first paragraph of the project: GitHub - Mic92/niks3: S3-backed Nix binary cache with garbage collection

10

No, nix doesn’t support de-duplication in the binary cache protocol and it’s goal is to be directly compatible with it. I have to do more measurement, but so far I have not seen too many space saving from the deduplication algorithm, I tried. Regarding the filtering, it’s also something I rather not want to add because:

  • s3 storage can actually be rather cheap (6 Dollar / TB is a lot of cache)
  • I noticed that serving from my cache with zstd compression is faster than cache.nixos.org
  • We want to garbage collect https://cache.nixos.org hopefully starting next year and Nix really doesn’t like it if you leave invalid references - I don’t want to have niks3 users having a broken cache.
1 Like