In recent memory there have been some issues with the official binary cache, and I’ve been working on improving things. I’m happy to say I’ve got a working “mirror” of the cache that can serve as a testing ground for several improvements I’m making to our Fastly configuration, and I’d like it if users would give it a try.
TL;DR: set your default substituter to https://aseipp-nix-cache.global.ssl.fastly.net, and that’s it! This uses the upstream S3 bucket as a backend, and therefore you do not have to change your trusted keys, and it has all channel builds!
Visit that page (click the link) for more info, a changelog, etc.
There’s also a wiki page describing the state of play at the moment, including known bugs, and more specific issues, and possible future plans. Please take a moment to read it over: https://nixos.wiki/wiki/Maintainers:Fastly
The source code (mostly a VCL configuration, and some HTML files) are not yet public. Stay tuned for that. (Nothing private, but I would like to consolidate it under the nixos organization, which will require someone making the repo.)
Check the changelog for some details about the improvements that were made: https://aseipp-nix-cache.global.ssl.fastly.net/changelog
This is basically an internal overhaul to the cache configuration which has a much more sophisticated implementation, written in VCL. In short: the cache configuration is now a program, more or less, so it should be easier to extend and customize in the long run. It enables several features we weren’t taking advantage of, including various performance enhancements. (You can find details on the wiki and the changelog.)
The goals of this first round of improvements are:
- Improve performance, latency and cache hit ratio for all objects (large objects that are not in the cache, repeated 404s, etc).
- Implement far more aggressive caching policies across all objects. narinfo files and manifests are cached for a month, while actual packages are cached for 1 week.
- Some better logging infrastructure so we can investigate and watch cache usage, growth, etc.
- Isolate some issues users have been having – including IPv6 errors that seem to have been causing user issues.
Most of this has been accomplished, so I’d appreciate some testing and tweaking.
If you are using IPv6+IPv4 and have been having issues, I would especially appreciate you testing this. For now, the above URL is IPv4 ONLY, but there is a separate IPv4+IPv6 dual stack address you can test as well, with HTTP/2 support. The wiki has details: https://nixos.wiki/wiki/Maintainers:Fastly#Beta_.2B_IPv6_.2B_HTTP.2F2
Please let me know if IPv6 is problematic for you, and we can dig into more details if so.
I would also appreciate any feedback from users who hit the cache from Asia/Australia for example. Previously, these users would normally connect somewhere like Singapore/Hong Kong, which would then later connect all the way to
us-east-1. This would require a very large TCP handshake across the globe, making latency quite bad, especially in the case of many small misses (e.g. for many narinfo files which do not exist, which would effectively be a worst-case.)
This experience should now be significantly improved – both the official cache.nixos.org server has seen several improvements since last week, but the Beta cache server should improve things even further (I hope).
Please be gentle with this service. It’s a personal account of mine, so don’t set it as the substituter for your CI machine, if you can. But feel free to use it directly via
--option, on your laptops/workstations, etc. It should be stable, but occasionally I might break something, so please report that.
If you have issues, please feel free to report them here, or contact me on IRC.