I’m excited to announce ncps, a new proxy server designed to speed up your Nix dependency retrieval and build times, especially in environments with multiple machines.
ncps acts as a local binary cache on your network, fetching store paths from upstream caches (like cache.nixos.org) and storing them locally. This means:
Reduced bandwidth usage: Each dependency is pulled only once onto the cache, and then all machines on your network can use it, saving on internet costs and improving network performance.
Faster build times: Get your projects built quicker with readily available dependencies.
Here’s what makes ncps stand out:
Easy setup: Get it running with just a few configuration flags.
Multiple upstream caches: Configure multiple sources for redundancy and flexibility.
Secure caching: ncps signs cached paths with its own key, ensuring integrity.
Cache size management: Prevent your cache from growing indefinitely with configurable maximum size and automated cleanup.
Zstandard compression support: Specifically designed for use with Harmonia, ncps efficiently stores and serves NAR files compressed with zstd as received during HTTP transport, without any decompression overhead. This results in faster storage and retrieval of dependencies.
OpenTelemetry support: Integrate with OpenTelemetry for centralized logging, metrics, and tracing.
no, nix does not handle this well; you need to put things like this in trusted-substituters and specify them on the command line when you want to use them.
One key hope for a proxy like this is that it might help handle the errors more gracefully, as long as the proxy itself is reachable (which perhaps means local)
I was thinking of adding a feature to return priority based on client’s IP address: Less than 30 for local clients and >100 for remote; But I wanted to wait to gather more use-cases before I commit to adding more complexity.
This looks really interesting, as this duplication of downloads is something I’m definitely hitting!
However I was hoping to use a binary cache also for anything I end up compiling, when it’s not already in the official cache.
Is there a way I could achieve that automatically, as well as ncps primary benefits?
If not, is some kind of ability to automatically push locally built packages to ncps a realistic possibility in future?
Is there any nix proposal that would allow us to “set a cache proxy” that would always be used when fetching store paths and would allow us to cache store paths from every nix caches we use?
In other words, with ncps, we need to specify the upstream caches in advance. Wouldn’t it be nice to just have a “proxy” that would cache everything (without having to use a MITM proxy)?
Yes, it does. I use it all the time. I build my servers locally, send them to my nix cache and when I apply each server it goes pretty fast since it pulls from the cache over my 10G network; My aim is to minimize downtime on my Kubernetes cluster.
That’s exactly why I added the PUT functionality gated with the --cache-allow-put-verb which allows you to push directly to the cache. Example of how I use it:
Not at this time since that will complicate garbage collection (implemented as LRU). I would be willing to review pull requests for it. Please consider opening an issue on ncps to allow people to vote on it, with enough votes, I’d consider spending the time on it.
Looks like that PR has since stalled out, and @philipwilk has now picked up the torch! Although they could use some help with patching nix for system wide testing and dogfooding:
Worked on the patch over nixcon and im pretty happy with it rn, it’s covering my expected scenarios - trying other substituters (at all) if one fails, and not exploding if one substituter starts giving 500s.
Well, now that it supports gentle failover I would like to deploy it to all individual machines in my fleet and put it in front of the local nix by default, to prevent nix to crash when a cache is not available. But I don’t want to cache store paths in a separate directory on each machine. Can I disable the ‘cache’ and instead just use it as a ‘proxy’? The idea would be to have one ‘caching’ instance per location, and one ‘proxy’ instance on each machine.
Also does it support parallel cache downloads from the same origin? cache.nixos.org is often extremely slow in Asia. I’m talking less than 1 mbyte/s despite being on a gigabit internet. If it could download 10 paths in parallel, that would improve the UX most likely.
