Several months ago I had an idea for an experiment to make Nix downloads faster: transfer only differences between versions of a package. It took some effort to make it all work out nicely, but I’ve been using it on my personal machines for a while and have been happy with the results. It often (but not always) does actually make system updates faster on my slow-ish home connection. You can read all the details and how to try it yourself here:
A quick summary:
- Acts as a local substituter
- Keeps track of packages present on the system
- To get a narinfo: Finds a suitable “base” package that we guess will have a small delta to the requested one
- To get a nar: Asks a remote “differ” for that binary delta, then reconstructs the requested nar from the base package and the delta
Where does it run? The remote differ runs in Lambda so it’s close to the public binary cache. It’s currently well under the free tier, but it wouldn’t be if it was open to everyone, so you can run one yourself, or send me a DM and a few people can use mine for a while.
How well does it work? I generally get a 5–15× reduction in data transferred for routine system updates. It often takes less overall time, but not always, since the deltas and reconstruction can be slow (especially for kernels), and the internet is pretty fast. This will be most interesting to people on slow/metered connections.
How does this relate to other store-related efforts that use content-defined chunking? It’s both better and worse in different ways, but mostly worse, since it can’t be scaled up as easily and ends up being more expensive. Still, it was a fun experiment and I plan to continue using it and maybe make a few extensions.