Announcing Harmonia, a nix binary cache written in rust

We maintain our own hydra at our company that continually builds our software
and systems, to keep deploy times to a minimum. We have been using eris as a
binary cache for quite some time we got more and more incomplete downloads. To
solve this problem we internally had a discussion if we should either add
http accept-ranges to eris or try to implement a binary cache in rust ourselves. We
went with the second option.

We’ve been using harmonia for over a month now and are now confident to
officially announce it here.

Harmonia is a binary cache that serves your local nix-store similar to
nix-serve. To make harmonia work, we first had to write rust bindings into libnixstore, to make access to the store really simple. This makes part
of the project unsafe but we accepted the drawback rather than
reimplementing a lot of the nix store internals. We have plans to make the
bindings in a separate crate and would accept PRs that improve the current
bindings

On downloading a nar we currently still do nix-store --dump as it was more
efficient than using the FdSink directly, because we had to wait till it was
done writing to a file descriptor. I still think an export path with a Sink
could be more efficient and I was working on an implementation using
BufferedSink but it was still not as fast as using an external command.

Some other noteworthy features include support for http accept-ranges and access to
build logs via /log/{drv}.

While we already use harmonia in production, I don’t consider it done yet and
still have the following tasks on my todo list

  • Compression, ideally zstd
  • libnixstore bindings cleanup and publish as separate crate so other rust
    projects can use it.
  • nar streaming using libnixstore
  • eris upstream option is still missing
  • providing .ls files for nix-index
  • nix integration tests

Contributors to the project are much appreciated.

Last but not least, I have to thank eris, which was used as inspiration while
implementing. @das_j for answering all my nix questions, and @ajs124 for testing.

11 Likes

I’ve written a similar project a while ago: GitHub - NickCao/carinae. And it does have streaming compression, however I did’t not find a feasible way to determine the compressed nar size (it’s streaming, after all). Hopefully you guys can come up with some interesting ideas.

2 Likes

@dramforever Just found a blind spot: we can rely on native http compressions, AKA transfer encoding. However nix is not currently sending the Accept-Encoding header on outgoing requests, rendering this ineffective.

Turned out to be trivial to implement with the help of libcurl: https://github.com/NixOS/nix/pull/6694

What’s the reason why you are not using hydra as the binary cache direct?

xz actually achieves smaller archives with the disadvantage of being slower to create and decompress.

I think the reasons are mixed, some is just having a database that’s already struggling to catch up with the pace of hydra, others might not have hydra running at all, they just have a shared builder.

This disadvantage was big enough for Arch to switch from xz to zstd about two and a half years ago:
https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/

2 Likes
Hosted by Flying Circus.