Announcing Harmonia, a nix binary cache written in rust

Announcements
1

We maintain our own hydra at our company that continually builds our software
and systems, to keep deploy times to a minimum. We have been using eris as a
binary cache for quite some time we got more and more incomplete downloads. To
solve this problem we internally had a discussion if we should either add
http accept-ranges to eris or try to implement a binary cache in rust ourselves. We
went with the second option.

We’ve been using harmonia for over a month now and are now confident to
officially announce it here.

Harmonia is a binary cache that serves your local nix-store similar to
nix-serve. To make harmonia work, we first had to write rust bindings into libnixstore, to make access to the store really simple. This makes part
of the project unsafe but we accepted the drawback rather than
reimplementing a lot of the nix store internals. We have plans to make the
bindings in a separate crate and would accept PRs that improve the current
bindings

On downloading a nar we currently still do nix-store --dump as it was more
efficient than using the FdSink directly, because we had to wait till it was
done writing to a file descriptor. I still think an export path with a Sink
could be more efficient and I was working on an implementation using
BufferedSink but it was still not as fast as using an external command.

Some other noteworthy features include support for http accept-ranges and access to
build logs via /log/{drv}.

While we already use harmonia in production, I don’t consider it done yet and
still have the following tasks on my todo list

  • Compression, ideally zstd
  • libnixstore bindings cleanup and publish as separate crate so other rust
    projects can use it.
  • nar streaming using libnixstore
  • eris upstream option is still missing
  • providing .ls files for nix-index
  • nix integration tests

Contributors to the project are much appreciated.

Last but not least, I have to thank eris, which was used as inspiration while
implementing. @das_j for answering all my nix questions, and @ajs124 for testing.

14 Likes
2

I’ve written a similar project a while ago: GitHub - NickCao/carinae: serve any nix store as a binary cache. And it does have streaming compression, however I did’t not find a feasible way to determine the compressed nar size (it’s streaming, after all). Hopefully you guys can come up with some interesting ideas.

3 Likes
3

@dramforever Just found a blind spot: we can rely on native http compressions, AKA transfer encoding. However nix is not currently sending the Accept-Encoding header on outgoing requests, rendering this ineffective.

Turned out to be trivial to implement with the help of libcurl: filetransfer: set Accept-Encoding to all values supported by libcurl for transparent compression by NickCao · Pull Request #6694 · NixOS/nix · GitHub

4

What’s the reason why you are not using hydra as the binary cache direct?

5

xz actually achieves smaller archives with the disadvantage of being slower to create and decompress.

6

I think the reasons are mixed, some is just having a database that’s already struggling to catch up with the pace of hydra, others might not have hydra running at all, they just have a shared builder.

7

This disadvantage was big enough for Arch to switch from xz to zstd about two and a half years ago:
https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/

2 Likes
8

We have multiple harmonias spread over the builders. This means we can build a derivation on any builder without having to involve Hydra and just use the regular substitution mechanism to get the result back.
In the past, we also had builders with better network connectivity than the Hydra server (don’t ask), meaning that we wanted to prefer substitution from the builders.

Another good reason is that the Hydra builtin nar server just shells out to nix-store --dump | pixz -0 and keeps a whole catalyst worker busy for streaming the nar.