A convenient suggestion: Binary-Derivations

1

I love NixOS because it makes system configuration reusable, reliable and modular. The issue I have is that (nixos-) rebuilds tend to be time intensive, big and small rebuilds alike, they just take inconveniently long. Of course you can deal with this by changing the way you work (you just do something else in the meantime). But it certainly is more convenient to do apt-get install xyz on ubuntu for example.

Now to make small rebuilds faster is something I am not quite sure how to tackle. I guess would mean making the nix language faster more performant and maybe to cache nix expressions in some way.

But for bigger rebuilds I have a few ideas.

The main idea is, to opt for binary derivations over source derivations as a general strategy (without eliminating source derivations). Of course I have not thought through all the consequences this might have, which is why I am putting it out here as a suggestion.

By this I mean something along the lines of what other distributions do, but in the nixy way: in debian, when you install a library, you pick a certain version of it. And when you run a program, it will search for the library, find the version that is installed and compatible, and load the dependencies of that library and so on. Now I appreciate that nixos does not do that, but that in nixos the dependencies are hardwired to fixed versions. My idea here, is to reduce those versions, or at least their build/download time. So to do the thing that debian does, but in a static way (per client application). Let me try to give an example.

Currently it is the case that:

  • application A uses library Y and library Y uses library Z
  • application B uses a different derivation of library Y since this one uses library G as a replacement for library Z

In nixos this means that A, B, Y1, Y2, Z and G will be built from source. And equally that they all have to be downloaded from the cache to every machine who uses them. E.g. all of them have build costs and/or bandwidth costs.

My suggestion is now that nix builds A, B, Y1, Z and G. And that Y2 is then derived from Y1 but on the client.

In this way we save the build costs for Y2 and the bandwidth for Y2 and most importantly for me personally the time for either compiling or downloading Y2.

I am perfectly aware that the derived Y2 is not exactly the same as the source-built-Y2. But the derived Y2 would still be pure.

On another note:

There are some applications in nixos (for example rustdesk), that are always rebuilt from source on my machine, instead of being loaded from the cache. I understand that there are license issues here.

But this is terribly inconvenient, and for other distributions there are always convenient ways of installing such packages in binary form.

And remember convenience wins over people! The reason why Amazon is so successful is because it is so convenient. So if we want nixos to grow, then convenience should be the default (without again eliminating the alternative).

Hence here my suggestion is, to make the default way that nixos goes, to not build from source (unless the license issue can somehow be navigated, then of course building from source is preferrable). Instead the default version of the app might be a derivation that downloads a .deb package and makes it nix-compatible.

And to make this more nixy, the community might implement a derivation that generates that very same .deb package from source but does not publish the binary but whose result is simply the checksum of the .deb package. In this way we won’t run into license issues, the binary is downloaded from whereever it can be legally downloaded, and yet we have purity and source->bin checksum verification…

In fact we might do this also for apps that we can build and distribute from source. In this way we could drastically reduce the load on the nix cache and thus make nixos a less cost intensive endeavor…

2

WDYM by “long”? As in a few seconds or minutes long?

A nixos-rebuild with a few smaller changes should take 30s at most on reasonably performant hardware but closer to 15s in the normal case.

Most of the time spent here is eval (5-10s), to set up/tear down sandboxes for a bunch of tiny builds (5s) and then to activate the new generation (5s).

Feedback cycles with 10-15s pauses are slightly annoying in some cases but not too bad.

That depends on what you’re trying to do. If you just need access to some program on NixOS, you don’t actually need to touch your system installation in any way. That’s a FHS-distro thing, we don’t need to do that.

In such a case, you’d simply open a nix-shell -p packagenamehere and go about your business inside that shell. Opening a nix-shell is probably faster than any password privilege escalation or apt process (gotta process them triggers for libc6 and mandb).

Note that the parts concerning the Nix language in any way are only a portion of the time spent; usually just a few seconds.

While there’s certainly some room for improvement in the Nix evaluator, I think most of the time complexity stems from inefficient use of the module system. This is a trade-off between features and performance where each module adds a tiny little bit of slowdown, even if entirely unused.
Still, when optimising this part I’d expect the most you could hope to achieve to be a 50-70% reduction in the 5-10s this part takes.

You might not realise it but this is the status quo.

There is no such thing as “binary derivations” though. There are only “source” derivations and binary substitutes for them.

Binaries suitable for substitution are produced by realising (=building) a derivation and storing the produced output. This means each substitute must have been built by someone somewhere at some point and that’s the hard part.

If a binary substitute is available in the configured substituters, it is always fetched and unpacked instead of realising the derivation.

You “pick” the one and only available version for that package from the current index.

Not quite. The program you execute looks for its dependencies’ library file names in canonical locations such as /usr/lib/. All it does is “does the file /usr/lib/libfoo.so exist? Yes? Okay, map its content into my memory.” for every of its dependencies. It never interacts with apt during that process in any way; it’s all based on file location and name conventions.

This is grafting. See Grafts, continued — 2020 — Blog — GNU Guix. We don’t have such a mechanism yet but I don’t see why it shouldn’t be possible.

There’s quite a few of us who’d like to see something like that in Nixpkgs.

We also have such means and its often available for such “problematic” packages but it requires work and maintenance to provide these. If there’s a from-source derivation, usually nobody bothers because you can just let it build for a bit on any machine to get a “binary version” for “free”. If you don’t want to use your local machine, you can use any other machine via remote builds.

Amazon also has tonnes of money to throw at people for developing said convenience. It’s not that we don’t want it to be a convenient as possible, it just takes time and effort to make it that way.

Sorry but no. That is wrong on many levels and also simply not realistically possible. There is no generic way to make some arbitrary binary “nix-compatible” for instance; that’s now how any of this works.

Also if you want Debian packages so bad, why not simply use Debian?

There is no generic way to produce the exact same binary artifact from the same source. It’s an extremely hard problem to solve.

At that point you could simply fetch the binary via a regular fixed-output derivation.

4 Likes