How to make Nixpkgs more eco-friendly / use less resources

It also has the problem that most users won’t be upgrading between the same versions as hydra is. The diff between builds in hydra is between very nearby commits, whereas users are likely to be upgrading between commits that are weeks apart. We’d have to keep diffs between HEAD and more than one previous commit, which would only exacerbate the storage problem.

EDIT: Plus the binary cache protocol would then get more complicated. User machines would have to ask the cache “I need this path, what paths do you have diffs from for it?” “I have diffs with these paths” “Ok I have that one, please give diff”

Is the current nix store design a real dead end with regard to reducing resource usage? :confused:

Not necessarily.

We probably need to increase the substitution granularity, though. I wrote an article evaluating the efficiency of some potential solutions to this problem earlier this year: Nix Substitution: the Way Forward

The TL;DR being: adding an output-adressed chunk/file store would greatly reduce the amount of data we’d have to download in most use cases.

3 Likes

Has anyone ever considered using something like bitorrent or IPFS to share the distribution load?

I’ve written https://github.com/input-output-hk/spongix which splits NARs into chunks using https://github.com/folbricht/desync and saves roughly 80% of space (always depends on the contents of the cache, of course).
The remaining issue is that Nix won’t down-/upload those chunks directly, so the transfer is still as inefficient as always.
Implementing chunking in Nix may work for uploading, but downloading would need a separate chunk store and the end result would be about 20% additional disk overhead because the /nix/store still needs to be uncompressed.
One alternative for that would be FUSE, but that’s going to be slow and probably buggy, as well as restricted to a few platforms, but in theory you’d trade time for space, if that’s critical.

6 Likes

How does it compare to Nix-casync, a more efficient way to store and substitute Nix store paths?

1 Like

there is a work in progress Nix × IPFS — Milestone 1

1 Like

Won’t the binary change every time the store path of one of its dependency changes? The binary rpath must contain those paths.

1 Like

The Solaris IPS packaging system did away with transferring package archives entirely, and transfers individual files, with a content-addressing scheme. This was in large part because even when a package update changes something, many files are unaffected and can be reused between versions.

While you might lose a little efficiency from single-archive-compression (many doc files together, etc) it seems that this efficiency comes instead from avoiding repeated downloads.

Because of reproducibility, it’s hard to skip rebuilds, guix has a “graft” system for packages update with really minor changes to avoid recompiling the whole dependency graph, but I suppose it kills reproducibility?

There already is, in the typical Nix fashion it’s called replaceRuntimeDependencies. It works by going through every file in the system and replacing the store path of the original package with a replacement.

It won’t work for any package update, though: it assumes the replacement store path to have the same length as the original. So, you could do this for security patches and minor bug fixes.

If the goal is to increase NixOS sustainability, I’m not sure using IPFS will be an improvement: the IPFS node software is a big resource hog. I admit it’s been a few years since the last time I tried it, but the node in idle would use a few cpu percents and several Mb/s of bandwidth with just a handful of pinned files.

IPFS is all nice (at least in theory) and it gives file level granularity, but we could probably go away with just distributing nars using plain old torrents much more efficiently.

1 Like

IPS was incredibly slow, I’m curious to know if this is due to updating files one by one.

IPFS draws CPU when it’s actively contributing to the P2P network routing, this is not mandatory and not really useful if you use it locally to access IPFS content. It got a better wrt resources too.

1 Like

I’m not sure exactly, and honestly my experience wasn’t too bad with it, but I’ll posit some combination of:

  • insufficient download parallelism and/or streaming with earlier HTTP
  • iops amplification and latency cascades with small files and duplicated metadata updates
    • likely running on early zfs, which had some pretty strong transaction commit latency
    • likely running on spinning media with short concurrency queues
    • conservative sync writes in the package manager
    • the need to keep two copies of the file (or hardlink in some cases?); one in the store and one in the system
  • different expectations; it may have seemed slow compared to (say) apt (on ext4), but it was already faster and more convenient that the previous Solaris pkg system so there was plenty of room to start with conservative implementation with room for further optimisations

I wrote Spongix after evaluating nix-casync. Compared to it, Spongix offers:

  • garbage collection based on LRU/max-cache-size and integrity checking
  • metrics
  • proxying and caching multiple upstream caches behind itself
  • uploading chunks to S3 compatible stores
  • signing narinfos if no signature is present (for automated build farms on top of Cicero)

It probably has a few more features that i forgot about, but we’ve been running it in production at IOG for a few months now and it performs much better than our previous setup with Hydra->S3.

6 Likes

CA derivations are strictly better, but It is really hard to get shit merged in this community — thus Hydra still doesn’t have support for them upstream, and we cannot being testing thing in Nixpkgs.

I remain incredibly frustrated that we have this thing 90% done, and there is no will to unblock willing maintainers ready to do work getting the feature out in the real world.

7 Likes

Part of the problem is that the roadmap targets 3.0 for flakes and CA for 4.0.

We are not 3.0 yet, and flakes are far from being ready, despite what Eelco says…

And last time I tried CA, despite having set up the CA enabled cache, it bootstrapped some compilers, which ultimately caused a world-rebuild. Also cachix does not yet support CA.

And what annoys me with CA: Its design still relies on a single authority per cache to transform IA to CA as known by that particulat cache.

What we need instead is a distributed “trust” network that can point from IA not only to the CA, but also to a “store”/“mirror”/“cache”/IPFS or whatever to download it.

3.0 is already a big giant mess that we cannot review, we need to minimize the scope. Flakes is a huge ball of unaudited complexity we are in no way ready to stabilize in one go.

3 Likes

I am not saying Flakes should be 4.0 and CA 3.0, but we should focus on layering so we can stabelize e.g. part of the new CLI (stuff like nix show-derivation) without worrying about Flakes.

8 Likes

This is an awesome tool!

Could this be modified to be used by end users (people not building projects) as a way to improve nix efficiency?

Hosted by Flying Circus.