Nix build RAM expectations

Development
1

Given a) Nix build ate my RAM 😭 b) openscad-unstable: init at 2024-01-22 by pca006132 · Pull Request #283651 · NixOS/nixpkgs · GitHub …
do we have and should we have some kind of guidelines how much RAM per build core are Nix builds supposed to be given, and what is the preference order of the mitigations after that point?

Presumably multi-resource build scheduling is too much of a pain to ask Nix to perform.

2 Likes
2

I’m unsure how nix build works under the hood, but here’s what I think.

  • A soft and/or hard limit should be something the user can set before the execution of a build.
  • Once any limit is reached, the build (if able to split into minor tasks under the hood) should intelligently schedule tasks to be completed before new ones are started. That way we don’t have the build continuously fighting the system for more resources.
3

That needs all the upstream build systems to fully support fine grained load control, doesn’t seem to be there.

Nix scheduling improvements are more realistic but still a lot of work.

But a question we probably could answer today is: how much RAM per core is expected before you should tell Nix to use less parallelism, and correspondingly what are the preferred measures to take on the Nixpkgs side if a build needs more RAM per thread. (Manually halve the impure-passthru core count variable at build time? Switch toolchains? Restrict parallel building? …)

4

The (future) make-jobserver connecting all builds on a single machine – that one could take free RAM into account and start reducing the number of tokens in circulation when under pressure (say with zram as a buffer).

5

Maybe we can have a memory limitation using cgroup for a build task, and when the oom event is triggered (monitored via cgroup event notification) we can kill the build task and restart it with half the number of cores until the core number is 1.

6

I think even with these solutions it would help to have an official guideline like «if your system has less than XGiB RAM per core, you will have problems all the time, if your Nixpkgs contribution wants more than YGiB RAM per core, you should drastically reduce parallelism and leave a comment in the source». I have a feeling that X≥1 and Y≤4, but maybe a more precise estimate of the expectations can be obtained (and then documented)?