General question: how to avoid running out of memory or freezing when building Nix derivations

[Aleksanaa]

I find this is a common complaint, and there doesn’t seem to be a completely satisfactory answer.

The goal here is to: When using Nix in a desktop environment, the build process will not cause

1. Any part of the desktop environment or other running software is forcibly exited (OOMed)
2. Other simple UI operations can still be completed smoothly, such as opening a new file manager or using a browser. IO-related operations can still continue.
3. Obviously, it shouldn’t freeze completely or crash the desktop session.

[Aleksanaa]

What I do now is approximately:

{
  nix = {
    daemonIOSchedClass = lib.mkDefault "idle";
    daemonCPUSchedPolicy = lib.mkDefault "idle";
  };
  # put the service in top-level slice
  # so that it's lower than system and user slice overall
  # instead of only being lower in system slice
  systemd.services.nix-daemon.serviceConfig.Slice = "-.slice";
  # always use the daemon, even executed  with root
  environment.variables.NIX_REMOTE = "daemon";
}

[waffle8946]

Lowering max-jobs and max-cores could help with builds as well, though it won’t help with eval.
Based on my experience I’d say the system should have 8 gigs of RAM minimum and 16 gigs to feel comfortable in general.
Also keeping a web browser (including electron apps) open will massively increase the likelihood of OOMing.

[arianvp]

There is cgroup support in Nix which in combination with systemd-oomd could help protect the rest of the system from a single build causing random OOMs. This way OOM will only kill the build causing a lot of memory usage instead of random processes

[waffle8946]

@arianvp

There is cgroup support in Nix

How so? The closest I found (4 years old, to be fair) was

Managing Nix build resources with cgroups?

If you want to constrain the nix-daemon (i.e. all builds, not individual ones) using cgroups (via systemd) you could use something like this: systemd.services.nix-daemon.serviceConfig = { MemoryHigh = "5G"; MemoryMax = "6G"; }; There are a lot of options (see e.g. systemd.resource-control and systemd.exec). If you want to instead constrains individual builds then it’ll be more difficult and probably require patching Nix (but I haven’t looked into this).

Did you mean something like this?

[arianvp]

Enable-cgroups nix setting causes it to create a cgroup per build. It’s relatively new. And only the version in the latest nix version (or maybe master?) has the cgroup tree in the right shape such that systemd works

[Balssh]

I guess it depends on the CPU? I have a Ryzen 7 5800H and 16GB of RAM and if I’m not limiting my cores to ~8-10 I go into OOM.

[waffle8946]

It depends on what you are evaluating. Evaluating all of nixpkgs, for example, took about 12 gigs last time I tried it (which doesn’t leave much memory for the system with only 16 gigs…). Same with builds, if I try to do a world rebuild of my system’s closure, I will even OOM on a 64 gig system unless I limit jobs.