Feature request: performance restricted rebuilds

Development Nix
1

Hi!
I would love a nix cli option that tries on a nix-level that would try to make builds not impact a devices performance - even if that would mean a e.g. 10x increase in build time. Putting the feature directly into nix might be great! But also just implementing it for NixOS’ system.autoUpgrade might be a good start.

I am not sure how that would in practice work out - that is beyond my expertise. Probably either nice+ionice idle and/or putting the own process in a custom cgroup with changed cpu shares, swappiness and OOM thresholds.

Best Regards,
Thomas131

2 Likes
2

https://nix.dev/manual/nix/2.24/advanced-topics/cores-vs-jobs

2 Likes
3

Thanks! This is a great start which I didn’t find until now!!

4

This option is useful as well if you are on a desktop: nixos/default.nix at 143d0e1ea4fac7e2b5ff9f556f8d073fad3827b0 - nixos - Gitea: Git with a cup of tea

2 Likes
5

If you also want to set some resource restrictions, look into options within systemd.services.nix-daemon.serviceConfig, such as CPUQuota, CPUWeight, IOWeight, OOMScoreAdjust, etc. See systemd.exec and systemd.resource-control for more info.

There’s also the use-cgroups setting in nix.conf (which requires the cgroups experimental feature), I haven’t got much understanding of that though.

1 Like
6

Thank you all! I will try that for now and see:

  system.autoUpgrade.flags = ["--max-jobs" "1" "--cores" "1"];
  nix.daemonCPUSchedPolicy = "idle";
  nix.daemonIOSchedClass = "idle";
#  nix.daemonIOSchedPriority = ;
  systemd.services.nix-daemon.serviceConfig = {
    OOMPolicy="stop";
    OOMScoreAdjust=100;
#    Nice=19; # It is claimed that this has no effect https://github.com/NixOS/nixpkgs/pull/138741#issuecomment-924955890

    #Already set with nix.-Settings
#    CPUSchedulingPolicy="idle";
    #CPUSchedulingPriority = ;
#    IOSchedulingClass="idle";
    #IOSchedulingPriority = ;
  };

Ideally I could choose on which build jobs to apply this settings (only to automatic ones or all - depending on the machine). But often the world is not perfect. Thank you all!

A related GitHub Issue btw: Feature: Enable specifying scheduling priority of nix builds · Issue #3020 · NixOS/nix · GitHub

7

I set those cpu and io scheduling options for the autoupgrade service as well - because it’s run as root, the work is done directly rather than via the daemon.

  systemd.services.nixos-upgrade = {
    # …
    serviceConfig = {
      CPUSchedulingPolicy = "idle";
      IOSchedulingClass = "idle";
      # …
    };
  };

This also means that interactive builds, typically done with sudo nixos-rebuild switch will not be niced either; just something to be aware of. I think those will still be limited by cores/jobs nix settings, but the trouble with those is they can often make things slower than needed, rather than just less impactful using priorities.

1 Like
8

Sure, I suppose using --remote-sudo might avoid that issue?

9

Thanks @uep - this is very helpful!!

@waffle8946 Thanks! That would in some cases also be an option. But to complicated for me now.

1 Like
10

Are you sure this works? I thought the work gets passed to nix-daemon and won’t stay in the nixos-upgrade cgroup.

11

How is it complicated? Just use the flag --use-remote-sudo instead of sudo.
That will cause nixos-rebuild to only up privileges when deploying the config, rather than running the entire command as root.

12

@waffle8946 Oh sorry, I thought this is for deplying to remote machines - sorry! And you mean for the unprivileged case it is easier to manage performance? I don’t see that helping …

13

@uep claims the build running as root will cause problems. This prevents the build from running as root. (Of course, the deployment must run as root.)

14

It’s my understanding, not carefully tested for specific scenarios, that:

  • nix passes the work to the nix daemon, when invoked as another user (and subject to trust settings)
  • nix runs the tasks directly, when invoked as root, bypassing the daemon (and any resource controls applied specifically around it).

My workaround is to add them to the autoupgrade service as well, but one could also run that service as a different user and thereby have it pass the work to the daemon.

15

I think the linked comment modules/nix-daemon: Replace daemon(IO)NiceLevel options by illdefined · Pull Request #138741 · NixOS/nixpkgs · GitHub is slightly incorrect, Nice = 19; does have a (limited) effect. If I set it without any other options, the nice build processes take up at most 50% of CPU time.

In any case I would expect nix.daemonCPUSchedPolicy = "idle"; to be more effective.

16

There is --store daemon and --option store daemon (some nixos commands only accept the latter.) This forces the command to use the daemon, you can also set store in nix.conf nix.conf - Nix Reference Manual.

1 Like