`system.autoUpgrade` nearly halts my system, even though `nixos-rebuild` doesn't

Help
1

This post’s title states my problem in its entirety. Any advice?

There’s a stale issue on Github about this problem

The nixpkgs issue system.autoUpgrade causes high cpu and memory load has existed for three years, but has been silent for the last 15 months.

My interim workaround, in case anyone else needs it

Whenever I find my computer inoperable, I switch to a non-GUI shell (Ctrl-Alt-F1), log in, find the rebuild process (ps -ef | grep nixos-rebuild), kill it, switch back to KDE and rebuild the system with low priority (sudo nice -n 19 nixos-rebuild switch). Then it rebuilds very politely, without noticeably interfering with anything. But each step before killing it is extremely slow, so this is an unsatisfying solution.

2

Do you have some logs from the autoUpgrade service? The issue you linked make sense (I can’t consider it an bug per se), but in their case nixos-rebuild will give a similar result.

Could you describe your autoUpgrade configuration? During autoUpgrade running, do you notice huge I/O? Is it system using swap?

Cachix Deploy — documentation could be useful for your use case if you often need to rebuild packages in your system updates.

3

Thanks, @Solene!

Here’s the entirety of my autoUpgrade config:

  system.autoUpgrade = {
    enable = true;
    dates = "02:00";
  };

I’m using the nixos-unstable channel; would that mean I’m rebuilding packages often?

I don’t know how to answer your other questions. Should I use iotop the next time I notice nixos-rebuild slowing the system to see if it’s using a lot of IO? How would I check the process’s swap usagef? (/proc/<nixos-rebuild's id>/smaps looked like a likely candidate but I just looked at an example and those smaps files are really big; I’m not sure how I would use one.)

4

Trying looking at top or htop output and see if there are processes owned by users named “nixbld”, if so, this mean nixos is rebuilding something.

I’m surprised nixos-rebuild is working fine though, with just using nice.

And yes, because you are tracking nixos-unstable, you often have to recompile packages that aren’t in the cache yet.

You could try to look for options nix.daemonCPUSchedPolicy, nix.daemonIOSchedClass and nix.daemonIOSchedPriority to make nix daemon less agressive in general.

5

He may already have these options enabled. There’s a somewhat odd quirk to nix that it doesn’t use the daemon when run as root, so these options don’t take effect in that case. (This is the main reason I use nixos-rebuild --use-remote-sudo instead of sudo nixos-rebuild) If the autoupgrade service is running nix as root (I think it does? Not very sure though. Never really wanted autoupgrade behavior, myself), then that could be why the compiles are grabbing resources abnormally.

1 Like