I’m running NixOS on a small VPS (1-core CPU, 1GB RAM), with the root partition encrypted through LUKS.
It was running fine on NixOS 25.11, but I tried upgrading to NixOS 26.05 yesterday and it now feels much slower. In particular, systemctl status (for any service) takes more than 10 seconds to output, and in the meantime I’m able to run top in another console and see that some kworker:kcryptd process is taking up half the CPU.
I guessed this could be due to the new systemd-based initrd, but I’ve double-checked my config according to the changelog, and even setting boot.initrd.systemd.enable = false doesn’t solve the problem.
Any idea what could be causing this?
My configuration is here
and uses some personal modules from here
Unrelated, but let me also mention that I tried just reverting the upgrade but that broke my headscale instance, so now I’m stuck with the slow-but-working 26.05 system.
I notice you’re using a LUKS disk. Reading/scanning large amounts of data from disk could cause kworker:kcryptd to use plenty of CPU, especially on a 1-core system. This can be triggered by systemctl status scanning for log items related to your service.
How big are your current logs? You can check with journalctl --disk-usage. If they are large, you may wish to enable some space saving options:
services.journald.extraConfig = '''
SystemMaxUse=200M
'''
As to root cause, I’m not sure why this is worse on 26.05. Perhaps because more of early boot/dmesg now flows into the journal? boot.initrd.systemd.enable = false doesn’t remove entries that are already logged, so I would not expect setting that would immediately fix your issue without triggering a vacuum.
In the interim, if you don’t need all the output from systemctl status, you can try systemctl is-active to check if the service is running, then run journalctl -b -u <unit> to get the logs when you need them.
1 Like
I see, thanks! The journal entries were indeed taking up 2.9GB, I cleaned them with --vacuum-time and the systemctl status time is now back to normal.
As to root cause, I’m not sure why this is worse on 26.05. Perhaps because more of early boot/dmesg now flows into the journal?
I tried rebooting and that did not increase the log size by more then 1M, so that doesn’t seem to be the cause. I’d expect something to do with the first boot under systemd or the upgrade from 25.11 to 26.05, as the problem disappeared when rolling back to 25.11 without touching the journal. But I can just clean the journal manually or limit its size, as you suggested, if I ever encounter the problem again.
2 Likes
systemctl status (for any service) takes more than 10 seconds to output
I’m not sure whether there was a regression that made this even worse, but this is a well known issue that boils down to the journal binary format being terrible. See
Showing status of service via systemctl is slow (>10s) if disk journal is used · Issue #2460 · systemd/systemd · GitHub