NixOS boots really slowly

Help
1

Hello, I have been using NixOS for a few months now and I absolutely love it, however an issue that has plagued me for a long time is the startup time. Not only does it take a long time to get to SDDM, anything past that takes even more time (Upwards of 2+ minutes). I’d appreciate any help in this regard.

my systemd-analyze

24.492s dev-bus-usb-002-003.device
24.492s sys-devices-pci0000:00-0000:00:1d.0-usb2-2\x2d1-2\x2d1.8.device
21.808s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1\x2dpart2.device
21.808s dev-disk-by\x2did-ata\x2dWDC_WD3200BPVT\x2d24JJ5T0_WD\x2dWXQ1EA1FZFUP\x2dpart2.device
21.808s dev-disk-by\x2duuid-0EC49547C4953247.device
21.808s dev-sda2.device
21.808s sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda2.device
21.808s dev-disk-by\x2dpartuuid-f71f8baf\x2d02.device
21.808s dev-disk-by\x2ddiskseq-2\x2dpart2.device
21.808s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.0\x2dpart2.device
21.808s dev-disk-by\x2did-wwn\x2d0x50014ee6ad37a6d1\x2dpart2.device
20.857s dev-disk-by\x2dpartuuid-f71f8baf\x2d04.device
20.857s sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda4.device
20.857s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1\x2dpart4.device
20.857s dev-disk-by\x2duuid-bf6dbd31\x2df857\x2d46a1\x2da7f3\x2d786a01ab745c.device
20.857s dev-disk-by\x2did-ata\x2dWDC_WD3200BPVT\x2d24JJ5T0_WD\x2dWXQ1EA1FZFUP\x2dpart4.device
20.857s dev-sda4.device
20.857s dev-disk-by\x2ddiskseq-2\x2dpart4.device
20.857s dev-disk-by\x2did-wwn\x2d0x50014ee6ad37a6d1\x2dpart4.device
20.857s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.0\x2dpart4.device
20.662s dev-disk-by\x2dpartuuid-f71f8baf\x2d03.device
20.662s dev-disk-by\x2ddiskseq-2\x2dpart3.device
20.662s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1\x2dpart3.device
20.662s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.0\x2dpart3.device
20.662s dev-sda3.device
20.662s sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda3.device
20.662s dev-disk-by\x2duuid-5d534d7d\x2d144f\x2d4225\x2db887\x2dcf3d041f07cb.device
20.662s dev-disk-by\x2did-wwn\x2d0x50014ee6ad37a6d1\x2dpart3.device
20.662s dev-disk-by\x2did-ata\x2dWDC_WD3200BPVT\x2d24JJ5T0_WD\x2dWXQ1EA1FZFUP\x2dpart3.device
20.652s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.0\x2dpart1.device
20.652s dev-disk-by\x2did-ata\x2dWDC_WD3200BPVT\x2d24JJ5T0_WD\x2dWXQ1EA1FZFUP\x2dpart1.device
20.652s dev-disk-by\x2duuid-CE828E70828E5CB9.device
20.652s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1\x2dpart1.device
20.652s sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda1.device
20.652s dev-disk-by\x2did-wwn\x2d0x50014ee6ad37a6d1\x2dpart1.device
20.652s dev-sda1.device
20.652s dev-disk-by\x2dlabel-System\x5cx20Reserved.device
20.652s dev-disk-by\x2ddiskseq-2\x2dpart1.device
20.652s dev-disk-by\x2dpartuuid-f71f8baf\x2d01.device
17.809s dev-ttyS2.device
17.809s sys-devices-platform-serial8250-tty-ttyS2.device
17.808s dev-ttyS0.device
17.808s sys-devices-platform-serial8250-tty-ttyS0.device
17.805s sys-devices-platform-serial8250-tty-ttyS1.device
17.805s dev-ttyS1.device
17.794s sys-devices-platform-serial8250-tty-ttyS3.device
17.794s dev-ttyS3.device
17.074s sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.device
17.074s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.device
17.074s dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.0.device
17.074s dev-disk-by\x2ddiskseq-2.device
17.074s dev-disk-by\x2did-ata\x2dWDC_WD3200BPVT\x2d24JJ5T0_WD\x2dWXQ1EA1FZFUP.device
17.074s dev-sda.device
17.074s dev-disk-by\x2did-wwn\x2d0x50014ee6ad37a6d1.device
11.298s systemd-remount-fs.service
10.809s home-manager-hikari.service
 7.734s systemd-modules-load.service
 7.106s dbus.service
 6.888s accounts-daemon.service
 5.668s iwd.service
 5.552s systemd-tmpfiles-clean.service
 5.224s firewall.service
 5.040s systemd-journal-flush.service
 2.831s udisks2.service
 2.658s upower.service
 1.998s NetworkManager.service
 1.944s systemd-udevd.service
 1.600s mount-pstore.service
 1.546s audit.service
 1.482s resolvconf.service
 1.047s systemd-random-seed.service
  963ms boot.mount
  957ms systemd-sysctl.service
  898ms tlp.service
  886ms logrotate-checkconf.service
  853ms modprobe@drm.service
  700ms user@1001.service
  685ms dev-zram0.swap
  656ms dev-hugepages.mount
  654ms dev-mqueue.mount
  581ms sys-kernel-debug.mount
  495ms polkit.service
  419ms systemd-zram-setup@zram0.service
  370ms modprobe@configfs.service
  342ms systemd-logind.service
  333ms systemd-udev-trigger.service
  315ms modprobe@fuse.service
  308ms sys-fs-fuse-connections.mount
  248ms systemd-tmpfiles-setup-dev.service
  239ms systemd-journald.service
  237ms sys-kernel-config.mount
  224ms systemd-backlight@leds:dell::kbd_backlight.service
  216ms systemd-backlight@backlight:acpi_video1.service
  208ms systemd-backlight@backlight:acpi_video0.service
  126ms network-setup.service
  120ms systemd-timesyncd.service
  110ms systemd-update-utmp.service
   67ms display-manager.service
   64ms systemd-tmpfiles-setup.service
   55ms systemd-user-sessions.service
   47ms rtkit-daemon.service
   40ms modprobe@efi_pstore.service
   36ms kmod-static-nodes.service
   25ms user-runtime-dir@1001.service
   17ms network-local-commands.service
   13ms NetworkManager-dispatcher.service
    7ms nscd.service

my system configuration

System:    Host: Tsuki Kernel: 6.2.12-xanmod1 x86_64 bits: 64 Desktop: KDE Plasma 5 
           Distro: NixOS 23.05 (Stoat) 
Machine:   Type: Laptop System: Dell product: Latitude E6420 v: 01 
           serial: <superuser required> 
           Mobo: Dell model: 038C0K v: A00 serial: <superuser required> BIOS: Dell v: A25 
           date: 03/06/2018 
Battery:   ID-1: BAT0 charge: 4.8 Wh (24.7%) condition: 19.4/40.0 Wh (48.6%) 
CPU:       Info: Dual Core Intel Core i5-2540M [MT MCP] speed: 1794 MHz 
           min/max: 800/2601 MHz 
Graphics:  Message: No device data found. 
           Device-1: CN0CJ3P2724871A90GSRA01 Laptop_Integrated_Webcam_FHD type: USB 
           driver: uvcvideo 
           Display: wayland server: X.org 1.21.1.8 driver: loaded: N/A 
           resolution: <missing: xdpyinfo> 
           Message: Unable to show advanced data. Required tool glxinfo missing. 
Network:   Message: No device data found. 
Drives:    Local Storage: total: 298.09 GiB used: 55.91 GiB (18.8%) 
Info:      Processes: 184 Uptime: N/A Memory: 3.72 GiB used: 1.83 GiB (49.3%) Shell: Zsh 
           inxi: 3.3.04

and my journalctl output https://0x0.st/HNf5.txt

2

I do know that my HDD takes the longest time to bootup but I do not have any idea how I can make it take less time, I use btrfs.

3

A hard disk is going to feel slow for any system these days, and booting involves a lot of seeking and cold caches.

On top of that, it sounds like you suspect there may be issues with the disk, like it takes a long time to become ready at power on? Here are some ideas for further investigation:

  • It might be old and worn and struggling to read sectors here and there, maybe spending a lot of time on retries.

    • You can run some disk benchmark programs, and watch iostat output, looking for large pauses.
    • You can look at disk SMART stats and see if there are any clues (large number of reallocated sectors, lots of pending sectors, etc).

  • This is a 4k-native, 512-emulation drive, and one of the earlier of these. WD called them “advanced format”. But it might not report the truth in the same way more modern drives do.

    • If your filesystem is treating it as a 512-native sector drive, then small writes (metadata updates) will require a read-modify-write cycle and will be slow. I’m sorry, I’m not familiar enough with btrfs to know whether there are configuration options relevant for sizing here; perhaps others have suggestions.
    • Or if the partition boundary is misaligned with the underlying 4k sectors, the emulation requires a read-modify-write for every write and it will be awful. You can check the partition alignment and sizes with smartmontools, which you’ll want for the above steps too.
    • Some WD drives (and I’m guessing this is probably one of them, but I’m not sure about 2.5" units) also had a jumper to basically offset the LBA by one. If that jumper is set wrong then you can have the same problem (but don’t change it now because it will make your data incomprehensible to the filesystem). This can make it hard to tell the true alignment. If you can wipe the disk, the best option is to benchmark with combinations of settings.

Well, if you can wipe the disk, the real best option is to replace it with an SSD, but I’m assuming you want to start with diagnostics on the existing hardware.

Addendum: I just noticed: 4G RAM. This is not going to allow much to stay in cache, so it will also be working the disk a lot harder re-reading disk blocks on each use. An SSD will likely make quite a difference to the responsiveness of this system, but you’re going to hit other limits soon too.

4 Likes
4

It seems buying an SSD and increased RAM may be the better idea. Although by switching to AHCI mode in BIOS I managed to shave off a few seconds, I’ll try your suggestions. Thank you!

1 Like
5

It is correct that a HDD will seem a bit slow today but there’s a lot you can do to improve. Essentially you want to delay or remove any IO heavy startup services.

For instance, the nix-gc.service is going to be IO heavy but doesn’t need to be run immediately on boot. You could set nix.gc.randomizedDelaySec to 10m to avoid it ever starting during the boot process.

Also, I’m not sure if the journald log or systemd-analyze will show data from the initrd process; I have found a number of options that can cause very slow boot and should be disabled if not needed.

fileSystems."/".autoResize = false;
fileSystems."/boot".autoResize = false;

Also on a HDD (or even SSD), I highly recommend mounting all filesystems with noatime to reduce the number of extraneous IO operations.

I recommend just using zram instead of an actual disk swap and it looks like you already are. I’m not sure if it’s still an issue but you might want to disable reloading the “zram-reloader” service on configuration changes. It won’t make the boot process faster but it will prevent having to recreate the zram during a “nixos-rebuild switch”.

systemd.services."zram-reloader".restartIfChanged = lib.mkForce false;

Of course the Archlinux wiki has a lot of info on performance tuning and most applies to NixOS as well: Improving performance - ArchWiki

1 Like
6

I had intended to mention atimes as part of the discussion about causes of small writes.

It’s interesting about the impact you see from the resize option, is there some clear reason why they’re slow? Are they reading a whole lot of filesystem metadata (~like a mini-fsck) before determining that the size is already correct? Does it depend on filesystem type? Could we improve this generally, if it’s real, or perhaps store some (non-essential) state that can skip it on subsequent boots? I expect the main use case is cloud hosts and other VM’s starting with a cloned base disk image.

7

False is the default for that option.

I would be quite surprised if an HDD is capable of boot times less than something like a minute and a half on any modern desktop OS, and that’s probably only if you disable like… everything and just use a VT or a really lightweight GUI. Most IO in desktop usage is random small block IO, which is the worst case for any drive but it’s even way worse specifically on HDDs.

A 2.5" SATA SSD of a greater capacity than the drive @HikariNee already has in that machine is pretty darn cheap these days. I don’t like telling people “just spend money” to solve problems, but in this case it’s really difficult to argue otherwise.

1 Like