I don’t think this is a NixOS issue but I’m asking here just in case and because I expect other fora would probably tell me to go here first.
So about half a year ago I bought two new harddrives and installed NixOS (used to be Ubuntu) on my server, an old PC that I repurposed.
Ever since I’ve had issues where sometimes on high disk usage userspace would just hang and not come back until a force reboot. I’ve had it happen on downloading new derivations into the Nix store, switching to a new configuration, seeking within media files streamed off of the hard drives, and the one that seems to trigger it the most, downloading files via qBittorrent. I think it’s userspace hanging from the fact that any services running on the server go down, websites, SSH, DNS, and email, but the kernel keeps running; my Wireguard VPN works fine. I unfortunately do not know how to reliably reproduce it. Sometimes it happens sometimes it doesn’t. On average it happens about once per week but it has also happened a couple minutes after booting. This did not happen before I installed NixOS and the harddisks.
Initially I thought this was an OOM situation (seeing the 4GB RAM) so I configured swap and systemd-oomd, but this didn’t have any effect. I now think it’s a disk issue because of the fact that it happens on high disk usage and the fact that journald doesn’t save any logs of something going wrong.
I managed to get some kernel logs out via netconsole but I do not really understand what they mean. Clearly it’s something going wrong with the disks seeing how all the hung tasks seem to involve some write subroutine of the kernel. The first log does not have the full boot log in there because I messed up on setting up loglevels initially and I don’t remember which boot it came from but I did not see anything out of the ordinary in its logs and it was pretty similar to the full log I do have. It seems tasks interacting with both harddisks and the SSD hang so I don’t think it’s anything wrong with the disks themselves, SMART didn’t report anything wrong with the disks either besides the high amount of force resets I’ve had to do to recover from a hang.
So now I’m basically asking, does anyone have any idea what could be the problem? I have no idea and I’m getting sick of the downtime, especially since the server is in a closet at my parents’ so I cannot just go up to it to reset it unless I’m visiting them. I tried reseating the SATA cables, I tried adding swap and systemd-oomd as mentioned earlier. I have no clue what to do further to diagnose this problem.
Details about the system:
NixOS 25.11, 25.05 at installation, they both had this problem, I haven’t had time to install 26.05 yet.
Acer Aspire XC-705
Intel i5-4460
4GB DDR3
256GB Sandisk SandForce Driven SSD, BTRFS with the rootfs and 10 GB swap on subvolumes, and the efi partition
2 8TB Seagate IronWolf HDDs, with BTRFS software RAID1 containing /var, /home and media files
This problem did not occur on this system running ubuntu and an older HDD containing the rootfs and the SSD containing data. I installed NixOS at the same time as I replaced the HDD with two new ones.
SMART data: SMART data of the three disks - Pastebin.com
First kernel log: First kernel log - Pastebin.com
Second kernel log: Second kernel log - Pastebin.com
If anyone knows or has some suggestions, I would really appreciate it!