I have a Raspberry Pi 4 running NixOS, but am having issues where the system becomes completely unresponsive when I run nixos-rebuild. I did wonder if the SD card I was using was too slow, but have replaced it with a high-class one to no avail. Not sure how to diagnose what’s happening here—any pointers would be appreciated. Currently having to power off the machine, try again and hope for the best. Often it works after a few goes.
1 Like
My first guess would be it runs out of memory. Second guess it gets a bit too hot and throttles the cpu.
To diagnose you could run
top
and
while true; do sleep 10; free -h; done
and
while true; do sleep 10; cat /sys/devices/virtual/thermal/thermal_zone0/temp; done
in separate terminals when running nixos-rebuild
Thanks for the help!
Judging from the output of your free command, I think I’m running out of memory:
total used free shared buff/cache available
Mem: 883Mi 331Mi 403Mi 8.0Mi 148Mi 462Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 331Mi 403Mi 8.0Mi 148Mi 462Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 335Mi 396Mi 8.0Mi 151Mi 458Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 335Mi 395Mi 8.0Mi 151Mi 458Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 342Mi 388Mi 8.0Mi 151Mi 451Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 343Mi 388Mi 8.0Mi 151Mi 450Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 344Mi 379Mi 8.0Mi 159Mi 448Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 674Mi 59Mi 8.0Mi 149Mi 119Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 797Mi 32Mi 8.0Mi 53Mi 22Mi
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 883Mi 796Mi 32Mi 8.0Mi 54Mi 21Mi
Swap: 0B 0B 0B
Temperature looks fairly stable:
46738
47712
46738
47712
49173
48686
47712
Can you suggest any steps for reducing memory usage? Or do I just need to add some swap?
Check if your /tmp is mounted on tmpfs or SD card partition. You might need to override boot.tmp.useTmpfs and before that mount /tmp to a disk partition/bind mount for rebuild.
The default for boot.tmp.useTmpfs is false, so they aren’t likely using it. And I mean their pi appears to only have 1G total RAM. You don’t need boot.tmp.useTmpfs to run out of that little memory with a nixos-rebuild.
Swap is unlikely to help. Nix is actively using the memory it’s allocating, and swap is only good for memory that isn’t actively being used. You’ll be thrashing between RAM and swap space, and the performance is going to be so bad that it’s indistinguishable from the crashing you’re already seeing.
Unfortunately the answer is to use a machine with more memory. Now, luckily, you can eval your config on some other x86_64 machine without building it, copy the resulting .drv closure over to the pi, and tell the pi to build that, and the pi won’t have to eval any Nix code. An automated way would be, on any machine with more memory, use nixos-rebuild --build-host pihostname --target-host pihostname switch, which will eval locally but automatically SSH into the pi to build and deploy the build result. There are other tools like deploy-rs or colmena that are designed around this sort of workflow.
Swap would help. But with only (a bit less than) 1GB of memory, swap usage during nixos-rebuild can be hefty. But worth a try. My smallest system - Librem 5 - has 3GB of memory and is capable of running nixos-rebuild including kernel compiles.
That makes sense, thank you! Having some issues with entering the password on the remote system. I think I’ve gotten past that by adding -tt to NIX_SSHOPTS, but the Pi still seems to then be locking up unfortunately. Maybe I just need to invest in one with more RAM!
1 Like
Today every time I tried to run nixos-rebuild on my Raspberry Pi 3B, the command crashed because it was out of memory. I solved the problem by enabling zram (zstd, 4 * ram). Before the first nixos-rebuild, I manually setup-ed zram following the Arch wiki.