Raspberry Pi has a hardware watchdog. It works on NixOS, as long as you set systemd.watchdog.rebootTime to 15 seconds or less (this is the longest supported by the hardware watchdog).
You can watch how often it is pinged with:
$ systemctl show | grep Watchdog
WatchdogDevice=/dev/watchdog0
WatchdogLastPingTimestamp=Tue 2024-10-15 17:25:28 MDT
WatchdogLastPingTimestampMonotonic=6964105699
RuntimeWatchdogUSec=15s
RuntimeWatchdogPreUSec=0
RebootWatchdogUSec=15s
KExecWatchdogUSec=0
ServiceWatchdogs=yes
Running a forkbomb as root freezes as expected, then reboots about 15 seconds later. Excellent.
I’ve seen several tutorials for other distros also refer to configuring /etc/watchdog.conf, which can take a number of more advanced / granular settings that seem like they would be handy: https://linux.die.net/man/5/watchdog.conf but I’m having a hard time figuring out whether or not they are taking effect on my NixOS Pi. The instructions generally suggest installing watchdog from the default system package manager.
I’ve tried using the test-binary setting to see if I can output some text to a file upon the watchdog rebooting the device, but it doesn’t seem to do anything. I’m not sure if I’ve misconfigured something, or if it just doesn’t work.
Ultimately my goal is to use the interface setting to try to keep my Pis online, as they seem to be unreachable from SSH and require a reboot every few days, regardless of the watchdog.
Does anyone know if environment.etc."watchdog.conf" can be used to configure the watchdog on a Pi? Thanks in advance for ideas.
I actually had trouble tracking down what package that watchdog.conf file relates to. It’s some barely maintained repo, and it’s nowhere near the top of the search results for “raspberry pi watchdog”. So I think you’re looking at a fairly obscure daemon for configuring a watchdog, which isn’t available in nixos.
However, using the nixos options search, there does appear to be another fairly comprehensive watchdog daemon already packaged and available in nixos called watchdogd. So probably take a look at its man page.
3 Likes
Yes, I ran across that old sourceforge page. I’m not sure about barely maintained – I see many commits about 18 months ago (which doesn’t seem awful for a presumably mature project, as its installation seems to be recommended for debian, fedora, others).
Hmmm, I see a lot of blog and SO posts saying that dtparam=watchdog=on in the Pi’s config.txt is required to enable the watchdog, but it seems like this isn’t the case.
On both my Pi3 and Pi4, I do not have this and it seems like the watchdog is running.
$ sudo dmesg | grep -i watchdog
[ 2.494148] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 12.514374] systemd[1]: Using hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0, device /dev/watchdog0
[ 12.514405] systemd[1]: Watchdog running with a hardware timeout of 15s.
$ ls -l /dev/watchdog*
crw------- 1 root root 10, 130 Jul 5 12:03 /dev/watchdog
crw------- 1 root root 248, 0 Jul 5 12:03 /dev/watchdog0
I presume this is due to something that NixOS is doing? Perhaps with its device tree?
I also didn’t know about the wdctl utility.
$ wdctl
Device: /dev/watchdog0
Identity: Broadcom BCM2835 Watchdog time [version 0]
Timeout: 15 seconds
Timeleft: 14 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 0 0
SETTIMEOUT Set timeout (in seconds) 0 0
Apparently dtparam=watchdog=on is the default for RPi3 and onwards: https://raspberrypi.stackexchange.com/a/113193/7056
Well the main point I was making is that there is a daemon that needs to be running for that particular config file to do anything, and nixos doesn’t even have that daemon packaged.
watchdog as packaged in other distros was actually very easy to compile and seems to work fine: watchdog: init at 5.16 by n8henrie · Pull Request #351037 · NixOS/nixpkgs · GitHub