Notification on systemd service failures

pSub · December 1, 2019, 7:14pm

I deploy my server with NixOps and I want to be notified about systemd service failures. Therefore I tried to extend every systemd service with onFailure. Currently I have something along the lines of

{ config, lib, pkgs, ... }:

{
  config = lib.setAttrByPath [ "systemd" "services" ]
    (lib.genAttrs (lib.attrNames config.systemd.services)
      (serviceName: lib.setAttrByPath [ "serviceConfig" "onFailure" ]  [ "email@%n.service" ]));
}

However, that will lead into an infinite recursion. Any ideas?

arianvp · December 5, 2019, 5:23pm

Just another idea. Systemd publishes all event changes of systemd units over dbus. You could listen to these events and trigger something any time a service goes to status failed.

Nixpkgs ships with the pystemd python package that easily allows you to do this https://github.com/facebookincubator/pystemd/blob/a51e19e1abe65498f2d74aa540c7016716c3e846/examples/monitor_all_units_from_signal.py

Or you could do it with busctl and some bash:

sudo busctl  monitor org.freedesktop.systemd1 --json=short | jq 'select(.member=="PropertiesChanged")'

You then just need one service that monitors all of them.

However, l’m also curious how we can solve your nixos config to not infinitely recurse. Ill have a better look later

ElvishJerricco · December 6, 2019, 6:37am

With systemd v244, we will be able to do it this way:

Unit files now support top level dropin directories of the form
<unit_type>.d/ (e.g. service.d/) that may be used to add configuration
that affects all corresponding unit files.

Zurdo · December 14, 2019, 2:54am

Journalwatch seems like a possible workaround.

makefu · December 17, 2019, 10:29am

In the stockholm there is a module called krebs.on-failure which essentially links a separate on-failure.plans.<service-name> to the service:
The essential piece of configuration can be found at:

However it only attaches to explicitly marked services:
krebs.on-failure.plans.snapraid-sync.name = "snapraid-sync"; [source]

The idea is that most of the time you know which services are important and where you definitly want to receive a mail once something dies.

Cheers

pSub · December 28, 2019, 7:45pm

Thank you for your suggestions. I learned about some cool stuff. Much appreciated
For the moment I just use my solution and specify explicitly the services for that I definitely want to receive notifications.

danbst · December 30, 2019, 10:33pm

# default.nix
let
  nixos =
    import <nixpkgs/nixos> {
        configuration = { lib, ...}: {

            options.systemd.services = lib.mkOption {
                type = with lib.types; attrsOf (submodule {
                    config = {
                        serviceConfig.onFailure = "email@%n.service";
                    };
                });
            };

            config = {
                boot.isContainer = true; # this is only as an example

                services.nginx.enable = true;
            };

        };
    };
in nixos.config

$ nix eval -f. systemd.services.nginx.serviceConfig.onFailure
"email@%n.service"

ElvishJerricco · December 31, 2019, 5:47pm

Note this probably won’t affect service units shipped builtin with systemd, nor service units added via the systemd.packages option.

Tmplt · August 10, 2024, 1:31pm

Resurrecting this: is there a canonical implementation of this today? I’d like to send an email to myself on user service failures also.

aanderse · August 10, 2024, 2:24pm

monitoring software is the answer, of course, but to directly answer your question is to use the systemd top level overrides feature like so:

{ config, lib, pkgs, ... }: {
  systemd.packages = [
    (pkgs.runCommandNoCC "on-failure.conf" {
      preferLocalBuild = true;
      allowSubstitutes = false;
    } ''
      mkdir -p $out/etc/systemd/system/service.d/
      echo "[Service]\nOnFailure=email@%n.service" > $out/etc/systemd/system/service.d/on-failure.conf
    '')
  ];
}

untested, so please ping back if this is incorrect and we can edit

reference on discourse: How to use toplevel-overrides for systemd - #4 by hmenke

Tmplt · August 10, 2024, 3:36pm

Thanks; added to my agenda.

monitoring software is the answer

This escaped me. I temporarily tried netdata out some time ago but stopped for several reasons. Any general recommendations to consider?

aanderse · August 10, 2024, 8:57pm

i think netdata is somewhat popular among people running NixOS for hobby usage…

payas · August 11, 2024, 7:44am

I personally use healthchecks.io (self-hosted version, but their free tier is pretty damn generous). It has integration for nearly everything, including email, slack, ntfy (which has mobile app), whathaveyou.

Works pretty well to keep me updated on automated NixOS updates and backup failures.

_Andrew · August 11, 2024, 11:30pm