Can't get `prometheus.exporters.restic` to work with the security wrapper

I have setup restic to use the security wrapper, which works fine with the module (thanks to pointers from Using restic service with the security wrapper):

services.restic.backups = {
  saveoneself = {
    # ...
    package = pkgs.writeShellScriptBin "restic-security-wrapper" ''
      #!${pkgs.runtimeShell}
      exec /run/wrappers/bin/restic "$@"
    '';
};

I would like to have prometheus expose metrics about it, so I use:

{
  config,
  lib,
  pkgs,
}:
let
  prometheusResticExporterOverlay = final: prev: {
    prometheus-restic-exporter = prev.prometheus-restic-exporter.overrideAttrs (oldAttrs: {
      installPhase =
        oldAttrs.installPhase
        + ''
          substituteInPlace $out/bin/restic-exporter.py --replace-fail \"${
            lib.makeBinPath [ pkgs.restic ]
          }/restic\" \"/run/wrappers/bin/restic\"
        '';
    });
  };
in
{
  nixpkgs.overlays = [ prometheusResticExporterOverlay ];

  services.restic = with config.services.restic.backups.saveoneself; {
    inherit rcloneConfigFile repository passwordFile;
    enable = true;
    user = "restic";
    # ...
  };
}

however, the service fails with:

Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]: 2025-03-05 12:44:45 INFO     Starting Restic Prometheus Exporter
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]: 2025-03-05 12:44:45 INFO     It could take a while if the repository is remote
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]: 2025-03-05 12:44:45 ERROR    Unable to collect metrics from Restic. Exception: Error executing restic snapshot command: failed to inherit capab>
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]: Traceback (most recent call last):
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:   File "/nix/store/6mdrs9gch88rq6dvnd0zgwpsyz4vz4zv-prometheus-restic-exporter-1.6.0/bin/restic-exporter.py", line 413, in <module>
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:     REGISTRY.register(collector)
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:   File "/nix/store/b5293rqai1a8w3cwwjgrp9fwpz0kzz69-python3-3.12.8-env/lib/python3.12/site-packages/prometheus_client/registry.py", line 40, in>
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:     names = self._get_names(collector)
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:   File "/nix/store/b5293rqai1a8w3cwwjgrp9fwpz0kzz69-python3-3.12.8-env/lib/python3.12/site-packages/prometheus_client/registry.py", line 80, in>
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:     for metric in desc_func():
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:                   ^^^^^^^^^^^
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:   File "/nix/store/6mdrs9gch88rq6dvnd0zgwpsyz4vz4zv-prometheus-restic-exporter-1.6.0/bin/restic-exporter.py", line 92, in collect
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:     check_success.add_metric([], self.metrics["check_success"])
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]:                                  ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Mar 05 12:44:45 myhost prometheus-restic-exporter-start[679795]: KeyError: 'check_success'
Mar 05 12:44:45 myhost systemd[1]: prometheus-restic-exporter.service: Main process exited, code=exited, status=1/FAILURE

looking at the unit configuration, I even tried:

{
  systemd.services."prometheus-restic-exporter".serviceConfig.NoNewPrivileges = lib.mkForce false;
}

to no avail :disappointed:

I don’t quite get what is going wrong…
There seems to be some conflict between the exporter service setup, and the way the wrapper behaves. I shallowly looked at the security.wrapper implementation, but it went way above my head…
Any ideas what is going wrong? And/or how one could mitigate this issue (without running everything as root)?

1 Like