`ExecStartPre` and `ExecStartPost` repeated multiple times in overridden serviceConfig

[TLATER]

I’ve finally been trying to get proper atomic backups for some services that don’t have a good way to limit file changes while hot. After sifting through some apparent gnarly systemd limitations I arrived at this:

  systemd.services.restic-backups-gitea.serviceConfig = {
    ExecStartPre = [
      "+${pkgs.local.backups.shutdownService "gitea.service"}"
      (pkgs.writeShellScript "dump-database" ''
        ${config.services.postgresql.package}/bin/pg_dump \
          ${config.services.gitea.database.name} > /var/lib/gitea/gitea-db.sql
      '')
    ];

    ExecStopPost = [
      "${pkgs.coreutils}/bin/rm /var/lib/gitea/gitea-db.sql"
      "+${pkgs.local.backups.restartService "gitea.service"}"
    ];
  };

Yes, this is ugly. That is besides the point, but suggestions are welcome.

No, gitea dump is apparently not atomic despite being partially recommended by their documentation, and Conflicts= doesn’t work with Wants= + Before=. Even if it did it would restart services that were manually stopped… And redirecting to a file for a specific Exec*= seems to require a shell.

Using the module’s built-in preStart replacement to at least get more sane scripting in turn doesn’t give sufficient permissions to start/stop systemd units, and it probably shouldn’t either.

If this was php I could probably flip off the nginx virtualHost and mask any cleanup services gitea would run, but it’s not, and that wouldn’t solve the problem generally (I run some game servers with similar problems).

Maybe this is worth a post to the systemd ML, I can’t believe nobody else has ever been bothered by this.

Weirdly, this results in the following actual ExecStart/Pre/Post declarations in the resulting systemd unit file:

ExecStart=/nix/store/vcwlr469vg7iyl36vcaqyasivn2lqqz6-restic-0.15.2/bin/restic backup  /var/lib/gitea/gitea-db.sql /var/lib/gitea/repositories/ /var/lib/gitea/data/ /var/lib/gitea/custom/
ExecStartPost=+/nix/store/fa9dxqqfb1lsdn34jnijij8rhr32yahp-backup-restart-service
ExecStartPost=+/nix/store/sxs0pjgbi59y5nshkqax9rr39dmbl2x1-backup-restart-service
ExecStartPre=/nix/store/0r5z4wyhf8a7dhd4948waf4jmbaripwv-unit-script-restic-backups-gitea-pre-start/bin/restic-backups-gitea-pre-start
ExecStartPre=+/nix/store/p5wlv9ci3wjnpw2vv1gwn5cafy1shbgg-backup-shutdown-service
ExecStartPre=+/nix/store/fac8bndidbyvpxcxrrjf3c009hxvvhl6-backup-shutdown-service
ExecStartPre=/nix/store/irv5adb08q34lrr99zh2v4wqw66gcfc3-dump-database
ExecStartPre=+/nix/store/cx2mariywa176zkmj8ggazapzdavhndy-backup-shutdown-service
ExecStopPost=/nix/store/j4fwy5gi1rdlrlbk2c0vnbs7fmlm60a7-coreutils-9.1/bin/rm /var/lib/gitea/gitea-db.sql
ExecStopPost=+/nix/store/izglrinzwj941jnv60ccywgfg11cwyia-backup-restart-service

Does anyone know why on earth they end up being duplicated (and into multiple derivations too??)? I can’t cobble together a meaningfully minimalier reproduction, unfortunately, it’s not just mixing preStart and serviceConfig.

[TLATER]

Ah, I found it… Caused by doing something similar to this:

Nginx default locations across all virtualHosts

You can also override options.services.nginx.virtualHosts to inject default values into every virtualhost. This is what I do to set listen addresses globally: { options.services.nginx.virtualHosts = lib.mkOption { type = lib.types.attrsOf (lib.types.submodule { config.listen = lib.mkDefault [ { addr = "0.0.0.0"; port = 443; ssl = true; } { addr = "[2a01:4f9:2b:1605::1]"; port = 443; ssl = true; } ]; }); }; }

I guess somehow that creates multiple instances of the option, and the resulting stack ends up being merged into a big list. 'suppose I’ll have to dig into exactly how that suggestion works. I can probably use a lib.mkForce or something to prevent the lists from merging for now.