Systemd hangs when running script that calls rclone

I have a script that uses rclone to download a file. Unfortunately executing the script hangs when the rclone command is executed:

  postgresPostStartScript = ''
    #!/bin/bash

    export PATH=${pkgs.rclone}/bin:${pkgs.coreutils}/bin:${pkgs.findutils}/bin:${pkgs.gnugrep}/bin:${pkgs.gnused}/bin:${pkgs.gnutar}/bin:${pkgs.gzip}/bin:${pkgs.util-linux}/bin:${config.services.postgresql.package}/bin:$PATH

   ... 

    # script hangs here when executed via systemd
    echo "Downloading dump file..."
    ${pkgs.rclone}/bin/rclone copyto \
      -vv \
      --config "/etc/rclone.conf" \
      hetzner-s3:my-bucket/db.dump \
      "$DUMP_FILE"
    echo "downloaded dump file"

    fi

   ...
  '';
  postgresql-poststart-script = pkgs.writeScriptBin "postgresql-poststart-script" postgresPostStartScript;

The systemd service definition is as follows:

  systemd.services.postgresql-poststart = {
    description = "PostgreSQL Post-Start Actions";
    after = [ "postgresql.service" ];
    wants = [ "postgresql.service" ];
    serviceConfig = {
      User = "bn";
      Group = "bn";

      EnvironmentFile = [ "${config.age.secrets.env-secret.path}" ];
      Type = "oneshot";
      ExecStart = "${pkgs.bash}/bin/bash ${postgresql-poststart-script}/bin/postgresql-poststart-script";
    };
    #wantedBy = [ "multi-user.target" ];
  };

When I run sudo -u bn <environment variables> bash /nix/store/.../bin/postgresql-poststart-script the script runs successfully. I’ve checked that the EnvironmentFile is correctly decrypted by agenix, so that’s not the reason.

All I get when starting the systemd service is the Download dump file... and then it hangs. Does anybody have an idea of what could go wrong here? Also is it possible to flush the actual output of the rclone command to journal somehow? I’ve tried prepending ${pkgs.coreutils}/bin/stdbuf -oL -eL to the rclone command, but with no success.

This seems rclone related. I still haven’t figured out why this doesn’t work, but using the following stops the hang:

      ExecStart = ''
        ${pkgs.util-linux}/bin/script -q -c "${pkgs.rclone}/bin/rclone copyto -vv --config /etc/rclone.conf hetzner-s3:my-bucker/my-file /var/lib/bn/file.dump" /dev/null
      '';

I’m running into the same issue. This is rclone fails to start as systemd service · Issue #8621 · rclone/rclone · GitHub, which is fixed by log: fix deadlock when using systemd logging - fixes #8621 · rclone/rclone@3cae373 · GitHub, which was backported to v1.70.2 in log: fix deadlock when using systemd logging - fixes #8621 · rclone/rclone@fa3b444 · GitHub.

nixpkgs received the fixed version in rclone: 1.70.1 -> 1.70.2 by onvik · Pull Request #420551 · NixOS/nixpkgs · GitHub.

1 Like