Systemd: Wait for port

I made this little hack to wait for a port to be up before a service that doesn’t support sd-notify is considered up. What do you think? Is this something we could make available in the systemd NixOS module as e.g. waitForPort: number? (probably in some more robust way)

let
  notifyWhenPort = pkgs.writeScript "notifyWhenPort.sh" ''
    #!${pkgs.bash}/bin/bash
    # start a subshell in the background that tests the port
    # systemd will kill it on stop
    (while sleep 0.5; do
      if (: </dev/tcp/localhost/''${1}) 2>/dev/null; then
        ${pkgs.systemd}/bin/systemd-notify --ready
        exit 0
      fi
    done) &
  '';
in
{
  systemd.services."${serviceName}" = {
    serviceConfig = {
      ExecStart = "${serviceCommand}";

      # Wait for the port to be available
      Type = "notify";
      ExecStartPre = "${notifyWhenPort} ${toString port}";
      NotifyAccess = "all";
    };
    # rest of the config
  };
};

This way, it first starts a little bash shell that tries to access the port and calls systemd-notify when it’s up, then it starts the real service.

Another way to do it would be to make ExecStart a script that runs the process in the background, that way it can notify with the correct PID.

1 Like

I use the following script for this, since it’s using netcat instead of raw files it is maybe more robust, I used it in the past in production with success:


function wait_for_tcp {
  local elapsed='1'
  local timeout="${1}"
  local host="${2%:*}"
  local port="${2#*:}"

  while true; do
    if timeout 1s nc -z "${host}" "${port}"; then
      return 0
    elif test "${elapsed}" -gt "${timeout}"; then
      error "Timeout while waiting for ${host}:${port} to open" \
        && return 1
    else
      info "Waiting 1 second for ${host}:${port} to open, ${elapsed} seconds in total" \
        && sleep 1 \
        && elapsed="$(("${elapsed}" + 1))" \
        && continue
    fi
  done
}

certainly this kind of things are useful in some scenarios, but maybe instead of being a module, it would be more flexible if exposed as a package

1 Like

I’d say bash is equally good at opening a tcp port as nc :slight_smile: but the timeout is a good idea. Where do you run this? Inside the main executable script for systemd?

Indeed maybe a waitForTcpPort pkg would be useful.

Not really in systemd, conceptually I used it like this:

serviceA --port 8000 &
wait 8000
serviceB-that-requiresA

My targets were CI/CD pipelines and kubernetes stuff

@wmertens, the following is equivalent but much simpler:

{
  systemd.services.myservice = {
    serviceConfig.ExecStart = "...";
    postStart = ''
      <wait for open port>
    '';
  };
};

postStart is the same as serviceConfig.ExecStartPost, and the systemd manual says that the ExecStartPost command is run “after the commands specified in ExecStart= have been invoked successfully, as determined by Type=”

Yes, but the point is to tell systemd when the service is up, which is not possible when you run after the service is up.

The service only gets active after postStart finishes, so postStart can be used to wait for service readiness:

read -d '' container <<'EOF' || :
{
  containers.tmp = {
    config = {
      systemd.services.demo = {
        wantedBy = [ "multi-user.target" ];
        script = ''
          echo "main script started"
          sleep infinity
        '';
        postStart = ''
          sleep 0.2
          echo "postStart finished"
        '';
      };
    };
  };
}
EOF
extra-container shell -E "$container" --run c journalctl --output=short-precise -u demo

Output:

Mar 12 19:03:52.371080 tmp systemd[1]: Starting demo.service...
Mar 12 19:03:52.372662 tmp demo-start[206]: main script started
Mar 12 19:03:52.575758 tmp demo-post-start[207]: postStart finished
Mar 12 19:03:52.576033 tmp systemd[1]: Started demo.service.
1 Like

OT: extra-container does not work for me in unstable, looks like due to logrotate: move wtmp/btmp rules to systemd · NixOS/nixpkgs@9917af7 · GitHub.

But TIL, looks very useful!

That is indeed very elegant, not even systemd-notify needed! Thanks!

1 Like