Trying to use github-runner

Help
1

Hello, everyone! As the title says, I’m trying.

Getting stuck at a weird timeout error, these are the systemd logs:

Jul 21 22:17:50 g14 systemd[1]: Starting GitHub Actions runner...
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24081]: Configuring GitHub Actions Runner
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: --------------------------------------------------------------------------------
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |        ____ _ _   _   _       _          _        _   _                      |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |                                                                              |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |                       Self-hosted runner registration                        |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: |                                                                              |
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: --------------------------------------------------------------------------------
Jul 21 22:17:50 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: # Authentication
Jul 21 22:17:54 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: √ Connected to GitHub
Jul 21 22:17:54 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: # Runner Registration
Jul 21 22:17:55 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: √ Runner successfully added
Jul 21 22:19:20 g14 systemd[1]: github-runner-runner4.service: start-pre operation timed out. Terminating.
Jul 21 22:19:20 g14 8kjs49yhyzs5kimiwvfpyg3cysiids1m-github-runner-runner4-configure.sh[24111]: Exiting...
Jul 21 22:19:20 g14 systemd[1]: github-runner-runner4.service: Control process exited, code=exited, status=1/FAILURE
Jul 21 22:19:20 g14 systemd[1]: github-runner-runner4.service: Failed with result 'timeout'.
Jul 21 22:19:20 g14 systemd[1]: Failed to start GitHub Actions runner.
Jul 21 22:19:20 g14 systemd[1]: github-runner-runner4.service: Consumed 3.095s CPU time, received 351.3K IP traffic, sent 26.6K IP traffic.

I’ve tried reading the source of the runner here. Just tracking that Runner successfully added message here.

My config is nothing fancy, something like:

services = {
  github-runners = {
    runner = {
      enable = true;
      name = "runner";
      tokenFile = "/home/user/token";
      url = "https://github.com/user/repo";
    };
  };
};

I generated the token in this page https://github.com/:user/:repo/settings/actions/runners/new and it seems to be working (No 404s)

I’m using the latest version (2.317.0) of the runner

2

i’m in the same boat… trying to start the github-runner(s) service on a fairly minimal nixos install. i get the same error: start-pre operation timed out. Terminating.

and just like the OP, the timeout is occuring exactly 90 seconds after systemd starts the service. it appears to me that the service is getting stuck (or is taking too long) during setup and systemd is saying “you’ve had your 90 seconds and now i’m killing you”. i’m about to experiment with using this setting in my runner to bump the timeout:

      serviceOverrides = {
        TimeoutStopSec=300;
      };

it appears that the OP’s access token is legit, just like mine. the service can connect to github. in my repo, the runner gets created on github, but is listed as “Offline”. i’ve successfully connected with both a fine-grained personal access token, and a “classic” one.

looking at other github-runner logs, after Runner successfully added, i would expect to see Runner connection is good but that never happens.

this ticket reports a similar issue, and there is a suggestion the disabling ipv6 would fix things.

in the github runner forums, troubleshooting is often done by running ./config.sh --check but i’m not sure how to do that on nix with the correct user environment.

speaking of which, without the user setting on the runner the docs say the service runs as a dynamically allocated user. i wonder if creating a user for the service is the right thing to do, but i have not investigated that yet.

3

progress! setting TimeoutStartSec (not TimeoutStopSec) did indeed bump the systemd timeout, and i now get an additional error. after √ Successfully replaced the runner, i now see The HTTP request timed out after 00:01:40.

next step: looking at the detailed service log in /var/lib/private/github-runner/<runner name>/_diag/Runner_<timestamp>.log, for which you need root access.

4

success!

turning off ipv6 fixes the problem on my machine.

explanation

why? it appears that one of github’s servers does not have ipv6 turned on. the nixos default is to enable both ipv4 and ipv6. and (i’m guessing here) the runner setup process either prefers or gets locked into using ipv6 for its requests. after creating the runner, it sends an HTTP request to the naughty github server over ipv6. that request times out after 100 seconds.

with default settings, systemd times out the entire setup process at 90 seconds (i.e. before the HTTP timeout happens and gets logged). so in this case, there’s no logging of the HTTP timeout and thus no info about what’s going wrong.

so i believe this issue is not nixos specific. any machine with ipv6 turned on could have the same trouble.

details

  • the ticket i mentioned is almost a year old as of this writing, so i’m guessing github isn’t going to address it anytime soon.
  • another ticket talks about ipv6 only systems.
  • in my logs, the host timing out on the ipv6 HTTP request was pipelinesghubeus26.actions.githubusercontent.com.
  • to turn off ipv6, add this line to your configuration.nix: networking.enableIPv6 = false;
2 Likes