Networking not working after update (Hetzner)

Help
1

Today I updated my NixOS server hosted on Hetzner and could no longer connect via SSH or reach the web services. After chrooting to it with nixos-enter and examining the journal, I noted that the system booted just fine, but there was apparently some issue with networking. I use soju IRC bouncer and noticed in the journal that it failed to connect to the IRC network.

The original networking config was unchanged from the time it was automatically generated by nixos-infect. I decided to try a systemd-networkd based config adapted from this example (I import the separate config file):

{ config, lib, pkgs, ... }: {
  # Use systemd networkd
  systemd.network.enable = true;
  networking.useNetworkd = true;

  # Static network configuration (Hetzner)
  networking.wireless.enable = false;
  networking.useDHCP = false;
  networking.usePredictableInterfaceNames = false;
  networking.interfaces.eth0 = {
    ipv4.addresses = [
      {
        address = "...";
        prefixLength = 32;
      }
    ];
    ipv6.addresses = [
      {
        address = "...";
        prefixLength = 64;
      }
    ];
  };
  networking.defaultGateway = {
    address = "172.31.1.1";
    interface = "eth0";
  };
  networking.defaultGateway6 = {
    address = "fe80::1";
    interface = "eth0";
  };
  networking.nameservers = [
    "185.12.64.1"
    "185.12.64.2"
    "2a01:4ff:ff00::add:1"
    "2a01:4ff:ff00::add:2"
  ];
}

In my configuration.nix I have additionally (unchanged):

  networking.hostName = "ubuntu-2gb-hel1-1";
  networking.domain = "";
  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 80 443 3001 6697 ];
    allowedUDPPorts = [ 3001 ];
  };

system.stateVersion is 23.11.

After rebuilding the config and booting the server, I still could not connect to it or reach my websites, but according to the journal the connection to the IRC network worked. Indeed, if I sit in an IRC channel as another user while booting the server, I can see my user joining and the connection has remained steady for 40 mins so far. Here is the journal with the systemd-networkd config.

I don’t notice any other weirdness besides this:

Jun 07 12:08:31 ubuntu-2gb-hel1-1 systemd-networkd-wait-online[894]: Timeout occurred while waiting for network connectivity.
Jun 07 12:08:31 ubuntu-2gb-hel1-1 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jun 07 12:08:31 ubuntu-2gb-hel1-1 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jun 07 12:08:31 ubuntu-2gb-hel1-1 systemd[1]: Failed to start Wait for Network to be Online.
Jun 07 12:08:31 ubuntu-2gb-hel1-1 systemd[1]: Reached target Network is Online.

Any ideas on where to go from here? I suppose I could build a config where I can log in with a password and use Hetzner’s console to connect to the running system, but what should I inspect?

2

I gave up, deleted the server and am trying from scratch. nixos-infect with cloud-init this time yielded a broken system, so used nixos-anywhere although now I’m in a strange half-working state. Will open a new thread.

3

That’s a private prefix, are you sure that gateway is correct?

It’d be nice to know what networkctl says.

4

FWIW I just bumped my hetzner cloud instance (3fbe2b9a53598b72d185b2b67c64159a066b8083 → 0265069a40f500e713cbf64f29185b3cfa6c9ba9) and it worked fine… But I have a very simple config which very closely resembles SrvOS’s…

5

In the end I got the reinstall to work with the “Traditional ISO installation” method from the wiki article. The static IPv4 configuration does have this: “The trick here is, that the gateway needs to be configured with the onlink flag, because it is not in the same subnet as your public IP address, but still very much on that same link”. Current working network config:

{ config, lib, pkgs, ... }: {
  systemd.network.enable = true;
  networking.useNetworkd = true;

  systemd.network.networks."30-wan" = {
    matchConfig.Name = "enp1s0";
    networkConfig.DHCP = "no";
    address = [
      ".../32"
      ".../64"
    ];
    routes = [
      { Gateway = "172.31.1.1"; GatewayOnLink = true; }
      { Gateway = "fe80::1"; }
    ];
  };
  networking.hostName = "ubuntu-4gb-hel1-2";
  networking.domain = "";
  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 22 80 443 3001 6697 ];
    allowedUDPPorts = [ 3001 ];
  };
}

I think the docs about nixos-anywhere should be amended with a mention that you can’t update a system set up with it the usual way. I found there were many forum topics asking about how to rebuild such a system.

6

Yeah, that’s a good idea. I’d suggest sending them a PR, you have fresh understanding.

7

The problem is I still don’t know how such a system is supposed to be rebuilt and if it can be converted into a “normal” one (which kind of seems what is being done at the very end of this article), so I’m not the best person to edit the wiki or submit a PR.