NixOS on Hetzner cloud servers: IPv6

I’m running NixOS on a Hetzner cloud server, but unfortunately, IPv6 does not work (“Destination unreachable: Address unreachable”).
Has anyone got it running and can tell me how to configure IPv6 routing?

The configuration on Debian is:

auto eth0:0
iface eth0:0 inet6 static
    address 2a01:....:....:....::1/64
    gateway fe80::1
    post-up route add -A inet6 default gw fe80::1%eth0 || true
    pre-down route del -A inet6 default gw fe80::1%eth0 || true

This might be an occurrence of a common issue on Hetzner servers which is caused by their router equipment. If so, it probably works for a couple of minutes after boot: try checking if it works immediately after boot. If so, pester Hetzner’s support about it :wink:

Now, I’ve solved it; the reason it did not work was a wrong IPv6-address.
So, IPv6 on Hetzner cloud servers works with:

networking.interfaces.ens3.ipv6.addresses = [ { address = "2a01:4f8:....:....::1"; prefixLength = 64; } ];
networking.defaultGateway6 = { address = "fe80::1"; interface = "ens3"; };

(replace .... by the IPv6-address-parts of your server, and maybe ens3 by the name of your ethernet-device)

4 Likes

Almost 4 years later, I have exactly the same problem, also on Hetzner.

I came up with exactly the same first half of the solution, even ens3 matches my case. But as I’m no networking maven, the gateway part leaves me stumped. Is fe80::1 going to be the correct value universally, or should I be looking for something specific to my situation? If so, where?

I use the following snippet with networkd:

{ lib, config, ... }:
with lib;
let
  cfg = config.modules.hetzner.wan;
in
{
  options.modules.hetzner.wan = {
    enable = mkEnableOption "Enable Hetzner Cloud WAN interface configuration";

    macAddress = mkOption {
      type = types.str;
      description = "MAC Address of the WAN interface";
    };

    ipAddresses = mkOption {
      type = types.listOf types.str;
      description = "List of IP Addresses on the WAN interface";
    };
  };

  config = mkIf cfg.enable {
    systemd.network.networks."20-wan" = {
      matchConfig = {
        MACAddress = cfg.macAddress;
      };
      address = cfg.ipAddresses;
      routes = [
        { routeConfig.Gateway = "fe80::1"; }
        { routeConfig = { Destination = "172.31.1.1"; }; }
        { routeConfig = { Gateway = "172.31.1.1"; GatewayOnLink = true; }; }
        { routeConfig = { Destination = "172.16.0.0/12"; Type = "unreachable"; }; }
        { routeConfig = { Destination = "192.168.0.0/16"; Type = "unreachable"; }; }
        { routeConfig = { Destination = "10.0.0.0/8"; Type = "unreachable"; }; }
        { routeConfig = { Destination = "fc00::/7"; Type = "unreachable"; }; }
      ];
    };
  };
}

Used like so

{
  imports = [
    ./modules/hetzner/wan.nix
  ];

  networking.useNetworkd = true;

  modules.hetzner.wan = {
    enable = true;
    macAddress = "aa:bb:cc:dd:ee:ff"; # changeme
    ipAddresses = [
      "192.0.2.0/32"
      "2001:db8::1/64"
    ];
  };
}

Yes, fe80::1 on ens3 is your IPv6 gateway on all Hetzner Cloud VMs.

Perhaps not universally, but it is commonly used. Hetzner uses it too, as I found multiple references to it in their documentation. E.g. on their Static IP configuration page. fe80:: is the link-local address prefix that refers to the physical (local) network. fe80::1 is the first address, so is the perfect candidate for gateway.

I’ve managed to get this idea to work. Thanks!

A few observations:

  1. I needed to add networking.useDHCP = false;

  2. In the following section there are 3 things that need to match the specifics of your server (MAC address, IPv4, IPV6):

    When embedded in a more general configuration, these are going to get lost in the noise, so I replaced them with variables which I defined right at the top with a lot of fanfare, in the hope that I won’t forget to update them when I want to use this on another server N years down the line.

  3. At the end it blocked for a while and eventually gave the following warning.

    warning: the following units failed: systemd-networkd-wait-online.service
    
    × systemd-networkd-wait-online.service - Wait for Network to be Configured
         Loaded: loaded (/etc/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: disabled)
        Drop-In: /nix/store/f9nnwa7d1irl4c3g5ghd0x0xj1i2xjnd-system-units/systemd-networkd-wait-online.service.d
                 └─overrides.conf
      Active: failed (Result: exit-code) since Tue 2022-01-25 18:38:02 UTC; 56ms ago
        Docs: man:systemd-networkd-wait-online.service(8)
     Process: 2757 ExecStart=/nix/store/zc9r3iin18bccr3sjgwsgd6snp8g9g27-systemd-249.7/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
    Main PID: 2757 (code=exited, status=1/FAILURE)
          IP: 0B in, 0B out
         CPU: 11ms
    
    Jan 25 18:36:02 nixa systemd[1]: Starting Wait for Network to be Configured...
    Jan 25 18:38:02 nixa systemd-networkd-wait-online[2757]: Timeout occurred while waiting for network connectivity.
    Jan 25 18:38:02 nixa systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
    Jan 25 18:38:02 nixa systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
    Jan 25 18:38:02 nixa systemd[1]: Failed to start Wait for Network to be Configured.
    warning: error(s) occurred while switching to the new configuration
    

    This means nothing to me. But I can now access the machine through its IPv6 address aswell as the IPv4, which always worked.

Any suggestions on how to resolve this problem?

Check networkctl to find out which interface is blocking here.

While it’s blocking, I get the following output from networkctl status and networkctl list

[root@nixa:~]# networkctl status
●        State: routable                    
  Online state: online                      
       Address: 78.47.150.14 on ens3
                2a01:4f8:c17:fb26::1 on ens3
       Gateway: 172.31.1.1 on ens3
                fe80::1 on ens3

Feb 04 09:49:46 nixa systemd[1]: Starting Network Configuration...
Feb 04 09:49:46 nixa systemd-networkd[3676]: ens3: Link UP
Feb 04 09:49:46 nixa systemd-networkd[3676]: ens3: Gained carrier
Feb 04 09:49:46 nixa systemd-networkd[3676]: lo: Link UP
Feb 04 09:49:46 nixa systemd-networkd[3676]: lo: Gained carrier
Feb 04 09:49:46 nixa systemd-networkd[3676]: Enumeration completed
Feb 04 09:49:46 nixa systemd[1]: Started Network Configuration.
Feb 04 09:49:46 nixa systemd[1]: Starting Wait for Network to be Configured...

[root@nixa:~]# networkctl list
IDX LINK TYPE     OPERATIONAL SETUP      
  1 lo   loopback carrier     unmanaged
  2 ens3 ether    routable    configuring

2 links listed.

After the timeout, the only change is that the networktl status output gains the following 3 lines.

Feb 04 09:51:46 nixa systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 09:51:46 nixa systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Feb 04 09:51:46 nixa systemd[1]: Failed to start Wait for Network to be Configured.

There is a pretty yellow highlight on configuring in the ens3-SETUP table entry of the networkctl list output, before and after.

Should I be using some other networkctl command?

Does your interface have an IPv6 link local address?

If not, networkd is stuck waiting for that, which you can tell by enabling debugging:

systemd.services.systemd-networkd.serviceConfig.Environment = "SYSTEMD_LOG_LEVEL=debug";

Then check the logs:

journalctl -u systemd-networkd

And you’ll see:

systemd-networkd[648]: enp1s0: link_check_ready(): IPv6LL is not configured yet.

You can force the kernel to reconfigure IPv6LL by toggling addr_gen_mode.

sysctl -w net.ipv6.conf.enp1s0.addr_gen_mode=1
sysctl -w net.ipv6.conf.enp1s0.addr_gen_mode=0

Looks like a NixOS bug related to switching from scripted to networkd mode, because this doesn’t occur anymore after rebooting.

Confirmed.

That makes me cry a little inside, shedding tears for the time lost on this that I will never get back. Let’s see if I can learn something from the experience …

Indeed. I also get 3 occurrences of

ens3: link_check_ready(): static addresses are not configured.

Are these just noise, or should I worry about them? How should I know that the IPv6LL message is relevant while the others can be ignored (if that is indeed the case)?

What effect should I observe? Doing this (after having replaced enp1s0ens3) networkctl list still reports ens3 as configuring.

In any case, is there no declarative alternative? I guess not, if it’s a NixOS bug.

It should assign a IPv6 link local address (fe80::) to that interface (check with ip address) and as a result the interface should become configured. If not, consult networkd logging again.

Just to update (and perhaps for any Googlers) - I got to the bottom of it and discovered it’s not a NixOS but, but in fact a systemd issue.

networkd: fails to restore IPv6LL address generation #22424

The good news is that it’s now fixed and should eventually make its way to NixOS. Because the PR involved quite some reworking, I’m not sure it will be backported.

The easiest workaround is to perform the sysctl toggle described above - it only needs to be performed once upon initially switching from scripted mode to networkd.

3 Likes