Wireguard DNS over systemd-resolved

I have multiple Wireguard VPNs for various environments which all use their own DNS server to resolve hostnames. Below is an excerpt of my Wireguard configurations:

networking.wg-quick.interfaces = {
  vpn1 = {
    autostart = false;
    address = [ /* VPN 1 addresses */ ];
    dns = [ /* DNS 1 IPs */ ];
    privateKeyFile = config.sops.secrets.vpn1-pkey.path;
    mtu = 1280;
    peers = [{
      publicKey = "...";
      presharedKeyFile = config.sops.secrets.vpn1-psharedkey.path;
      allowedIPs = [ "10.36.0.0/15" ];
      endpoint = "w.x.y.z:1234";
      persistentKeepalive = 25;
    }];
  };
  vpn2 = {
    autostart = false;
    address = [ /* VPN 2 addresses */ ];
    dns = [ /* DNS 2 IPs */ ];
    privateKeyFile = config.sops.secrets.vpn2-pkey.path;
    mtu = 1280;
    peers = [{
      publicKey = "...";
      presharedKeyFile = config.sops.secrets.vpn2-psharedkey.path;
      allowedIPs = [ "10.4.0.0/15" ];
      endpoint = "w'.x'.y'.z':1234";
      persistentKeepalive = 25;
    }];
  };
};

This setup works fine, however I’m not able to start both VPNs at the same time because the DNS resolution fails for the first started VPN while it works for the second started VPN.

I found out that with the default resolvconf package, Wireguard was replacing the whole content of /etc/resolv.conf with the DNS settings of the last started VPN. Hence the DNS resolution issue.

I read online that systemd-resolved was fixing this issue, so I set services.resolved.enable = true. However, after doing that, when I start the VPNs there isn’t any DNS set for the interfaces, as seen with resolvectl status vpn1 vpn2:

Link 13 (vpn1)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 12 (vpn2)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

I found out that when using systemd-resolved, the DNS settings of Wireguard configuration where not taken into account and one should use PostUP to run resolvectl dns %i <DNS IPs> to set the DNS servers for the Wireguard interfaces. Therefore, I added this to my config:

networking.wg-quick.interfaces = {
  vpn1 = rec {
    # Omitting previous unchanged configuration
    postUp = "${pkgs.systemd}/bin/resolvectl dns vpn1 ${lib.concatStringsSep " " dns}";
  };
  vpn2 = rec {
    # Omitting previous unchanged configuration
    postUp = "${pkgs.systemd}/bin/resolvectl dns vpn2 ${lib.concatStringsSep " " dns}";
  };
};

I can see that the postUp script is being run on wg-quick systemd services start:

juin 20 11:52:06 g-xps wg-quick-vpn1-start[5080]: [#] /nix/store/s29qf5z3iy7s8gmsng7xjw8v2h1yb6x0-postUp.sh/bin/postUp.sh

And the content of the postUp script does include the resolvectl command:

#!/nix/store/h3bhzvz9ipglcybbcvkxvm4vg9lwvqg4-bash-5.2p26/bin/bash
wg set vpn1 private-key <(cat /run/secrets/vpn1-pkey)
wg set vpn1 peer [REDACTED] preshared-key <(cat /run/secrets/vpn1-psharedkey)
/nix/store/9cxd17xnmw0bi8n4nf722ysqj2bjlh8s-systemd-255.4/bin/resolvectl dns nstoolingsnc [REDACTED IP ADDRESSES]

However, when the systemd service is started, the DNS servers are not set as seen with resolvectl status vpn1:

Link 13 (vpn1)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

But, if I manually run the postUp script after the service is started, it works:

❯ sudo /nix/store/0f712568ksyaafb1h640xzkxzg9yavr0-postUp.sh/bin/postUp.sh
❯ resolvectl status vpn1
Link 13 (vpn1)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: [REDACTED]
       DNS Servers: [REDACTED]

Any idea why this is not working?

I was trying to tinker with the config to see if was able to make this work and I found out that when using systemctl restart instead of systemctl start, the DNS were set!

So, this doesn’t seem to be an issue with Wireguard itself but rather with the way its managed by systemctl.

It might not be a good idea to reference the direct store path if you want to be able to easily update. Try looking in /run/current-system/sw/bin (off the top of my head, could be that one of the folders is swapped)

To which part are you refering? I specifically used ${pkgs.systemd}/bin/resolvectl so this would always use the correct store path.

If you’re talking about the last part where I manually executed the postUp script, that was just to ensure that the script was indeed setting the DNS settings of the vpn, I’m not planning on using it each time I start the vpn service (that would indeed render me crazy).

I’d suggest using networkd to configure this.
You can create wireguard netdevs to set up the interfaces and then you define a network for each of them. As part of the network config, you will configure DNS.

So I should create the wireguard networks with networking.wg-quick.interfaces without DNS settings and in addition to that create networking.interfaces (with the same name, I guess?) which define the DNS config for said VPN?

Sorry I’m not sure if I understood you correctly :sweat_smile:

I don’t use wg-quick at all. You can do everything with systemd-networkd. There’s two things to configure:

  1. a netdev of type wireguard that will create your wireguard interface. This has all the wireguard specific stuff like your private key and your peers.
  2. a network config that will configure the network settings for the interface that you configured above, which in the case of a wireguard network is usually pretty simple. Here you can also configure the per-network DNS settings that will be used to configure resolved.

I do something similar in my own config but it’s a bit complicated because I generate multiple configs, and I tunnel wireguard through wstunnel. But if you’re comfortable reading nix, you can get inspiration here: ~r-vdp/nixos-config (de5c7097cac09a3fb4e66203d039e684f933f53b): modules/wg-vpn.nix - sourcehut git

Don’t bother with the policy routing stuff unless you want to route all traffic over wireguard. If you only route specific address ranges, you don’t need to do anything special and networkd will create the routing table entries for you automatically.

1 Like

Thanks for explaining this further! I’ll definitely take a look at it tomorrow and let you know :slightly_smiling_face:

Sorry to bother you with this but I tried to implement the VPNs with systemd.network by using your config, NixOS options search, the wiki and the systemd documentation but I can’t make it work and I don’t know why…

Here is what I did:

  systemd.network = {
    enable = true;

    netdevs = {
      "50-vpn1" = {
        netdevConfig = {
          Kind = "wireguard";
          Name = "50-vpn1";
          Description = "Wireguard device config for vpn1";
          MTUBytes = "1280";
        };

        wireguardConfig.PrivateKeyFile = config.sops.secrets.vpn1-pkey.path;

        wireguardPeers = [
          {
            wireguardPeerConfig = {
              PublicKey = "...";
              PresharedKeyFile = config.sops.secrets.vpn1-psharedkey.path;
              AllowedIPs = [ "10.36.0.0/15" ];
              Endpoint = "w.x.y.z:1234";
              PersistentKeepalive = 25;
            };
          }
        ];
      };

      "50-vpn2" = {
        netdevConfig = {
          Kind = "wireguard";
          Name = "50-vpn2";
          Description = "Wireguard device config for vpn2";
          MTUBytes = "1280";
        };

        wireguardConfig.PrivateKeyFile = config.sops.secrets.vpn2-pkey.path;

        wireguardPeers = [
          {
            wireguardPeerConfig = {
              PublicKey = "...";
              PresharedKeyFile = config.sops.secrets.vpn2-psharedkey.path;
              AllowedIPs = [ "10.4.0.0/15" ];
              Endpoint = "w'.x'.y'.z':1234";
              PersistentKeepalive = 25;
            };
          }
        ];
      };
    };

    networks = {
      "50-vpn1" = {
        address = [ /* VPN 1 addresses */ ];
        dns = [ /* DNS 1 IPs */ ];
        DHCP = "no";
        linkConfig.ActivationPolicy = "manual";
        name = "50-vpn1";

        networkConfig = {
          Description = "Wireguard network config for vpn1";
          DNSOverTLS = "opportunistic";
        };
      };

      "50-vpn2" = {
        address = [ /* VPN 2 addresses */ ];
        dns = [ /* DNS 2 IPs */ ];
        DHCP = "no";
        linkConfig.ActivationPolicy = "manual";
        name = "50-vpn2";

        networkConfig = {
          Description = "Wireguard network config for vpn2";
          DNSOverTLS = "opportunistic";
        };
      };
    };
  };

I had to also disable networking.useDHCP to avoid having the following warning:

trace: warning: The combination of `systemd.network.enable = true`, `networking.useDHCP = true` and `networking.useNetworkd = false` can cause both networkd and dhcpcd to manage the same interf
aces. This can lead to loss of networking. It is recommended you choose only one of networkd (by also enabling `networking.useNetworkd`) or scripting (by disabling `systemd.network.enable`)

But then I lost network connectivity… So I ended up re-enabling networking.useDHCP and also enabled networking.useNetworkd. The warning went away, I got the network connectivity back, but I can’t see the VPN networks with networkctl list and I can’t start them with networkctl up vpn1 / networkctl up vpn2.

I checked in /etc/systemd/network, the .netdev and .network files are created but somehow they don’t get picked up by networkd?

❯ lsa /etc/systemd/network/
total 12K
drwxr-xr-x 2 root root 4,0K 23 juin  15:15 .
drwxr-xr-x 3 root root 4,0K 23 juin  15:15 ..
lrwxrwxrwx 1 root root   47 23 juin  15:15 40-vboxnet0.network -> /etc/static/systemd/network/40-vboxnet0.network
lrwxrwxrwx 1 root root   49 23 juin  15:15 50-vpn1.netdev -> /etc/static/systemd/network/50-vpn1.netdev
lrwxrwxrwx 1 root root   50 23 juin  15:15 50-vpn1.network -> /etc/static/systemd/network/50-vpn1.network
lrwxrwxrwx 1 root root   51 23 juin  15:15 50-vpn2.netdev -> /etc/static/systemd/network/50-vpn2.netdev
lrwxrwxrwx 1 root root   52 23 juin  15:15 50-vpn2.network -> /etc/static/systemd/network/50-vpn2.network
lrwxrwxrwx 1 root root   60 23 juin  15:15 99-ethernet-default-dhcp.network -> /etc/static/systemd/network/99-ethernet-default-dhcp.network
lrwxrwxrwx 1 root root   59 23 juin  15:15 99-wireless-client-dhcp.network -> /etc/static/systemd/network/99-wireless-client-dhcp.network

This is the output of networkctl list, if that’s helpful:

❯ networkctl list
IDX LINK            TYPE     OPERATIONAL SETUP      
  1 lo              loopback carrier     unmanaged
  3 wlp59s0         wlan     routable    configured 
  4 vboxnet0        ether    no-carrier  configuring
  5 br-694a23a7f57a bridge   no-carrier  unmanaged
  6 br-73eab72191a2 bridge   routable    unmanaged
  7 docker0         bridge   routable    unmanaged
  9 vetha8e1398     ether    enslaved    unmanaged
 11 veth533d162     ether    enslaved    unmanaged
 12 enp58s0u1u2     ether    no-carrier  configuring

9 links listed.

Note that vboxnet0 and enp58s0u1u2 are « stuck » in configuring state.

I don’t have any knowledge of networkd, neither did I hear of its existence before you mentioned it. I feel that it may be overkill for the simple use case I have. I don’t quite understand the behavior of wg-quick with my current setup (ie. having to start VPN service then restart it to get the postUp script to actually set the DNS).

Any help would be greatly appreciated!

You are missing match configs in your networks, which is what links the network to the interface that it’s supposed to configure, so you should definitely add those.

If that isn’t enough to get things to work, then turn on networkd debug logging and post the logs here (journalctl -u systemd-networkd), to turn on the debug logging, you set an env var on the networkd systemd service:

systemd.services.systemd-networkd.environment.SYSTEMD_LOG_LEVEL = "debug";

The matches are there, I don’t use the systemd.network.networks.<name>.matchConfig option. I instead use the systemd.network.networks.<name>.name option:

The name of the network interface to match against.

❯ cat /etc/static/systemd/network/50-vpn1.netdev -p
[NetDev]
Description=Wireguard device config for vpn1
Kind=wireguard
MTUBytes=1280
Name=50-vpn1

[WireGuard]
PrivateKeyFile=/run/secrets/vpn1-pkey

[WireGuardPeer]
AllowedIPs=10.36.0.0/15
Endpoint=w.x.y.z:1234
PersistentKeepalive=25
PresharedKeyFile=/run/secrets/vpn1-psharedkey
PublicKey=...

❯ cat /etc/static/systemd/network/50-vpn1.network
[Match]
Name=50-vpn1

[Link]
ActivationPolicy=manual

[Network]
DHCP=no
DNSOverTLS=opportunistic
Description=Wireguard network config for vpn1
Address=...
Address=...
DNS=...
DNS=...

I’ll enable networkd debug logging and keep you posted.

Thanks for your help!

Ah ok, yeah, I didn’t know about that option. Then the logs would probably be the best way to understand what’s happening.

Oh, now I understand why it’s not working:

juin 22 21:16:37 g-xps systemd-networkd[7872]: 50-vpn1: Failed to read private key from /run/secrets/vpn1-pkey. Ignoring network device.
juin 22 21:16:37 g-xps systemd-networkd[7872]: 50-vpn2: Failed to read private key from /run/secrets/vpn2-pkey. Ignoring network device.

I granted access to the secrets to the systemd-network group (thanks to your config):

sops.secrets.<secret>.group = "systemd-network"

And now it works like a charm!

A note if anyone stumble on this same issue, the network device name cannot exceed a certain length (nothing is noted in the documentation, but I believe its 15 characters).

The final configuration is:

  sops.secrets = {
    vpn1-pkey = {
      sopsFile = ./path/to/secret.yaml;
      mode = "0440";
      group = "systemd-network";
    };
    vpn1-psharedkey = {
      sopsFile = ./path/to/secret.yaml;
      mode = "0440";
      group = "systemd-network";
    };

    vpn2-pkey = {
      sopsFile = ./path/to/secret.yaml;
      mode = "0440";
      group = "systemd-network";
    };
    vpn2-psharedkey = {
      sopsFile = ./path/to/secret.yaml;
      mode = "0440";
      group = "systemd-network";
    };
  };

  # Only if you need to use networking.* options (like useDHCP)
  networking.useNetworkd = true;

  systemd.network = {
    enable = true;

    netdevs = {
      "50-vpn1" = {
        netdevConfig = {
          Kind = "wireguard";
          Name = "50-vpn1";
          Description = "Wireguard device config for vpn1";
          MTUBytes = "1280";
        };

        wireguardConfig.PrivateKeyFile = config.sops.secrets.vpn1-pkey.path;

        wireguardPeers = [
          {
            wireguardPeerConfig = {
              PublicKey = "...";
              PresharedKeyFile = config.sops.secrets.vpn1-psharedkey.path;
              AllowedIPs = [ "10.36.0.0/15" ];
              Endpoint = "w.x.y.z:1234";
              PersistentKeepalive = 25;
            };
          }
        ];
      };

      "50-vpn2" = {
        netdevConfig = {
          Kind = "wireguard";
          Name = "50-vpn2";
          Description = "Wireguard device config for vpn2";
          MTUBytes = "1280";
        };

        wireguardConfig.PrivateKeyFile = config.sops.secrets.vpn2-pkey.path;

        wireguardPeers = [
          {
            wireguardPeerConfig = {
              PublicKey = "...";
              PresharedKeyFile = config.sops.secrets.vpn2-psharedkey.path;
              AllowedIPs = [ "10.4.0.0/15" ];
              Endpoint = "w'.x'.y'.z':1234";
              PersistentKeepalive = 25;
            };
          }
        ];
      };
    };

    networks = {
      "50-vpn1" = {
        address = [ /* VPN 1 addresses */ ];
        dns = [ /* DNS 1 IPs */ ];
        DHCP = "no";
        linkConfig.ActivationPolicy = "manual";
        name = "50-vpn1";

        networkConfig = {
          Description = "Wireguard network config for vpn1";
          DNSOverTLS = "opportunistic";
        };
      };

      "50-vpn2" = {
        address = [ /* VPN 2 addresses */ ];
        dns = [ /* DNS 2 IPs */ ];
        DHCP = "no";
        linkConfig.ActivationPolicy = "manual";
        name = "50-vpn2";

        networkConfig = {
          Description = "Wireguard network config for vpn2";
          DNSOverTLS = "opportunistic";
        };
      };
    };
  };

Thanks a lot for your help @R-VdP!!!

1 Like