Setting up WireGuard in a network namespace for selectively routing traffic through VPN

[kylegentle]

EDIT: Now that this is solved, a working configuration can be found further down the thread.

Hello, I’m trying to set up WireGuard for the first time in NixOS. The desired outcome is a network namespace that has the WireGuard interface as its only interface, as described in the WireGuard docs.

There have been a couple prior posts here that I’ve referred to over the past few days as I’ve worked on this:

Understanding wireguard, network namespaces and running services in them

Hi, So I’d like to have a service running through a VPN. I’ve been trying to read up on this lately but I’m generally confused on how to do it. Have I misunderstood something with the following: I create a new netns, create a new wireguard device, set it up, and move the wireguard device to that new netns, then I start applications and services in this namespace. So far so good? How do I actually do this? Any obvious caveats?

Run systemd service in network namespace

I could prepend ip netns exec xx where applicable (e.g. serviceConfig.ExecStart). But ip netns requires root privileges. What can I do if serviceConfig.User isn’t "root"?

Here’s the relevant configuration:

{ config, pkgs, ... }:
  systemd.services."netns@" = {
    description = "%I network namespace";
    before = [ "network.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = "${pkgs.iproute}/bin/ip netns add %I";
      ExecStop = "${pkgs.iproute}/bin/ip netns del %I";
    };
  };

  systemd.services.wg = {
    description = "wg network interface";
    bindsTo = [ "netns@wg.service" ];
    requires = [ "network-online.target" ];
    after = [ "netns@wg.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      NetworkNamespacePath = "/var/run/netns/wg";
      ExecStart = with pkgs; writers.writeBash "wg-up" ''
        ${iproute}/bin/ip link add wg0 type wireguard
        ${iproute}/bin/ip link set wg0 netns wg
        ${iproute}/bin/ip -n wg address add <ipv4 VPN addr/cidr> dev wg0
        # ${iproute}/bin/ip -n wg -6 address add <ipv6 VPN addr/cidr> dev wg0
        ${wireguard}/bin/wg setconf wg0 /root/myVPNprovider.conf
        ${iproute}/bin/ip -n wg link set wg0 up
        ${iproute}/bin/ip -n wg route add default dev wg0
        # ${iproute}/bin/ip -n wg -6 route add default dev wg0
      '';
      ExecStop = with pkgs; writers.writeBash "wg-down" ''
        ${iproute}/bin/ip -n wg route del default dev wg0
        # ${iproute}/bin/ip -n wg -6 route del default dev wg0
        ${iproute}/bin/ip -n wg link del wg0
      '';
    };
  };

The WireGuard config at /root/myVPNprovider.conf looks like so:

[Interface]
PrivateKey = <my private key>

[Peer]
PublicKey = <VPN provider endpoint pubkey>
AllowedIPs = 0.0.0.0/0,::0/0
Endpoint = <VPN provider IPv4:port>

When I start wg.service, the wg namespace is created, wg0 interface exists inside it, and the default route is added. However, I have no connectivity at all inside the wg namespace; no DNS, no ping, nothing.

Output from ip netns exec wg wg shows that no handshake has been exchanged, and no traffic is received:

interface: wg0
  public key: <my pubkey>
  private key: (hidden)
  listening port: 53739

peer: <VPN provider endpoint pubkey>
  endpoint: <VPN provider IPv4:port>
  allowed ips: 0.0.0.0/0, ::/0
  transfer: 0 B received, 36.57 KiB sent

ip info:

[me@pc:~]$ sudo ip -n wg link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 

[me@pc:~]$ sudo ip -n wg addr
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet <ipv4 VPN addr/cidr> scope global wg0
       valid_lft forever preferred_lft forever

[me@pc:~]$ sudo ip -n wg route
default dev wg0 scope link 

I’ve spent many hours working on this, and I feel stuck.

Does anything look wrong? How would you troubleshoot this setup, and what can I try next?

[mth]

I’ll try to reproduce this later, but:

• Is there anything in wg.service’s syslog?
• Could it be your peer?
• Not sure that’s significant, but my ip -n wg r has a second line which looks like a.b.0.0/x dev wg proto kernel scope link src a.b.c.d. It might be a consequence of a successful handshake.

[kylegentle]

Thanks mth, I look forward to hearing if you’re able to repro.

I think there’s not much useful in wg.service’s syslog; the service starts and stops successfully, so just lots of this from me turning off and on again:

Nov 28 16:17:50 pc systemd[1]: Stopping wg network interface...
Nov 28 16:17:50 pc systemd[1]: wg.service: Succeeded.
Nov 28 16:17:50 pc systemd[1]: Stopped wg network interface.
Nov 28 16:17:52 pc systemd[1]: Starting wg network interface...
Nov 28 16:17:52 pc systemd[1]: Finished wg network interface.

Problems with the peer are always possible, but the VPN provider’s service status page shows that the two different endpoints I’ve tried are both healthy, and I’ve been trying this for a couple days.

That additional route is interesting. I don’t know why it’s there either, but I’ll keep tinkering, and maybe try setting things up from scratch in a non-NixOS VM to see what differs.

[mth]

Ah, I think the problem is that your wg.service runs in the wg network namespace. It shouldn’t, because your wg0 interface should be created in your default namespace and only then moved to wg. This is the point of the line:

${iproute}/bin/ip link set wg0 netns wg

If wg.service runs in wg, this line does nothing.

[mth]

From the WireGuard docs:

WireGuard does something quite interesting. When a WireGuard interface is created (with ip link add wg0 type wireguard ), it remembers the namespace in which it was created. “I was created in namespace A.” Later, WireGuard can be moved to new namespaces (“I’m moving to namespace B.”), but it will still remember that it originated in namespace A.

[Ma27]

Can you please show us your full routing table? If you have a default route to through the WireGuard interface, but no route for <VPN provider IPv4> through your uplink interface (e.g. eth0/wlan0) your traffic will never reach your peer since your system tries to route the UDP traffic to the peer through the VPN itself which doesn’t work.

[mth]

@Ma27 I believe that’s the full routing table, in the wg netns at least. When using network namespaces one does not need to add the kind of routes you describe. Instead, the wireguard interface accepts plain packets in one netns and drops the encrypted (and redirected, etc.) packets in the network namespace it was originally created in. But since @kylegentle’s wg.service originally creates wg0 in wg, packets never reach the default namespace with the physical interfaces and indeed:

your traffic will never reach your peer since your system tries to route the UDP traffic to the peer through the VPN itself

[kylegentle]

You’re exactly right @mth, the fundamental issue was running wg.service inside the newly created wg namespace. The service needs to run in the default namespace. Thank you very much for your help!

I’ll include a summary here for those who may find this post later.

Overall I had to make two changes:

1. Remove the NetworkNamespacePath line from wg.service’s serviceConfig
2. Update the wg setconf command to run via ip netns exec, since it runs after wg0 is moved to the wg namespace.

My working config is thus:

{ config, pkgs, ... }:
  systemd.services."netns@" = {
    description = "%I network namespace";
    before = [ "network.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = "${pkgs.iproute}/bin/ip netns add %I";
      ExecStop = "${pkgs.iproute}/bin/ip netns del %I";
    };
  };

  systemd.services.wg = {
    description = "wg network interface";
    bindsTo = [ "netns@wg.service" ];
    requires = [ "network-online.target" ];
    after = [ "netns@wg.service" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = with pkgs; writers.writeBash "wg-up" ''
        set -e
        ${iproute}/bin/ip link add wg0 type wireguard
        ${iproute}/bin/ip link set wg0 netns wg
        ${iproute}/bin/ip -n wg address add <ipv4 VPN addr/cidr> dev wg0
        ${iproute}/bin/ip -n wg -6 address add <ipv6 VPN addr/cidr> dev wg0
        ${iproute}/bin/ip netns exec wg \
          ${wireguard}/bin/wg setconf wg0 /root/myVPNprovider.conf
        ${iproute}/bin/ip -n wg link set wg0 up
        ${iproute}/bin/ip -n wg route add default dev wg0
        ${iproute}/bin/ip -n wg -6 route add default dev wg0
      '';
      ExecStop = with pkgs; writers.writeBash "wg-down" ''
        ${iproute}/bin/ip -n wg route del default dev wg0
        ${iproute}/bin/ip -n wg -6 route del default dev wg0
        ${iproute}/bin/ip -n wg link del wg0
      '';
    };
  };

With this updated config, services can now be defined that will be exclusively routed through the wg0 interface. These new services can utilize the NetworkNamespacePath option to run inside the wg namespace. The basic skeleton is:

systemd.services.myWireguardOnlyService = {
  description = "Service that will only have access to the wg0 interface"
  bindsTo = [ "netns@wg.service" ];
  requires = [ "network-online.target" ];
  after = [ "wg.service" ];
  serviceConfig = {
    ...
    NetworkNamespacePath = "/var/run/netns/wg";
    ...
  };
}

I’m sure this could be refactored and cleaned up a bit, but it will serve my needs nicely for now.

[mth]

• Update the wg setconf command to run via ip netns exec , since it runs after wg0 is moved to the wg namespace.

Or just run wg setconf before moving the interface? As long as it isn’t up it should make no difference.