Wireguard client with Linuxserver.io wireguard server

I have wireguard setup using the Linuxserver.io containers. I’ve documented my “server” setup here: Wireguard – self hosted VPN – Roo's View

This works great. I’m able to connect from multiple devices. Some of them I can trivially ‘crawl back’ from the server side - to the remote device that is running wireguard (very handy).

The linuxserver.io docker image - puts the devices on a 10.13.13.0/24 network - so this means my ‘client’ gets say 10.13.13.3 as it’s “IP”. From the server - with the right ‘ip route’ magic - I can then actually reach that remote device that is maybe running SSHD on port 8022 by doing.

ssh -P 8022 user@10.13.13.3

and it works! Magic. Amazing.

So - I have some evidence that I have a working wireguard “server”, but I’m having trouble setting up a NixOS system to be a remote client that reaches back to that server and will let data flow.

I followed the first section in the documentation - specifically the client section. I did generate a new ‘peer’ configuration for my linuxserver.io wireguard server setup… so I had this as my configuration from the wireguard server side…

$cat peer_mynewclient.conf 
[Interface]
Address = 10.13.13.9
PrivateKey = SECRET111XXX=
ListenPort = 51820
DNS = 192.168.1.8

[Peer]
PublicKey = SECRET222XXX=
PresharedKey = SECRET333XXX=
Endpoint = example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0

With this information - I then was use it as a reference to setup my /etc/nixos/configuration.nix file to have this.

  networking.firewall = {
    allowedUDPPorts = [ 51820 ]; # wireguard
  };

  # Enable WireGuard
  networking.wireguard.interfaces = {
    # "wg0" is the network interface name. You can name the interface arbitrarily.
    wg0 = {
      # Determines the IP address and subnet of the client's end of the tunnel interface.
      ips = [ "10.13.13.9/32" ];
      listenPort = 51820; # to match firewall allowedUDPPorts (without this wg uses random port numbers)

      # Path to the private key file.
      #
      # Note: The private key can also be included inline via the privateKey option,
      # but this makes the private key world-readable; thus, using privateKeyFile is
      # recommended.
      privateKeyFile = "/etc/nixos/secrets/privatekey.wg";

      peers = [
        # For a client configuration, one peer entry for the server will suffice.

        {
          # Public key of the server (not a file path).
          publicKey = "SECRET222XXX=";

          # Forward only particular subnets
          allowedIPs = [ "192.168.1.0/22" ];

          # Set this to the server IP and port.
          endpoint = "example.com:51820"; # ToDo: route to endpoint not automatically configured https://wiki.archlinux.org/index.php/WireGuard#Loop_routing https://discourse.nixos.org/t/solved-minimal-firewall-setup-for-wireguard-client/7577

          # Send keepalives every 25 seconds. Important to keep NAT tables alive.
          persistentKeepalive = 25;
        }
      ];
    };
  };

And yes, I put the SECRET111XXX= into the file /etc/nixos/secrets/privatekey.wg

At this point - my NixOS machine appears to create a wireguard tunnel from it - to my existing server. On the server side I can query docker exec -it wireguard wg show and see that it is connected.

On the NixOS client - it seems that it’s setup too

$ ifconfig -a wg0
wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1420
        inet 10.13.13.9  netmask 255.255.255.255  destination 10.13.13.9
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 586  bytes 86728 (84.6 KiB)
        TX errors 0  dropped 31 overruns 0  carrier 0  collisions 0

but… I can’t seem to ping from the client to the server (network)

$ ping 192.168.1.99
PING 192.168.1.99 (192.168.1.99) 56(84) bytes of data.
^C
--- 192.168.1.99 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

On the client - I can see sudo ip route showing me that indeed, I have a route defined

$ sudo ip route

192.168.1.0/22 dev wg0 scope link 

I’ve also tried turning off the firewall entirely on the NixOS client

networking.firewall.enable = false;

Still no joy.

Additionally - but only a bonus - I can’t seem to crawl back from the server side.

$ ping 10.13.13.9
PING 10.13.13.9 (10.13.13.9) 56(84) bytes of data.
^C
--- 10.13.13.9 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms

but this same ‘ping’ works for other clients that I have running and connected via wireguard.

This must be some sort of route/firewall issue on NixOS – but I’m just so confused.

I’ve reviewed the most relevant post - but it doesn’t seem to apply? Or I’m confused as to the ‘solution’.

This means that there are no reply packets on the “client”, so the probably the connection was not established. What exactly does wg output on both ends? Anything interesting if you enable the debug logs?

Here is the relevant ‘server’ side wg show output

peer: JS6SOMEGIANTHASHkk71jk=
  preshared key: (hidden)
  endpoint: 142.146.96.11:51820
  allowed ips: 10.13.13.9/32
  transfer: 3.58 MiB received, 2.67 MiB sent

The ‘endpoint’ IP address I changed - but it was the client machine IP address.

I think there is some evidence that the connection is ‘alive’ - but something on the NixOS client side preventing data flow

Looks like server is sending replies but the client is not getting them(assuming wg show shows 0 rx on client). Do check the debug logs on both sides. The nix config looks ok at a glqnce, maybe there is something between the client and the server that’s preventing the traffic flow?

Ok - update. If I remove the ‘split tunnel’ - and force all data to go through wireguard - then when I enable the configuration I’m fully broken for the network.

Thus changing


          # Forward only particular subnets
          allowedIPs = [ "192.168.1.0/22" ];

to all things

          # Forward all traffic
          allowedIPs = [ "0.0.0.0/0" ];

This configuration also makes it much more obvious that something isn’t right.

Again, recall that I have evidence my wireguard ‘server’ side is working fine as I have other non-NixOS devices working fine with it.

On the NixOS ‘client’ - sudo wg show shows 0 B received but I see sent data
On the server side - I see data going both ways

peer: JS6SOMEGIANTHASHkk71jk=
  preshared key: (hidden)
  endpoint: 142.146.96.11:51820
  allowed ips: 10.13.13.9/32
  transfer: 13.59 KiB received, 8.45 KiB sent

Hmm… Looking at the debug log enablement - the modprobe gives me nothing on my NixOS client machine?

That feels wrong… but enabling debug seems to work?

echo 'module wireguard +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

and in the logs journalctl -xe I see regular [note: maybe these errors were coming out even without the debug stuff added?]

Invalid handshake response from 142.146.96.11:51820

So progress… something is amiss with my configuration

Of course - this is made more difficult to share as I’ve got multiple computers and they can’t see each other easily… – here is a more complete log from the NixOS client

Jul 30 15:47:16 nixos kernel: wireguard: wg0: Handshake for peer 2 (142.146.96.11:51820) did not complete after 5 seconds, retrying (try 15)
Jul 30 15:47:16 nixos kernel: wireguard: wg0: Sending handshake initiation to peer 2 (142.146.96.11:51820)
Jul 30 15:47:16 nixos kernel: wireguard: wg0: Invalid handshake response from 142.146.96.11:51820
Jul 30 15:47:21 nixos kernel: wireguard: wg0: Handshake for peer 2 (142.146.96.11:51820) did not complete after 5 seconds, retrying (try 16)
Jul 30 15:47:21 nixos kernel: wireguard: wg0: Sending handshake initiation to peer 2 (142.146.96.11:51820)
Jul 30 15:47:21 nixos kernel: wireguard: wg0: Invalid handshake response from 142.146.96.11:51820

So maybe it’s not an ‘invalid’ payload – but the lack of a payload due to the timeout.

As I iterate towards a ‘working’ solution (but one I don’t yet understand… ) it does seem that I can add a route to a specific IP and reach it over the wireguard connection!?

This makes no sense as I’m still getting regular errors in the logs as in my comment above. However I can see that yes, the NixOS client is on a different network (mobile hotspot).

I used this stackoverflow post as a guide

sudo ip route add 192.168.1.50 via 192.168.90.67

Where the IP address 192.168.90.67 is the gateway that my mobile hotspot is offering up to the NixOS client machine.

I can then successfully reach the web server found at 192.168.1.50 which is NOT exposed to the internet… and is also NOT on the same network as the NixOS client machine…

I’m still stumped here with a few things
a) Why am I getting all of those handshake errors
b) Why does this work at all? I don’t see how routing traffic destined for a private IP on the otherside of the wireguard tunnel to the gateway for the NixOS client should ‘fix’ traffic.

Hmm… pivoting to using the wg-quick … it just works? In this case I also get to use the presharedKey data that my wireguard server (linuxserver.io) creates.

I just followed the doc here

Well - thanks for following along. I’m not sure I totally get what wasn’t working or now why it’s working just fine. Moving from the networking.wireguard.interfaces to networking.wg-quick.interfaces just fixed me.

Here is a recap - and I’ll tag this as the ‘solution’.

  1. I’m running the linuxserver.io wireguard container on a machine I’ll call the server
  2. I have a second machine running NIxOS which I will call the client

My server is configured as I wrote up here - having wireguard setup has been amazing, I would highly recommend it.

This post is about managing a NixOS client which lives at a different physical location than the server, and has a completely different internet connection. I first tried the documented client setup - and ran into problems - as you can read in this thread.

FWIW - the client is a desktop install of NixOS. While it is currently headless, if you plug in a monitor you’ll get the full gnome desktop experience.

My solution ended up being simple. I just followed the client setup for wg-quick

I take the generated peer_client.conf file that I generate using the linuxserer.io container that is running on my server. It looks something like this:

$cat peer_client.conf 
[Interface]
Address = 10.13.13.9
PrivateKey = SECRET111XXX=
ListenPort = 51820
DNS = 192.168.1.8

[Peer]
PublicKey = SECRET222XXX=
PresharedKey = SECRET333XXX=
Endpoint = example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0

And map it into the client nixos configuration file

  networking.firewall = {
    allowedUDPPorts = [ 51820 ]; # wireguard
  };

  # Enable WireGuard
  networking.wg-quick.interfaces = {
    # "wg0" is the network interface name. You can name the interface arbitrarily.
    wg0 = {
      # Determines the IP address and subnet of the client's end of the tunnel interface.
      address = [ "10.13.13.9/32" ];
      listenPort = 51820; # to match firewall allowedUDPPorts (without this wg uses random port numbers)
      privateKeyFile = "/etc/nixos/secrets/privatekey.wg";

      peers = [
        {
          # Public key of the server (not a file path).
          publicKey = "SECRET222XXX=";

          # Pre-shared key file
          presharedKeyFile = "/etc/nixos/secrets/preshare.wg";

          # Forward only particular subnets
          allowedIPs = [ "192.168.1.0/22" ];

          # Set this to the server IP and port.
          endpoint = "example.com:51820"; 

          # Send keepalives every 25 seconds. Important to keep NAT tables alive.
          persistentKeepalive = 25;
        }
      ];
    };
  };

You will notice I’ve got the firewall allowedUDPPorts I’m not sure this is required, but I had it from before and it seems to at least be harmless. I’m also specifying a listenPort which may not be required, but again doesn’t seem to hurt.

At this point - it all just works. I don’t see any particular errors in the logs. I can reach back from the server to the client

me@server:~$ ping 10.13.13.9
PING 10.13.13.9 (10.13.13.9) 56(84) bytes of data.
64 bytes from 10.13.13.9: icmp_seq=1 ttl=63 time=30.7 ms
64 bytes from 10.13.13.9: icmp_seq=2 ttl=63 time=29.6 ms
^C
--- 10.13.13.9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 29.633/30.188/30.743/0.555 ms

I can even ssh ‘back’ over the wireguard link. My client location has zero ports opened in the firewall – but from that remote client I can resolve things on my local (to the server) network.

This is pretty awesome. Sure I could just start using tailscale – but this is all 100% self hosted (yes, I’m also aware of headscale) – and it has only a few moving parts.

In the end - this setup does exactly what I set out to do. Woo hoo!