Getting K3S to work with "external" wireguard interface

Xnyle · May 20, 2025, 7:16pm

I’m currently stuck with a network connectivity issue regarding k3s and wireguard.

I configured k3s as desribed here:

pkgs/applications/networking/cluster/k3s/docs/USAGE.md

master

# K3s Usage

## Single Node

```
{
  networking.firewall.allowedTCPPorts = [
    6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
    # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
    # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
  ];
  networking.firewall.allowedUDPPorts = [
    # 8472 # k3s, flannel: required if using multi-node for inter-node networking
  ];
  services.k3s.enable = true;
  services.k3s.role = "server";
  services.k3s.extraFlags = toString [
    # "--debug" # Optionally add additional args to k3s
  ];
}

This file has been truncated. show original

It is for now a single node cluster but I want to change the external interface that k3s/flannel uses to wgX, a wireguard interface that is also set up and working.

So I add the following optinons:

extraFlags = toString [
“–disable traefik”
“–disable local-storage”
“–disable servicelb”
“–flannel-iface=wgX”
“–flannel-external-ip=false”
“–node-ip=172.27.24.222”
“–advertise-address=172.27.24.222”
#“–debug”
];

I tested different things, the line that breaks things is --flannel-iface=wgX
(ofc I always k3s-uninstall.sh and even reboot after attempts so I start fresh)

From the logs I can see that flannel reacts to the change and elects 172.27.24.222 as external address for its flannel.0 bridge.

But what is happening?
As soon as I add that change, Pods cant talk to each other and nodes anymore, basically nothing works.

I don’t understand how changing --flannel-iface=wgX has any influence on pod to pod or pod to node communication at all? My understanding is that in a single node setup, the flannel.0 bridge isn’t even used as no traffic needs to leave it / the node? all communication goes through the veth/cni0 interfaces?

For testing I also disabled the NixOS firewall which did not change anything.

Im currently trying to (manually) figure out the iptables rules, flannel creates, but maybe I’m missing something obvious?

So question: Does anyone else have k3s running under NixOS with the external iterface/addresses set to be a wireguard interface?

rorosen · May 21, 2025, 8:10am

Does anyone else have k3s running under NixOS with the external iterface/addresses set to be a wireguard interface?

I have never tried, but there are people doing this (not with NixOS but I think that doesn’t make a difference here).

Just guessing: Setting --advertise-address to what I think is the address of wgX looks suspicious. I don’t think the apiserver can advertise itself via the flannel iface. Try using another address that the other nodes can reach, or consider using wgX only for advertise-address and set --flannel-backend=wireguard-native to let k3s manage its own wireguard interface for flannel. Another thing I noted is --flannel-external-ip=false. This is only a flag without a value and I’m unsure how the =false bit is interpreted.

Xnyle · May 21, 2025, 9:48am

But that is the address that other nodes can reach. (and the only one they should reach)
This setup runs perfectly fine on other Cluster setups (Talos/Vanilla on Debian) so I suspected either something going on with either the k3s embedded flannel or NixOS itself.

I now replaced k3s with k8s and get the exact same problem. k8s depends on flannel.
And as soon as I add

services.flannel.iface = “${flannelIface}”;

communication between pods (that should not even rely on the flannel iface itself cause it just uses the cni0 bridge) breaks.

It must be something regarding how flannel sets up iptables rules in conjunction with the wgX interface and or the NixOS firewall. (but that one is “disabled”)

Bonus Question: Is there an easy solution / helper app/programm that can tell me the path through the firewall rules a given packet would take?

Xnyle · May 21, 2025, 11:21am

After trying for 2 more hours I gave up to debug the problem and just did what you suggested: Added another dummy bridge interface and flannel is happy with that.
Only downside now is additional routing setup in wireguard.