NixOS Firewall with Kubernetes

Help
1

I’ve set up a couple of Kubernetes clusters with firewall interfaces like ufw, iptables, nftables, etc.

Recently, I set up the first cluster running on NixOS.

I cannot enable the NixOS-managed firewall without the cluster becoming unreachable.
From the pods’ logs, it seems like all internal traffic is blocked, as well. Not sure.

In the process of debugging this & trying to make the situation work, I punched so many holes into the firewall with tons of allow rules for wide ranges of TCP & UDP ports. It’s way too much, but I wanted to make this work.
Now, even with all these holes, it still won’t work. Perhaps, because all the ports wouldn’t help, if the internal traffic is entirely blocked, as well or something resembling this behaviour.

Is there something obvious about NixOS’ firewall management I am missing?

If not, I’ll have to debug this really manually in an extremely circuitous way…

This is how the scenario looks like:

  1. Firewall is disabled.
    Everything works.
  2. Firewall is enabled.
    Getting log messages with health checks failing, because ClusterIP:Port is unreachable. These are internal addresses, not reachable from the outside.
  3. Firewall is disabled.
    Health checks start working again. Everything works.
2

For debugging purposes, I opened all ports shown by ss -ntlp and ss -nulp, respectively. Did not help.

As a last resort, I assigned all Kubernetes related network interfaces to networking.firewall.trustedInterfaces.

This will do, for now.

3

The NixOS firewall is set up to log rejected connections (on the receiving side) to the system journal. Perhaps checking it would help you discover the issue.

Feel free to also check out my nixos-ha-kubernetes project, you might spot some configuration line you might be missing.

4

Greetings @justinas, thank you for the hints. It is highly appreciated. :+1:

I will check those out, the next time I’m dealing with the cluster. :slightly_smiling_face: