Docker's firewall conflicting with NixOS's networking

[PadowYT2]

I’m using Docker to host mailcow (as there are no better alternatives to it and isn’t available as a service) via docker compose. This is my Nix networking configuration:

networking.nix

{
  networking = {
    nameservers = ["1.1.1.1" "1.0.0.1"];
    useDHCP = false;
    interfaces.ens3 = {
      useDHCP = false;
      ipv4.addresses = [
        {
          address = "REDACTED";
          prefixLength = 24;
        }
      ];
    };
    defaultGateway = "REDACTED";
    enableIPv6 = false;
    firewall.enable = true;
    nat = {
      enable = true;
      internalInterfaces = ["br-mailcow"];
      externalInterface = "ens3";
    };
  };

  services.resolved = {
    enable = true;
    fallbackDns = ["1.1.1.1" "1.0.0.1"];
  };
}

The problem is, when I have virtualisation.docker.daemon.settings.iptables enabled, randomly after a couple of hours all incoming packets just refuse to come in. You can use the internet on the server (access websites, ping services) but all pings (and e.g. accessing the website hosted on the server) coming to the server just results in packet loss. Only after restarting Docker, everything goes back to working. If I however disable the option, it leaves the server to a lot of vulnerabilities (which I have experienced).

Any help would be welcome, as I’m not sure where I should be looking for the problem

[withakay]

Sorry this is probably less help and more just rubber-ducking:

You can use the internet on the server (access websites, ping services) but all pings (and e.g. accessing the website hosted on the server) coming to the server just results in packet loss.

That does sound like firewall shenanigans, but

randomly after a couple of hours

seems not typical firewall behavior, unless somehow the iptables/nftables rules are different between when you started the mailcow container and when things stopped working.

Going back to:

You can use the internet on the server (access websites, ping services) but all pings (and e.g. accessing the website hosted on the server) coming to the server just results in packet loss.

When you say “server”, does this mean you have outgoing access from within the mailcow container, but that incoming traffic from the host to the container fails? And by extension that incoming traffic from outside the host also fails to reach the container? Is this with IPs or with domain names?

Have you tried disabiling the docker firewall but a strict firewall on the part of the NixOS host (that would render mailcow container inoperable from the internet, but still reachable from the NixOS host) and seeing if after some time the NixOS host becomes unable to reach the container?

[PadowYT2]

@withakay

When you say “server”, does this mean you have outgoing access from within the mailcow container, but that incoming traffic from the host to the container fails? And by extension that incoming traffic from outside the host also fails to reach the container?

Outgoing access from the whole machine and incoming traffic to the machine itself. The Docker container can communicate with the outside and internally but as you can’t access the machine itself, you can’t access the container neither.

@withakay

Is this with IPs or with domain names?

This happens with both IPs and domains which means it shouldn’t be a DNS problem (hopefully).

@withakay

Have you tried disabiling the docker firewall but a strict firewall on the part of the NixOS host (that would render mailcow container inoperable from the internet, but still reachable from the NixOS host) and seeing if after some time the NixOS host becomes unable to reach the container?

I actually did that but it’s not really “secure”. It doesn’t have the same issue as I have with the Docker’s firewall enabled but there are still some security problems I’m experiencing (e.g. somehow malicious bots are able to send emails from non-existing accounts with Docker local IPs). It would still be better to have the Docker’s firewall enabled in my opinion

[withakay]

@PadowYT2

I actually did that but it’s not really “secure”. It doesn’t have the same issue as I have with the Docker’s firewall enabled but there are still some security problems I’m experiencing (e.g. somehow malicious bots are able to send emails from non-existing accounts with Docker local IPs).

Hrmm… that seems bizarre. The test setup I’m envisioning would have the NixOS host firewall set so that no state=NEW incoming traffic is allowed (except maybe 22 for SSH if you don’t have physical access to the NixOS host), so I’m not really clear on how anything is getting from the broader internet to your container in that firewall setup. Does the docker container create a new iptables rule to punch through the firewall when you disable firewall for the container?

What does something like nft list ruleset from the NixOS host say?

[Sped0n]

perhaps you can try to play around with below two options? they used to give me so much headache while setting my docker stuff in nixos

• networking.firewall.trustedInterfaces (try add docker0)
• networking.firewall.checkReversePath (try disable it)

still as @withakay say, i would still recommend you to disable docker’s internal firewall management, since it really don’t play nice with the system firewall

maybe not that helpful, but here is my current docker and firewall setup

[PadowYT2]

@withakay

Does the docker container create a new iptables rule to punch through the firewall when you disable firewall for the container?

I’m not sure how can I “disable” firewall for the container.

@withakay

What does something like nft list ruleset from the NixOS host say?

I’m not running nftables but iptables:

iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
MAILCOW    all  --  anywhere             anywhere             /* mailcow */
nixos-fw   all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
MAILCOW    all  --  anywhere             anywhere             /* mailcow */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-FORWARD  all  --  anywhere             anywhere            
nixos-filter-forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain DOCKER-BRIDGE (1 references)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            

Chain DOCKER-CT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain DOCKER-FORWARD (1 references)
target     prot opt source               destination         
DOCKER-CT  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
DOCKER-BRIDGE  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         

Chain MAILCOW (2 references)
target     prot opt source               destination         

Chain nixos-drop (0 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain nixos-filter-forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED

Chain nixos-fw (1 references)
target     prot opt source               destination         
nixos-fw-accept  all  --  anywhere             anywhere            
nixos-fw-accept  all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:ssh
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:smtp
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:http
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:pop3
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:imap
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:https
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:urd
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:submission
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:imaps
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:pop3s
nixos-fw-accept  tcp  --  anywhere             anywhere             tcp dpt:sieve
nixos-fw-accept  icmp --  anywhere             anywhere             icmp echo-request
nixos-fw-log-refuse  all  --  anywhere             anywhere            

Chain nixos-fw-accept (14 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain nixos-fw-log-refuse (1 references)
target     prot opt source               destination         
LOG        tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN LOG level info prefix "refused connection: "
nixos-fw-refuse  all  --  anywhere             anywhere             PKTTYPE != unicast
nixos-fw-refuse  all  --  anywhere             anywhere            

Chain nixos-fw-refuse (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

This is when the Docker firewall is disabled, I have also added a rule to atleast deflect some spam: iptables -I INPUT -d 172.22.1.1 ! -s 127.0.0.1 -j DROP.

@Sped0n

perhaps you can try to play around with below two options? they used to give me so much headache while setting my docker stuff in nixos

• networking.firewall.trustedInterfaces (try add docker0)
• networking.firewall.checkReversePath (try disable it)

still as @withakay say, i would still recommend you to disable docker’s internal firewall management, since it really don’t play nice with the system firewall

maybe not that helpful, but here is my current docker and firewall setup

I will try it out and report, thanks

[PadowYT2]

I have applied some changes to Mailcow’s configuration to defend against spam, made the iptables rule declarative and honestly that’s it. Originally I have thought it would be better to have the Docker firewall enabled but as I’ve read, it would be better not to.

Thanks to @Sped0n and @withakay for the tips