Hello!
I am having issues running docker containers in nix.
In particular docker containers suddenly stopped having network connection one day, both during building and running, without a change to my network config.
Here is said config (As well as docker and libvirtd-related config, as I have those too, and I understand these don’t often work well.)
{
networking.hostName = "workstop";
# Use NetworkManager
networking.networkmanager.enable = true;
# Firewall Settings
networking.firewall.allowedTCPPorts = [
53317 # LocalSend
];
networking.firewall.allowedUDPPorts = [
53317 # LocalSend
];
virtualisation.docker.enable = true;
programs.virt-manager.enable = true;
environment.systemPackages = with pkgs; [ virtiofsd ];
virtualisation.libvirtd.enable = true;
users.users.adriano.extraGroups = [ "libvirtd", "docker" ];
}
Things I have tried:
- Restarted libvirtd/dockerd/both
- Flushed iptables rules while services are down with
iptables -F
iptables -X
sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
networking.firewall.trustedInterfaces = ["docker0" "virbr0"];
networking.firewall.checkReversePath = false;
- Checking /etc/resolv.conf (Its correct)
- Confirmed I have network connection outside the docker containers (Images pull fine, and I’m writing this through the same machine)
For completeness, here’s the iptables rules
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
nixos-fw all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-FORWARD all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
Chain DOCKER-BRIDGE (1 references)
target prot opt source destination
DOCKER all -- anywhere anywhere
Chain DOCKER-CT (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
target prot opt source destination
DOCKER-CT all -- anywhere anywhere
DOCKER-INTERNAL all -- anywhere anywhere
DOCKER-BRIDGE all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain DOCKER-INTERNAL (1 references)
target prot opt source destination
Chain DOCKER-USER (1 references)
target prot opt source destination
Chain nixos-drop (0 references)
target prot opt source destination
DROP all -- anywhere anywhere
Chain nixos-fw (1 references)
target prot opt source destination
nixos-fw-accept all -- anywhere anywhere
nixos-fw-accept all -- anywhere anywhere ctstate RELATED,ESTABLISHED
nixos-fw-accept tcp -- anywhere anywhere tcp dpt:53317
nixos-fw-accept udp -- anywhere anywhere udp dpt:53317
nixos-fw-accept icmp -- anywhere anywhere icmp echo-request
nixos-fw-log-refuse all -- anywhere anywhere
Chain nixos-fw-accept (5 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain nixos-fw-log-refuse (1 references)
target prot opt source destination
LOG tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/SYN LOG level info prefix "refused connection: "
nixos-fw-refuse all -- anywhere anywhere PKTTYPE != unicast
nixos-fw-refuse all -- anywhere anywhere
Chain nixos-fw-refuse (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
# iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-BRIDGE
-N DOCKER-CT
-N DOCKER-FORWARD
-N DOCKER-INTERNAL
-N DOCKER-USER
-N nixos-drop
-N nixos-fw
-N nixos-fw-accept
-N nixos-fw-log-refuse
-N nixos-fw-refuse
-A INPUT -j nixos-fw
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A DOCKER ! -i docker0 -o docker0 -j DROP
-A DOCKER-BRIDGE -o docker0 -j DOCKER
-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-FORWARD -j DOCKER-CT
-A DOCKER-FORWARD -j DOCKER-INTERNAL
-A DOCKER-FORWARD -j DOCKER-BRIDGE
-A DOCKER-FORWARD -i docker0 -j ACCEPT
-A nixos-drop -j DROP
-A nixos-fw -i lo -j nixos-fw-accept
-A nixos-fw -m conntrack --ctstate RELATED,ESTABLISHED -j nixos-fw-accept
-A nixos-fw -p tcp -m tcp --dport 53317 -j nixos-fw-accept
-A nixos-fw -p udp -m udp --dport 53317 -j nixos-fw-accept
-A nixos-fw -p icmp -m icmp --icmp-type 8 -j nixos-fw-accept
-A nixos-fw -j nixos-fw-log-refuse
-A nixos-fw-accept -j ACCEPT
-A nixos-fw-log-refuse -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "refused connection: " --log-level 6
-A nixos-fw-log-refuse -m pkttype ! --pkt-type unicast -j nixos-fw-refuse
-A nixos-fw-log-refuse -j nixos-fw-refuse
-A nixos-fw-refuse -j DROP
I have found the problem was a DNS problem, not a connectivity problem. The bridge interfaces had no DNS configured. Manually setting DNS in networkmanager works, but it is a bandaid solution. Any idea how this could happen?