No internet through docker

stianlagstad · January 30, 2023, 7:45am

At my office’s network I started getting this issue with internet access in docker last week. Here’s what happens:

➜ docker run busybox nslookup google.com
;; connection timed out; no servers could be reached

➜ nslookup google.com
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.207.238
Name:   google.com
Address: 2a00:1450:400f:80c::200e

➜ docker run busybox cat /etc/resolv.conf
# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.1.1.1
options edns0

➜ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.1.1.1
options edns0

➜ docker run busybox route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 eth0

➜ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    3003   0        0 wlp59s0
10.180.91.0     0.0.0.0         255.255.255.0   U     3003   0        0 wlp59s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet0

➜ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:72ff:fe03:3615  prefixlen 64  scopeid 0x20<link>
        ether 02:42:72:03:36:15  txqueuelen 0  (Ethernet)
        RX packets 32  bytes 1696 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91  bytes 11764 (11.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 116  bytes 13670 (13.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 116  bytes 13670 (13.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vboxnet0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.56.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 0a:00:27:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp59s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.180.91.211  netmask 255.255.255.0  broadcast 10.180.91.255
        inet6 fe80::bae8:a31d:41ba:b1b7  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::aa6d:aaff:fe2b:98a  prefixlen 64  scopeid 0x20<link>
        ether a8:6d:aa:2b:09:8a  txqueuelen 1000  (Ethernet)
        RX packets 64726  bytes 71990806 (68.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21695  bytes 7359838 (7.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

➜ docker run busybox ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:200 (200.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

➜ docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
7596438b2c25   bridge    bridge    local
617a6542f360   host      host      local
2575c2c192a9   none      null      local

➜ docker run --network host busybox nslookup google.com
Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   google.com
Address: 216.58.207.238

Non-authoritative answer:
Name:   google.com
Address: 2a00:1450:400f:80c::200e

➜ docker run --network bridge busybox nslookup google.com
;; connection timed out; no servers could be reached

If I instead of using the office wifi use the hotspot through my phone, then I don’t have this issue. However, no one else in the office has this issue. What can I do to debug this further?

iFreilicht · January 30, 2023, 10:21am

Are you expecting the bridge network to forward requests from containers to the internet? That is not enabled by default. The setting is dependent on your kernel-configuration and iptables, so I would assume that’s why you’re only seeing this problem in one specific network configuration. From the link above:

By default, traffic from containers connected to the default bridge network is not forwarded to the outside world. To enable forwarding, you need to change two settings. These are not Docker commands and they affect the Docker host’s kernel.
Configure the Linux kernel to allow IP forwarding.
$ sysctl net.ipv4.conf.all.forwarding=1
Change the policy for the iptables FORWARD policy from DROP to ACCEPT .
sudo iptables -P FORWARD ACCEPT
These settings do not persist across a reboot, so you may need to add them to a start-up script.

stianlagstad · January 30, 2023, 10:31am

I do expect containers that I run is able to reach the internet. They did up until late last week until I started experiencing this issue. I have not changed anything network-related on my computer. And, when I use another wifi connection, without changing anything else on my computer, things suddenly work.

stianlagstad · January 30, 2023, 12:01pm

journalctl logs while doing docker run busybox nslookup google.com:

Jan 30 12:56:49 nixos-dell systemd[1]: var-lib-docker-overlay2-2f2d1fdb7946c3d363265e124055ba8a2d95d56c22cf226fed3890d5918766a1\x2dinit-merged.mount: Deactivated successfully.
Jan 30 12:56:49 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered blocking state
Jan 30 12:56:49 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered disabled state
Jan 30 12:56:49 nixos-dell kernel: device vethdaac7d5 entered promiscuous mode
Jan 30 12:56:49 nixos-dell dockerd[26758]: time="2023-01-30T12:56:49.729679158+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Jan 30 12:56:49 nixos-dell dockerd[26758]: time="2023-01-30T12:56:49.729780260+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Jan 30 12:56:49 nixos-dell dockerd[26758]: time="2023-01-30T12:56:49.729813574+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Jan 30 12:56:49 nixos-dell dockerd[26758]: time="2023-01-30T12:56:49.730141584+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da pid=31163 runtime=io.containerd.runc.v2
Jan 30 12:56:49 nixos-dell systemd[1]: Started libcontainer container 1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da.
Jan 30 12:56:49 nixos-dell kernel: eth0: renamed from veth2bf7173
Jan 30 12:56:49 nixos-dell kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethdaac7d5: link becomes ready
Jan 30 12:56:49 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered blocking state
Jan 30 12:56:49 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered forwarding state
Jan 30 12:56:59 nixos-dell 1c565bef62ce[26743]: ;; connection timed out; no servers could be reached
Jan 30 12:56:59 nixos-dell 1c565bef62ce[26743]:
Jan 30 12:56:59 nixos-dell systemd[1]: docker-1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da.scope: Deactivated successfully.
Jan 30 12:56:59 nixos-dell systemd[1]: docker-1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da.scope: Consumed 41ms CPU time, received 0B IP traffic, sent 448B IP traffic.
Jan 30 12:57:00 nixos-dell dockerd[26758]: time="2023-01-30T12:57:00.000809533+01:00" level=info msg="shim disconnected" id=1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da
Jan 30 12:57:00 nixos-dell dockerd[26758]: time="2023-01-30T12:57:00.000897935+01:00" level=warning msg="cleaning up after shim disconnected" id=1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da namespace=moby
Jan 30 12:57:00 nixos-dell dockerd[26758]: time="2023-01-30T12:57:00.000951679+01:00" level=info msg="cleaning up dead shim"
Jan 30 12:57:00 nixos-dell dockerd[26743]: time="2023-01-30T12:57:00.001069447+01:00" level=info msg="ignoring event" container=1c565bef62ce5c16e2b928a9f90a4c3f936c8f7953743fecc8ba53b6800cc2da module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 30 12:57:00 nixos-dell dockerd[26758]: time="2023-01-30T12:57:00.034680882+01:00" level=warning msg="cleanup warnings time=\"2023-01-30T12:57:00+01:00\" level=info msg=\"starting signal loop\" namespace=moby pid=31256 runtime=io.containerd.runc.v2\n"
Jan 30 12:57:00 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered disabled state
Jan 30 12:57:00 nixos-dell kernel: veth2bf7173: renamed from eth0
Jan 30 12:57:00 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered disabled state
Jan 30 12:57:00 nixos-dell kernel: device vethdaac7d5 left promiscuous mode
Jan 30 12:57:00 nixos-dell kernel: docker0: port 1(vethdaac7d5) entered disabled state
Jan 30 12:57:00 nixos-dell systemd[1]: run-docker-netns-a2e77515e954.mount: Deactivated successfully.
Jan 30 12:57:00 nixos-dell systemd-udevd[31269]: veth2bf7173: Process '/nix/store/4xw8n979xpivdc46a9ndcvyhwgif00hz-bash-5.1-p16/bin/sh -c 'echo 2 > /proc/sys/net/ipv6/conf/veth2bf7173/use_tempaddr'' failed with exit code 1.
Jan 30 12:57:00 nixos-dell systemd[1]: var-lib-docker-overlay2-2f2d1fdb7946c3d363265e124055ba8a2d95d56c22cf226fed3890d5918766a1-merged.mount: Deactivated successfully.
Jan 30 12:57:00 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such file or directory
Jan 30 12:57:00 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such file or directory
Jan 30 12:57:00 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such process

The same journal logs when doing docker run busybox nslookup google.com on the mobile hotspot, where things work:

Jan 30 12:58:41 nixos-dell systemd[1]: var-lib-docker-overlay2-f4ebb452dd5d12a24be9e79707dfe573a7121565193edf4940e5ad520fcf139a\x2dinit-merged.mount: Deactivated successfully.
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered blocking state
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered disabled state
Jan 30 12:58:41 nixos-dell kernel: device veth12fd5b2 entered promiscuous mode
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered blocking state
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered forwarding state
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered disabled state
Jan 30 12:58:41 nixos-dell dockerd[26758]: time="2023-01-30T12:58:41.689990064+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Jan 30 12:58:41 nixos-dell dockerd[26758]: time="2023-01-30T12:58:41.690058142+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Jan 30 12:58:41 nixos-dell dockerd[26758]: time="2023-01-30T12:58:41.690070019+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Jan 30 12:58:41 nixos-dell dockerd[26758]: time="2023-01-30T12:58:41.690264865+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f pid=31936 runtime=io.containerd.runc.v2
Jan 30 12:58:41 nixos-dell systemd[1]: Started libcontainer container 56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f.
Jan 30 12:58:41 nixos-dell kernel: eth0: renamed from veth2cca38e
Jan 30 12:58:41 nixos-dell kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth12fd5b2: link becomes ready
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered blocking state
Jan 30 12:58:41 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered forwarding state
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Server:                192.168.236.121
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address:        192.168.236.121:53
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]:
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Non-authoritative answer:
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.101
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.102
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.113
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.138
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.139
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 142.250.147.100
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]:
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Non-authoritative answer:
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 2a00:1450:4025:c03::64
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 2a00:1450:4025:c03::65
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 2a00:1450:4025:c03::66
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Name:        google.com
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]: Address: 2a00:1450:4025:c03::8a
Jan 30 12:58:41 nixos-dell 56a5b9b610ef[26743]:
Jan 30 12:58:41 nixos-dell systemd[1]: docker-56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f.scope: Deactivated successfully.
Jan 30 12:58:41 nixos-dell systemd[1]: docker-56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f.scope: Consumed 42ms CPU time, received 320B IP traffic, sent 112B IP traffic.
Jan 30 12:58:42 nixos-dell dockerd[26758]: time="2023-01-30T12:58:42.037484587+01:00" level=info msg="shim disconnected" id=56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f
Jan 30 12:58:42 nixos-dell dockerd[26743]: time="2023-01-30T12:58:42.037533384+01:00" level=info msg="ignoring event" container=56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 30 12:58:42 nixos-dell dockerd[26758]: time="2023-01-30T12:58:42.037655639+01:00" level=warning msg="cleaning up after shim disconnected" id=56a5b9b610ef4ccb09473957f6dc2759ddbc41cb08845e022f5e68ece118e51f namespace=moby
Jan 30 12:58:42 nixos-dell dockerd[26758]: time="2023-01-30T12:58:42.037707517+01:00" level=info msg="cleaning up dead shim"
Jan 30 12:58:42 nixos-dell dockerd[26758]: time="2023-01-30T12:58:42.068650680+01:00" level=warning msg="cleanup warnings time=\"2023-01-30T12:58:42+01:00\" level=info msg=\"starting signal loop\" namespace=moby pid=32024 runtime=io.containerd.runc.v2\n"
Jan 30 12:58:42 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered disabled state
Jan 30 12:58:42 nixos-dell kernel: veth2cca38e: renamed from eth0
Jan 30 12:58:42 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered disabled state
Jan 30 12:58:42 nixos-dell kernel: device veth12fd5b2 left promiscuous mode
Jan 30 12:58:42 nixos-dell kernel: docker0: port 1(veth12fd5b2) entered disabled state
Jan 30 12:58:42 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such file or directory
Jan 30 12:58:42 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such file or directory
Jan 30 12:58:42 nixos-dell dhcpcd[1125]: ps_root_dispatch: No such process
Jan 30 12:58:42 nixos-dell systemd[1]: var-lib-docker-overlay2-f4ebb452dd5d12a24be9e79707dfe573a7121565193edf4940e5ad520fcf139a-merged.mount: Deactivated successfully.

iFreilicht · January 30, 2023, 12:57pm

Those logs don’t contain any helpful info, unfortunately. If it works on one network but not another, I would assume that whatever is managing your network connections (KDE, network-manager, etc. ?) might modify some firewall rules on one network that docker needs for containers to access the internet. You could try restarting the docker daemon after connecting to the “faulty” network to reset those rules.

stianlagstad · January 30, 2023, 12:58pm

Thanks for commenting! Any tips for specific rules I should look closer at? I did try restarting the docker service without luck.

iFreilicht · January 30, 2023, 1:06pm

I would run iptables -S and if that’s too much maybe grep -b 5 docker0 the output (maybe higher -b parameter if you can’t see the legends of the table) when connected to either network and see if you can spot any differences. There should be some FORWARD rule related to docker, that would be what allows containers to access the internet.

stianlagstad · February 1, 2023, 6:30pm

Thanks for the tips! I need to gather some motivation beforing diving into this rabbit hole again, but will report back eventually.

dorvo · July 17, 2023, 10:45am

Hey, were you able to fix @stianlagstad , i having the same issue, i had a change on my cabled network and i had to enable resolve on my nix config and after i rebuild a docker i couldn’'t connect