So, I’ve got a NixOS server over at Packet. In my server I’ve got virtualization (i.e. docker containers) enabled. Sometimes I’ll run a VirtualBox VM up there briefly for friends.
Recently, every few upgrades the network conection goes down. Rebooting the server works but that’s not something I’d like to adopt as a “fix”. Today I sat down and decided to hack at it to see what, precisely, the problem was.
My packet server has a layer 3 bond interface. They appear, to udev, as two separate interfaces – enp0s20f0 and enp0s20f1. There are systemd services for bond0 as well, though I can’t see that in ip link show. My server is currently in this “down” state at the time of writing: it can’t see the outside internet (name or service not known) and the outside internet can’t see it (timeout). When I attempt to bring enp0s20f0 (the main interface) up (enp0s20f1 is up already) I get RTNETLINK answers: No such device. This is strange because ip link show shows all interfaces:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s20f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:b5:85:c6 brd ff:ff:ff:ff:ff:ff
3: enp0s20f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:b5:85:c7 brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 56:ab:99:8d:df:d7 brd ff:ff:ff:ff:ff:ff
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:e9:e1:e9 brd ff:ff:ff:ff:ff:ff
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:94:f1:fe:a6 brd ff:ff:ff:ff:ff:ff
The other weird thing is that this “blackout” problem happens wheever I instruct the virtualbox configuration to add the VBox network interfaces. Every time an interface is added, removed, or changed, the server loses network connectivity. What can I do to bring the interfaces back online and how can I permanently prevent it in future?