Issues using nixos-container to set up an etcd cluster

Hi everyone!

I’m currently running into a small issue using nixos-container.
I’m testing out an etcd cluster using nixos-container, and I have a mostly-working configuration:

However, when I import this and run nixos-rebuild switch, starting the containers take a long time (I was forced to set timeoutStartSec = "5min"; to work around this). The reason for this was that etcd runs into what seems to be networking issues at startup (e.g. health check for peer 84d1d2056184ffa could not connect: dial tcp 10.233.0.104:2380: connect: no route to host (prober "ROUND_TRIPPER_SNAPSHOT")), and dies after a certain number of retries (at which point nixos-rebuild switch recognizes that the containers have finished starting). What’s curious about this is that (simultaneously) restarting etcd on all containers will get etcd working just fine.

My question is, is there some way to get etcd to work at container startup without needing to manually restart etcd on all containers? Is this expected behavior or an issue with nixos-container, the etcd module, or my configuration?

Here’s the full journal: etcd-full.log · GitHub
Here’s the log for the etcd service in particular: etcd-specific.log · GitHub

Yeah this is actually a known issue with NixOS containers.
During a container’s startup the container-side network (using veth-pairs) is configured and network-online.target is also reached (which is why systemd thinks it has a working connection), however the uplink is broken until the container is fully started up since the host-side interface will be configured in the ExecStartPost-hook of the corresponding systemd-unit:

I stumbled upon this several times in the past[1][2] and unless you do some ugly workarounds (like delaying the startup of etcd inside the unit until you have working networking), you probably want to use something else than nixos-containers.

About a year ago I tried to investigate this issue more thorougly and realized that this can only be fixed for good when using systemd-networkd(8). An issue has been opened for that[3]. However this will cause a bigger impact which is why I decided to write an RFC first (I’m already on it, but it isn’t published yet).

[1] Services depending on `keys.target` can cause hanging boots on NixOS containers · Issue #67265 · NixOS/nixpkgs · GitHub
[2] nixos/acme: don't depend on multi-user.target inside a container by Ma27 · Pull Request #83704 · NixOS/nixpkgs · GitHub / nixos/acme: renew after rebuild and on boot by mweinelt · Pull Request #81371 · NixOS/nixpkgs · GitHub
[3] Implement NixOS container networking with networkd · Issue #69414 · NixOS/nixpkgs · GitHub

2 Likes

Thank you so much for the quick & detailed answer! Based on the issue you’ve described, I’ve managed to find an “ugly workaround” for etcd to get it to work:

# The etcd service is of type "notify", which means that
# etcd would not be considered started until etcd is fully online;
# however, since networking only works sometime *after*
# multi-user.target, we forgo etcd's notification entirely.
systemd.services.etcd.serviceConfig.Type = lib.mkForce "exec";
1 Like