42jd
November 3, 2021, 12:55am
1
Hi! I am working on setting up an etcd cluster with nix containers along the lines of this config . However, I am having issues connecting the containers to each other. I already encountered the issue with networking not being setup until after container started (see here ) using the exec
solution for etcd.
What is happening is the containers can connect to host but not each other. Eg 2 containers:
Host: ping c1
and ping c2
successful.
C1: ping host
successful, ping c2
fails
C2: ping host
successful, ping c1
fails
I am using extra-containers so I can get the declarative options imperatively.
1 Like
Can you post the config you are using? Or a minimal example?
42jd
November 3, 2021, 5:35pm
3
Here is the config I use (feed it into extra-containers
). I set the numNodes to X and it starts X containers.
{ lib, ... }:
let
numNodes = 2;
addressMap = genHosts numNodes;
genHosts = with builtins; num: listToAttrs (map numToHost (genList (x: x + 1) num));
numToHost = with builtins; num: {
name = "etcdN${toString num}";
value = { localAddress = "10.233.0.10${toString num}"; hostAddress = "10.233.1.10${toString num}"; };
};
toHostsEntry = name: { localAddress, ... }: "${localAddress} ${name}";
extraHosts = with builtins; concatStringsSep "\\n" (lib.attrsets.mapAttrsToList toHostsEntry addressMap);
nodeConfig = hostName: { localAddress, hostAddress }: {
inherit localAddress hostAddress;
ephemeral = true;
privateNetwork = true;
config = { config, pkgs, ... }: {
networking = {
inherit hostName extraHosts;
firewall.allowedTCPPorts = [ 2379 2380 ];
};
# etcd nixos-container: https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438
# etcd security settings https://etcd.io/docs/v3.2/op-guide/security/
# Static method for clusters: https://etcd.io/docs/v3.4/op-guide/clustering/#static
services.etcd =
let
peerUrl = "http://${localAddress}:2380";
clientUrl = "http://${localAddress}:2379";
toClusterEntry = name: { localAddress, ... }: "${name}=http://${localAddress}:2380";
in
{
enable = true;
name = hostName;
## Checks all incoming HTTPS requests for a client certificate signed by the trusted CA
#clientCertAuth = true;
## Checks all incoming peer requests for valid client certificate signed by supplied CA
#peerClientCertAuth = true;
advertiseClientUrls = [ clientUrl ];
listenClientUrls = [ clientUrl "http://127.0.0.1:2379" ];
initialAdvertisePeerUrls = [ peerUrl ];
listenPeerUrls = [ peerUrl ];
# Specify the cluster token so no unintended cross-cluster interaction
initialClusterToken = "etcd-cluster";
# Tell each cluster the advertised peer url of other etcd nodes (static method)
initialCluster = lib.attrsets.mapAttrsToList toClusterEntry addressMap;
initialClusterState = "new";
};
# https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438/2
systemd.services.etcd.serviceConfig.Type = lib.mkForce "exec";
};
};
in
{
containers = builtins.mapAttrs nodeConfig addressMap;
}
Hello, @42jd have you got your way around this?
42jd
November 22, 2022, 5:57am
6
I did not find a solution at the time and haven’t tried again since.