Connecting NixOS containers

Hi! I am working on setting up an etcd cluster with nix containers along the lines of this config. However, I am having issues connecting the containers to each other. I already encountered the issue with networking not being setup until after container started (see here) using the exec solution for etcd.

What is happening is the containers can connect to host but not each other. Eg 2 containers:

Host: ping c1 and ping c2 successful.
C1: ping host successful, ping c2 fails
C2: ping host successful, ping c1 fails

I am using extra-containers so I can get the declarative options imperatively.

1 Like

Can you post the config you are using? Or a minimal example?

Here is the config I use (feed it into extra-containers). I set the numNodes to X and it starts X containers.

{ lib, ... }:
let
  numNodes = 2;
  addressMap = genHosts numNodes;

  genHosts = with builtins; num: listToAttrs (map numToHost (genList (x: x + 1) num));
  numToHost = with builtins; num: {
    name = "etcdN${toString num}";
    value = { localAddress = "10.233.0.10${toString num}"; hostAddress = "10.233.1.10${toString num}"; };
  };

  toHostsEntry = name: { localAddress, ... }: "${localAddress} ${name}";
  extraHosts = with builtins; concatStringsSep "\\n" (lib.attrsets.mapAttrsToList toHostsEntry addressMap);

  nodeConfig = hostName: { localAddress, hostAddress }: {
    inherit localAddress hostAddress;

    ephemeral = true;
    privateNetwork = true;


    config = { config, pkgs, ... }: {
      networking = {
        inherit hostName extraHosts;

        firewall.allowedTCPPorts = [ 2379 2380 ];
      };

      # etcd nixos-container: https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438
      # etcd security settings https://etcd.io/docs/v3.2/op-guide/security/
      # Static method for clusters: https://etcd.io/docs/v3.4/op-guide/clustering/#static
      services.etcd =
        let
          peerUrl = "http://${localAddress}:2380";
          clientUrl = "http://${localAddress}:2379";
          toClusterEntry = name: { localAddress, ... }: "${name}=http://${localAddress}:2380";
        in
        {
          enable = true;
          name = hostName;

          ## Checks all incoming HTTPS requests for a client certificate signed by the trusted CA
          #clientCertAuth = true;

          ## Checks all incoming peer requests for valid client certificate signed by supplied CA
          #peerClientCertAuth = true;


          advertiseClientUrls = [ clientUrl ];
          listenClientUrls = [ clientUrl "http://127.0.0.1:2379" ];

          initialAdvertisePeerUrls = [ peerUrl ];
          listenPeerUrls = [ peerUrl ];


          # Specify the cluster token so no unintended cross-cluster interaction
          initialClusterToken = "etcd-cluster";

          # Tell each cluster the advertised peer url of other etcd nodes (static method)
          initialCluster = lib.attrsets.mapAttrsToList toClusterEntry addressMap;
          initialClusterState = "new";
        };

      # https://discourse.nixos.org/t/issues-using-nixos-container-to-set-up-an-etcd-cluster/8438/2
      systemd.services.etcd.serviceConfig.Type = lib.mkForce "exec";
    };
  };
in
{
  containers = builtins.mapAttrs nodeConfig addressMap;
}

Hello, @42jd have you got your way around this?

I did not find a solution at the time and haven’t tried again since.