NixOS container networking and Zyre

Development
1

I made a package for zyre and wanted to play around with it inside nixos containers.

zyre.nix 
with import <nixpkgs> { };

stdenv.mkDerivation rec {
  pname = "zyre";
  version = "2.0.0";

  src = fetchFromGitHub {
    owner = "zeromq";
    repo = "zyre";
    rev = "latest_release";
    sha256 = "1iwg0yqnrn8njjqv2ycifygzxrwd5g5kaszln0hy05m1c0xi44ik";
  };

  nativeBuildInputs = [ cmake asciidoc libsodium zeromq czmq ];

  enableParallelBuilding = true;

  doCheck = false; # fails all the tests (ctest)

  meta = with stdenv.lib; {
    branch = "latest_release";
    homepage = https://github.com/zeromq/zyre;
    description = "A Framework for Distributed Computing";
    license = licenses.mpl20;
    platforms = platforms.all;
  };
}

Zyre comes with a utility called “zping” which can be used for testing it’s functionality.
Opening two terminals on my machine and executing it looks something like this:

term1$ zpinger
I: 20-02-14 16:27:49 Create Zyre node, uuid=9714E9D4DCC2A9400BED4C99C230CE9C, name=9714E9
I: 20-02-14 16:27:51 [E3E3FD] peer entered
I: 20-02-14 16:27:51 [E3E3FD] received ping (WHISPER)
I: 20-02-14 16:27:51 [E3E3FD](GLOBAL) received ping (SHOUT)

term2$ zpinger
I: 20-02-14 16:27:51 Create Zyre node, uuid=E3E3FDBF0CB48F3238192F48548FF0E6, name=E3E3FD
I: 20-02-14 16:27:51 [9714E9] peer entered
I: 20-02-14 16:27:51 [9714E9] received ping (WHISPER)
I: 20-02-14 16:27:51 [9714E9](GLOBAL) received ping (SHOUT)

Doing the same inside a nix container shows that it does not work there…
Here is a stripped down (untested) version of my container:

container.nix 
{
  containers.host1 =
  {
    privateNetwork = true;
    hostAddress = "192.168.100.1";
    localAddress = "192.168.100.10";

    config = {
      networking.firewall.enable = false;
      environment.systemPackages = with pkgs; [
        (import zyre.nix)
      ];
    };
  };
}

AFAIK zyre relies on UDP multicast which appears to be enabled inside the container.

[root@host1:~]# ip a
2: eth0@if78: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:a8:da:04:53:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.100.10/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::54a8:daff:fe04:5386/64 scope link 
       valid_lft forever preferred_lft forever

UDP itself appears to be working inside (and between) containers which I tested with netcat like so:

[root@host1:~]# nc -l -u 1234

[root@host1:~]# nc -v -u 192.168.100.10 1234
Connection to 192.168.100.10 1234 port [udp/search-agent] succeeded!

Obviously the problem must be related to container networking but I am relatively new to NixOS and
out of ideas.
Is there anybody who can give some insight on how container networking works in regard to this issue and/or reproduce this behaviour?

2

I wouldn’t be so sure about nc -v -u test. I have run your command on my machine and it also succeeds:

> nc -v -u 192.168.100.10 1234
Connection to 192.168.100.10 1234 port [udp/search-agent] succeeded!

because UDP doesn’t need confirmation that packet was received by other side.

Can you ping containers each other? what are IPs? You may also try bridged network, I had better experience with bridged.

3

i think you are right with netcat not being a reliable test for udp connectivity. pinging between the containers worked and i could send strings via udp (using netcat again).

i think the network config on my host was broken and i am right now in the process of doing a proper brige setup as you suggested. i will update this thread as soon as i have got the details figured out.

4

It is working as expected now.
Here is a working example with static ip configuration and the appropriate firewall rules:

host-network.nix 
  networking.firewall.enable = true;
  networking.firewall.allowedTCPPorts = [ 49152 ]; # zpinger
  networking.firewall.allowedUDPPorts = [ 5670 ]; # zyre

  networking.bridges = { br0 = { interfaces = [ "enp39s0" ]; }; };
  
  networking.interfaces.br0.ipv4.addresses = [ {
    address = "192.168.100.1";
    prefixLength = 24;
  } ];
container.nix 
{ config, lib, pkgs, ... }:
{
  containers.host1 = {
    autoStart = true;
    privateNetwork = true;
    hostBridge = "br0";
    hostAddress = "192.168.100.1";
    localAddress = "192.168.100.11/24";

    config = {
      networking.firewall.enable = false;
      environment.systemPackages = with pkgs; [
        (import zyre.nix)
      ];
    };
  };
}

the output of the zpinger utility should look something like this:

[root@adelskronen:~]# zpinger 
I: 20-02-15 15:29:11 Create Zyre node, uuid=1A7CCEBC33EEE3870A04B280B469FA20, name=1A7CCE
I: 20-02-15 15:29:13 [AFAF3E] peer entered
I: 20-02-15 15:29:13 [AFAF3E] received ping (WHISPER)
I: 20-02-15 15:29:13 [AFAF3E](GLOBAL) received ping (SHOUT)

[root@adelskronen:~]# nixos-container run host1 zpinger
I: 20-02-15 14:29:13 Create Zyre node, uuid=AFAF3EA667A99FE025C2068CEA7A83CE, name=AFAF3E
I: 20-02-15 14:29:13 [1A7CCE] peer entered
I: 20-02-15 14:29:13 [1A7CCE] received ping (WHISPER)
I: 20-02-15 14:29:13 [1A7CCE](GLOBAL) received ping (SHOUT)

it also works in between containers.

1 Like