How to connect two or more nixos-containers together (their internet ports)

Help
1

Background

I am setting up a server. The design of it is this (also here: https://codeberg.org/BlastboomStrice/SelfHostedPlan/) :
drawio scheme
I have my configuration.nix under the nixos-base host here: https://codeberg.org/BlastboomStrice/dotfiles/src/branch/main/.config/nixos-config/hosts/nixos-base

I am using nixos-containers (which are systemd-nspawn containers essentially). Some of them, especially the *arr stack, need to be interconnected. Lidarr needs to talk to prowlar, qbittorrent and jellyfin for example. I am using the option privateNetwork = true; which, as per its description, gives the container its own private virtual Ethernet interface, the interface is called eth0, and is hooked up to the interface ve-«container-name» on the host.
Here is a test container module I’ve made: https://codeberg.org/BlastboomStrice/dotfiles/src/branch/main/.config/nixos-config/modules/nixos/containers/test-container.nix

This helps a bit with my set up, because each container gets its own hostname and it can exist as a seperate machine on tailscale. Thus, each container has its own address in my tailnet. (Tailscale, unlike headscale, does not let you create subdomains and the new feature “services” doesn’t really help me.) This isn’t an issue, but I’m just mentioning it.

Also, as per the wiki https://wiki.nixos.org/wiki/NixOS_Containers, I’ve enabled nat to let each container have access to the internet:

networking.nat = {
  enable = true;
  # Use "ve-*" when using nftables instead of iptables
  internalInterfaces = [ "ve-+" ];
  externalInterface = "wlp5s0"; # my wifi interface
  # Lazy IPv6 connectivity for the container
  enableIPv6 = true;
};

I use wifi btw (not ethernet).

In my configuration.nix I use this option

networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.

Also this may be useful https://www.calculator.net/ip-subnet-calculator.html

Issue

What I can’t figure out is how to connect two or more containers together (not with the host, the host can already access them). I want the container A to be able to ping the container B. I don’t want any container to be able to ping container B, only A. I may also want container C ping D. I want to keep the privateNetwork = true; option and I don’t want to make so many bypasses that will render it useless.

What the docs say

The wiki https://wiki.nixos.org/wiki/NixOS_Containers has a bridge section, which from what I understand, it bridges the container with the host internet interface. It’s confusing me a lot as not only I’m not sure whether it is what I need, but also it has options that are not very self-explanatory. I copied the snipped and added some comments with questions:

networking = {
  # what do I put here? any interface that I want to be bridged? if so, why does it only have one interface "eth0s31f6"?
  # two containers may have interfaces starting with "ve-", can I put these interfaces? (probably no)
  # does it create a new interface named "br0"? (probably yes)
  bridges.br0.interfaces = [ "eth0s31f6" ]; # Adjust interface accordingly
  
  # Get bridge-ip with DHCP
  # why do we deactivate that? doesnt that affect every thing on the machine?
  useDHCP = false;
  # why do we activate that?
  interfaces."br0".useDHCP = true;

  # Set bridge-ip static
  interfaces."br0".ipv4.addresses = [{
    # where does this point to?
    # I suppose it creates a vlan with the address below and a netmask of prefixLength
    address = "192.168.100.3";
    # whats the length for? I just want to bridge 2 IPs together, could I use 31 bits?
    prefixLength = 24;
  }];
  # what is this about?
  # in my containers I set hostAddress to "192.168.100.10" etc.
  defaultGateway = "192.168.100.1";
  # is that the Broadcast Address?
  # do I need that? what do I do with it? what is it??
  nameservers = [ "192.168.100.1" ];
};

containers.<name> = {
  privateNetwork = true;
  # I suppose this connects the container to the bridge
  # hostAdress is not used when you use hostBridge from what the option description says
  hostBridge = "br0"; # Specify the bridge name
  # the option description says to use the /netmask when you use the hostBridge option
  localAddress = "192.168.100.5/24";
  config = { };
};

As you can see it has confused me a lot:/ To add to this, the description of the option networking.interfaces says “Please note that systemd.network.netdevs has more features and is better maintained. When building new things, it is advised to use that instead.” which makes me think that the whole snippet may not be the best idea. But setting up systemd.network seems to be another huge thing to do (https://budimanjojo.com/2025/03/06/systemd-networkd-nixos-pain/?).

Also here https://forum.manjaro.org/t/setting-up-a-bridge-with-nmcli/186663/11, they say in a related (I think) attempt:

True bridging happens at Layer 2, and standard wifi client mode generally doesn’t support that.
Does that mean my whole set up won’t work?

https://microvm-nix.github.io/microvm.nix/advanced-network.html this here is just too confusing for me and I don’t know if it fits my case

https://www.geeksforgeeks.org/linux-unix/disabling-and-enabling-an-interface-on-linux-system/ this tells me that I probably can’t create a bridge when the interface I connect each container is disabled, so I guess I can’t use the eth interface.
https://unix.stackexchange.com/questions/363332/how-do-i-configure-a-network-interface-bridge-from-wifi-to-ethernet-with-debian I suppose I could bridge the ethernet and the wifi interfaces as well, but at this point I think that’s not needed.

https://oneuptime.com/blog/post/2026-03-04-configure-networking-for-systemd-nspawn-containers-on-rhel/view this gives me slop vibes, but if this snippet is correct it may point a serious issue in my config:

[Network]
# Use private veth
VirtualEthernet=yes
# Or use a bridge
# Bridge=br0

Can I use a bridge if I use privateNetwork = true;??

Other stuff I found that may be related:

Things I’ve tried

I created 2 test containers to play around, lets call them containerA and containerB.

I tested by pinging the localAddress of containerA from containerB (which probably wouldn’t work as they are private networks?) and also I think I tried pinging the open port on the hostAddress or bridge of containerA from containerB. Many times the rebuilding failed before I even got to test the setup.

I tried the following:

# configuration.nix
networking = {
  bridges.br0.interfaces = [ "wlp5s0" ]; # Also tried [ "enp4s0" ], [ "ve-containerA" "ve-containerB" ] and [ "wlp5s0" "ve-containerA" "ve-containerB" ]
  
  # Get bridge-ip with DHCP
  useDHCP = false; # also tried it with this commented out
  interfaces."br0".useDHCP = true;

  # Set bridge-ip static
  interfaces."br0".ipv4.addresses = [{
    address = "192.168.100.10";
    prefixLength = 24; # also tried 30
  }];
  defaultGateway = "192.168.100.1"; # also tried it with this commented out
  nameservers = [ "192.168.100.1" ]; # also tried it with this commented out
};

# containerA
privateNetwork = true;
localAddress = "192.168.100.11";
hostBridge = "br0"

# containerB
privateNetwork = true;
localAddress = "192.168.100.12";
hostBridge = "br0"

I tried skipping the bridge and simply using the same hostAddress in both containers

# containerA
privateNetwork = true;
hostAddress = "192.168.100.10";
localAddress = "192.168.100.11";

# containerB
privateNetwork = true;
hostAddress = "192.168.100.10";
localAddress = "192.168.100.12";

I tried using containers.<name>.interfaces, but the way I did it it failed badly

# containerA
privateNetwork = true;
hostAddress = "192.168.100.10";
localAddress = "192.168.100.11";
interfaces = "ve-containerB";

# containerB
privateNetwork = true;
hostAddress = "192.168.100.12";
localAddress = "192.168.100.13";
interfaces = "ve-containerA";

Some other stuff that may or may not work is using the following options:

containers.<name>.extraVeths
networking.ipvlans
networking.wlanInterfaces

I thinking that maybe I should somehow create an interface on the host and somehow bind/bridge the container interfaces (of the containers I want to communicate with each other) to that. Maybe containers.<name>.extraVeths would be helpful?

Other solutions

Again, we assume I created 2 test containers to play around, lets call them containerA and containerB. I don’t like any of these solutions as they aren’t real solutions, they simply bypass the issue.

We assume:

# containerA
privateNetwork = true;
hostAddress = "192.168.100.10";
localAddress = "192.168.100.11";

# containerB
privateNetwork = true;
hostAddress = "192.168.100.12";
localAddress = "192.168.100.13";
  • ContainerA listens on port 8000
  • ContainerB listens on port 8000
  1. Tailscale
    Since I have tailscale set up already, my containers can work as if they weren’t on the same machine. I don’t like this, as it adds an extra point of failure (tailscale) and would be very problematic if I needed low latency or fast speeds. Steps to do this:
  2. Many proxies (Two sounds as Too, makes a good pun)
    This is very messy and essentially uses the host instead of tailscale servers. I do 4 proxyings per 2 containers…
  3. Single container
    Another option is to put everything that needs access to another container into a single big container. While this is simple, it has some extra issues:
    • It reduces the isolation
    • Since tailscale doesn’t support subdomains, I won’t be able to use subdomains and I will have to use subpaths (handles in caddy terms) or ports, if I want to have each service available to other other devices (so that I can control lidarr from my laptop for example)
    • I want to have qbittorrent in a seperate container which will be behind a vpn. This solution would force me to somehow redirect only the traffic of qbittorrent throught the vpn and leave the other services intact.
2

I haven’t looked into your plan deep enough, so excuse me if some questions are already answered.

Do you want to use nixos containers as virtual machine replacement to install services in them? Or do you plan to use existing docker images for each of your services?

The easiest solution (to start with) would be using docker compose. If your containers need dedicated IPs in your host-network, use docker with macvlan networks. Put a reverse proxy (I’d recommend traefik) in front of your services, it can provide lets encrypt certificates and doesn’t require routing in tailscale. (Recommendation: Create a “core” deployment for traefik with 2 networks, one “public” network reachable from tailscale, one “shared” network. Create additional compose deployments for each “topic” with 2 networks, one internal network for your services to talk to each other and the shared network (import, not redefine) for the containers requiring public access. Traefik can configure the routes automatically, based on container labels. It’s the best option you can get without using kubernetes.)

And if that doesn’t satisfy you in the end, you can move the existing setup to any other solution step by step.

1 Like
3

Do you want to use nixos containers as virtual machine replacement to install services in them?

Yes, I’m trying to avoid docker and all these.

I have a feeling this throws your recommendation a bit off://

(Also about trarfik, I used to use it cuz it auto-fetched/renew certificates from tailscale, but my config stopped working and I realised caddy daus the autostuff too, with even less setup:)

4

Out of curiosity: Why do you try to avoid docker?

I don’t force anyone to follow my recommendations :smiley:

5

Oh, well for 2 main reasons:

  1. I think nixos-containers are more “native” (which means fewer resources used and possibly I won’t have to learn lots of new stuff)
  2. Docker has a weird liscense that only allows personal use for free. There is podman that fixes that, but that’s another whole thing to learn and deal with and eh
6

I think you might be a bit off on both reasons.

I’m not sure if nixos containers spawn systemd - that would cause more overhead than a normal container, which simply runs processes in separate namespaces. No kernel, no init-system, no supervisor. Just clean privilege separation provided by the underlying linux kernel.

The Docker engine itself is opensource (Apache 2.0) and can be used privately and commercially. The only limited component is docker desktop - on windows and osx. It’s an optional gui - console tools are enough to do anything you want.

Podman and docker use the same mechanics for running containers. They are interchangeable.

I’d definitely recommend running simple application containers over any other solution. It’s not even really complicated once you got the mechanics. Your CV would also profit immensely - if you’re working in IT (or plan to) since basically every modern (or “cloud”) company uses them.

1 Like
7

Quadlets (podman’s containers as systemd services) are actually really nice, although for better integration you need a third-party flake (either mirkolenz/quadlet-nix or SEIAROTg/quadlet-nix).

Native NixOS containers seem neat as well, though, although I have not personally used them.

1 Like
8

Thanks for your suggestions, but I think I will keep the nixos-containers for now. Also I’d like the solution to the issue (if a solution is found) to be used to update the nixos-containers wiki, as some people may benefit from it.

2 Likes
9

I configured all my containers with self defined bridges using systemd.network.netdevs.

{
  config,
  lib,
  pkgs,
  ...
}:

{
  systemd.network = {
    enable = true;

    config = {
      routeTables = {
        bridge = 259;
      };
    };

    netdevs = {
      "10-br0" = {
        netdevConfig = {
          Kind = "bridge";
          Name = "br0";
          MACAddress = "10:00:00:00:00:01";
        };
      };
    };

    networks = {
      "25-br0" = {
        matchConfig.Name = "br0";
        address = [
          "10.0.0.1/24"
        ];

        routingPolicyRules = [
           # custom rules
        ];

        routes = [
          {
            Gateway = "0.0.0.0";
            Table = "bridge";
          }
          {
            Gateway = "::";
            Table = "bridge";
          }
        ];
      };
    };
  };
}

A container setup could look like this:

  containers.example = {
    privateNetwork = true;
    hostBridge = "br0";
    localAddress = "10.0.0.2/24";
    ...
    config =
      {
        config,
        pkgs,
        lib,
        ...
      }:
      {
        networking = {
          defaultGateway = {
            address = "10.0.0.1";
          };
        };
        ...
      }
  }

Then you can set up a route table on you host and route traffic between them.

3 Likes
10

Woah
It’s getting late for me today, so I can’t experiment with this now, but I will try tomorrow (and I may have lots of questions regarding the options used). Looks nice

11

I just don’t think there is any way for you to avoid routing the traffic via your host, I guess you don’t even need the bridge, when you have the ve’s but it is definitely easier to understand in my opinion.

Just route the traffic where you want it. (I think the routing table is enough if you just want all containers on the same bridge to be able to talk to each other)

12

I just don’t think there is any way for you to avoid routing the traffic via your host

It’s ok, would be cooler if I could skip the host, but not that big of an issue. But I couldn’t figure out how to do it even if I routed traffic through the host:/

I guess you don’t even need the bridge, when you have the ve’s

How could I do that? I have the ve-'s, but only the host knows these. Is there a way to avoid a bridge? Is it systemd.network specific?

13

Well I never tried and probably never will, also because that will become quite hacky pretty quickly I guess, something like this

{
  boot.kernel.sysctl = {
    "net.ipv4.ip_forward" = 1;
  };

  networking.nftables.enable = true;

  networking.firewall = {
    enable = true;
    filterForward = true;

    extraForwardRules = ''
      iifname "ve-container1" oifname "ve-container2" ip daddr 10.0.2.2 tcp dport 5432 accept # container2's address is 10.0.2.2
    '';
  };
}

Also none of this is systemd.network specific

1 Like
14

You need a macvlan interface. That would avoid the host, since it gets its own mac-address.

1 Like
15

There is really no point in avoiding the host in this scenario.

16

Indeed that looks veery hacky haha, I see

I will try doing the systemd.network bridges (and learn systemd.network I guess) tomorrow

17

Ohh I see I see, good to know.

18

You don’t have to define the bridge using systemd, but you need to figure out the routing that you want. You might still need to add forwarding if you want them to reach the internet.

2 Likes
19

To be clear, the reason this is complex is because the nixos containers suite doesn’t come with a pre-defined networking thing.

Docker and podman explicitly do, and are therefore a bit simpler to set up. Networking is quite complex, so that can be convenient, but they’re ultimately still routing everything through the host. You can define the same thing by hand.

On the flip side, docker is also infamously coupled to iptables for this, preventing the adoption of nftables. If you grok networking, you can probably do a better job than docker, and leave your host’s networking under your (and NixOS’) control. Plus you avoid OCI containers so you can stick to one software distribution method.

But yes, you’ll need to learn networking, alongside the whole shebang of Linux’ networking features. It’s good guru meditation.

1 Like
20

I don’t have an example for connection nixos-containers at hand, but the example given for IfState in the Wiki might give you a start: https://wiki.nixos.org/wiki/IfState

1 Like