Setup networking between multiple VMs

Help
1

I use nixos and am running multiple virtual machines (nixos). I want to create a virtual switch in a ipv4 subnet like 192.168.2.0/24. The setup should have the following properties:

  • every VM schould be able to have an network interface connected to that subnet (if it contains the correct configuration)
  • the host should be connected to that subnet
  • ip routing between the interfaces on the subnet should work
  • No firewall inside of this subnet
  • The configuration should be done declratively in the nix-files

I hope that networkd can be used for setup. After many tries and failures I only guess that openvswitch and tab devices can be used for this. I also hope that I dont have to change my host configuration for every new VM that can be connected to the subnet.
Configuring connecting VMs via dhcp would be nice to have but static ip configuration of Ip-addresses is ok.

How can I achieve this ?
Is there a configuration that I can copy ?

2

After some investigation I found out myself, how to do this. (With help from Arch Linux Wiki)

Create some tap and a bridge device in the configuration.nix on the host. Maybe I could leave out some of the options because they are not necessary, but they looked plausible. Note that the bridge device is the only device that gets an IP-address.

  networking.networkmanager = {
    enable = true;
    unmanaged = [ "tap0" "br0" ];
  };

  systemd.services."systemd-networkd".environment.SYSTEMD_LOG_LEVEL = "debug";
  systemd.network = {
    enable = true;
    wait-online.enable = false;
    netdevs = {
      # Create the tap interface
      "20-tap2" = {
       enable = true;
        netdevConfig = {
          Kind = "tap";
          Name = "tap2";
        };
      };
      "20-tap3" = {
       enable = true;
        netdevConfig = {
          Kind = "tap";
          Name = "tap3";
        };
      };
      "20-bridge0" = {
        enable = true;
        netdevConfig = {
          Kind = "bridge";
          Name = "br0";
        };
      };
    };
    networks = {
      "30-enp5s0" = {
        matchConfig.Name ="enp5s0";
        linkConfig = {
          Unmanaged = "yes";
        };
      };
      "40-tap2" = {
        matchConfig.Name ="tap2";
        bridgeConfig = {   };
        linkConfig = {
          ActivationPolicy = "always-up";
          RequiredForOnline = "no";
        };
        networkConfig = {
          Bridge = "br0";
        };
      };
      "40-tap3" = {
        matchConfig.Name ="tap3";
        bridgeConfig = {   };
        linkConfig = {
          ActivationPolicy = "always-up";
          RequiredForOnline = "no";
        };
        networkConfig = {
          Bridge = "br0";
        };
      };
      "40-bridge0" = {
        matchConfig.Name = "br0";
        linkConfig = {
          ActivationPolicy = "always-up";
          RequiredForOnline = "no";
        };
        networkConfig = {
          Address = ["192.168.100.1/24"];
          # Bridge = "br0";
        };
      };
    };
  };

The vm is setup like in NixOS virtual machines — nix.dev documentation. It has the following additions in the config.nix file.

{ config, pkgs, ... }:
let 
  # I guess that every vm needs a unique MAC address.
  tap_macaddress = "52:54:98:76:54:02";
  # Every vm needs a unique IP address.
  tap_network_addr = "192.168.100.2/24";
  # Every tap device on the host can only be connected to one vm.
  tap_if_name = "tap2";
  # I dont know if this is needed.
  tap_bridge_name = "br0";
in
{
  networking.firewall.enable = false;
  networking.networkmanager.enable = false;
  networking.hostName = "vm-test-tapdev";
  networking.useDHCP = false;
  systemd.network = {
    enable = true;
    networks."30-ethernet-dummy" = {
      enable = true;
      matchConfig.MACAddress = tap_macaddress;
      linkConfig = {
        # or "routable" with IP addresses configured
        ActivationPolicy = "always-up";
        RequiredForOnline = "yes";
      };
      networkConfig = {
        Address = [ tap_network_addr ];
        # Bridge = "br0";
      };
    };
  };
  virtualisation = {
    qemu = {
      networkingOptions = [  "-netdev tap,id=nd0,ifname=${tap_if_name},script=no,downscript=no,br=${tap_bridge_name}"  "-device virtio-net-pci,netdev=nd0,mac=${tap_macaddress}" ];
    };
  };

I don’t know how to provide IP-addresses by dhcp, but because individual configuration (like MAC-address, tap device,…) is needed in any case this is not a problem.
I don’t know how to avoid to create the default eth0 device, but it is not up and does not interfere.
I don’t know how to handle IP-V6 addresses, I don’t know why the tap device in the host and the tap device in the guest are different if you run “ip a”, if you know this you can write a reply.

3

The NixOS tests are always useful as inspiration for these kind of things, e.g. for DHCP for your virtual machines:

Edit: VLANs can be an alternative to creating a TAP device on the host:
https://search.nixos.org/options?channel=23.11&show=virtualisation.vlans
but that might conflict with the host being part of the subset.

Yet another possibility is setting up sth. like ZeroTier, in case you later want to migrate VMs to different hosts while keeping the logical addressing.

4

Thank you for the info. But I am still new to nixos. The example from nixos tests doesn’t help me, i have no clue what nodes, router and client are. I will stick with my solution for now.

5

I have the exact setup you desire with GitHub - astro/microvm.nix: NixOS MicroVMs - the networkd configuration is documented at A simple network setup - microvm.nix

1 Like
6

This is a great documentation, thank you.