[kinda solved] Combine bridges and VLANs

For simplicity, let’s say that I have one interface, eth0. I want to turn it into a bridge (for VMs, containers), and at the same time use a couple ot VLANS with it. Bridges over eth0 work fine. VLANs over eth0 also work fine. But I’m failing to combine the two.

Here’s what I got so far. In this configuration everything is created seemingly ok, but DHCP does not work. Static IP (used in my management VLAN) works fine, I can access the machines in that VLAN.

Any ideas?

  networking = {
    useNetworkd = false;
    useDHCP = false;    # off by defalut, enable per-interface
    hostName = "ago";
    # zfs needs hostId, so we derive it from hostname
    hostId = lib.mkDefault (builtins.substring 0 8 (builtins.hashString "md5" config.networking.hostName));
    firewall.enable = false; # let's not complicate things while debugging
    bridges = {
      "br0" = {
        interfaces = [ "eth0" ]; # sits on top of eth0
      };
    };

    vlans = {
      vlan30 = {
        id = 30;
        interface = "br0";
      };
      vlan99 = {
        id = 99;
        interface = "br0";
      };
    };

    interfaces = {
      eth0.useDHCP = false;  # Interface is bridged
      br0.useDHCP = true;    # Bridge gets IP via DHCP
      vlan30.useDHCP = true; # VLAN 50 gets IP via DHCP
      vlan99 = {
        ipv4.addresses = [{
          address = "10.99.99.30";
          prefixLength = 24;
        }];
      };
    };
  };

This results in:

🟢  nc -vz 10.99.99.1 82
Connection to 10.99.99.1 82 port [tcp/xfer] succeeded!

🟢  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
5: vlan30@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
6: vlan99@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet 10.99.99.30/24 scope global vlan99
       valid_lft forever preferred_lft forever

Update: it looks like dhcpcd doesn’t like that I’ve got IPV6 disabled. WAT!

Feb 07 23:57:35 ago systemd[1]: Starting DHCP Client...
Feb 07 23:57:35 ago (e-dhcpcd)[12345]: dhcpcd.service: Failed to set up mount namespacing: /proc/sys/net/ipv6: No such file or directory
Feb 07 23:57:35 ago (e-dhcpcd)[12345]: dhcpcd.service: Failed at step NAMESPACE spawning /nix/store/jm83gcsczykqcsix267lfb6l6f9d82c4-migrate-dhcpcd: No such file or>
Feb 07 23:57:35 ago systemd[1]: dhcpcd.service: Control process exited, code=exited, status=226/NAMESPACE
Feb 07 23:57:35 ago systemd[1]: dhcpcd.service: Failed with result 'exit-code'.
Feb 07 23:57:35 ago systemd[1]: Failed to start DHCP Client.
Feb 07 23:57:35 ago systemd[1]: dhcpcd.service: Scheduled restart job, restart counter is at 2.

Indeed, enabling IPv6 fixed the error - but I do not wish to have it on my systems! How to fix???

Have you tried setting useNetworkd = true;? I find it handles stuff like this better usually.

The whole idea was to avoid using networkd :slight_smile:

My previous attempt was with systemd/networkd and I failed as well.

Now this weird artefact with IPv6 (see update)!

For the record, you don’t have to configure networkd manually. useNetworkd = true; will cause the config you already have there to be translated to networkd configs. And upstream networkd is much better architected and maintained than the scripted networking we have in nixos.

1 Like

Tried that. Immediately lost the ability to set the metric (2000):

error: A definition for option `systemd.network.networks."40-eth1".routes."[definition 1-entry 1]"' is not of type `attribute set of (systemd option)'. Definition values:
       - In `/nix/store/f30jn7l0bf7a01qj029fq55i466vmnkh-source/nixos/modules/tasks/network-interfaces-systemd.nix':
           {
             Metric = "2000";

…and DHCP stopped working :man_facepalming:

Anyway, thank you for assistance, I’ll leave it for another day. At least with IPv6 enabled I managed to get it working.

Just in case anyone else encouners this, here’s my final working configuration (albeit with IPv6 which I really do not want to have enabled).

If anyone comes up with a way to convert it to networkd/systemd I’d be ecstatic!

  networking = {
    useNetworkd = false; # no need for networkmanager/systemd
    useDHCP = false;    # off by defalut, enable per-interface
    hostName = "ago";
    # zfs needs hostId, so we derive it from hostname
    hostId = lib.mkDefault (builtins.substring 0 8 (builtins.hashString "md5" config.networking.hostName));
    firewall.enable = false;
    bridges = {
      "br0" = {
        interfaces = [ "eth0" ];
        rstp = true;  # Enable rapid spanning tree protocol
      };
      "br1" = {
        interfaces = [ "eth1" ];
        rstp = true;  # Enable rapid spanning tree protocol
      };
    };

    vlans = {
      vlan30 = {
        id = 30;
        interface = "br0";
      };
      vlan40 = {
        id = 40;
        interface = "br0";
      };
      vlan99 = {
        id = 99;
        interface = "br0";
      };
    };

    interfaces = {
      eth0.useDHCP = false;  # Interface is bridged
      eth1.useDHCP = false;  # Interface is bridged

      br0.useDHCP = true;    # Bridge gets IP via DHCP
      br1.useDHCP = true;    # Bridge gets IP via DHCP

      vlan30.useDHCP = true;
      vlan40.useDHCP = true;
      vlan99 = {
        ipv4.addresses = [{
          address = "10.99.99.30";
          prefixLength = 24;
        }];
      };
    };

    # dhcpcd configuration
    dhcpcd = {
      enable = true;
      wait = "ipv4";
      extraConfig = ''
        # Configure metrics per interface
        interface br0
        metric 100    # Lower metric = higher priority

        interface br1
        metric 200    # Lower metric = higher priority

        interface vlan30
        nogateway
        metric 2000    # Higher metric = lower priority

        interface vlan40
        nogateway
        metric 2000    # Higher metric = lower priority

        interface vlan99
        nogateway
        metric 2000    # Higher metric = lower priority
      '';
    };
  };

gives:

🟢  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::215:5dff:fe0e:cb43/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
    link/ether 00:e0:4c:68:00:2e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2e0:4cff:fe68:2e/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
4: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:e0:4c:68:00:2e brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.30/24 brd 192.168.10.255 scope global dynamic noprefixroute br1
       valid_lft 42952sec preferred_lft 37552sec
    inet6 fe80::2e0:4cff:fe68:2e/64 scope link
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.39/24 brd 192.168.11.255 scope global dynamic noprefixroute br0
       valid_lft 42923sec preferred_lft 37523sec
    inet6 fe80::215:5dff:fe0e:cb43/64 scope link
       valid_lft forever preferred_lft forever
6: vlan30@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.249/24 brd 192.168.30.255 scope global dynamic noprefixroute vlan30
       valid_lft 42927sec preferred_lft 37527sec
    inet6 fe80::215:5dff:fe0e:cb43/64 scope link
       valid_lft forever preferred_lft forever
7: vlan99@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet 10.99.99.30/24 scope global vlan99
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe0e:cb43/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
8: vlan40@br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:15:5d:0e:cb:43 brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.248/24 brd 192.168.40.255 scope global dynamic noprefixroute vlan40
       valid_lft 42923sec preferred_lft 37523sec
    inet6 fe80::215:5dff:fe0e:cb43/64 scope link
       valid_lft forever preferred_lft forever
9: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 100.84.116.68/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::2001:7444/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::585b:4d2d:50ea:e920/64 scope link stable-privacy proto kernel_ll
       valid_lft forever preferred_lft forever

🟢  ip r
default via 192.168.11.1 dev br0 proto dhcp src 192.168.11.39 metric 100
default via 192.168.10.1 dev br1 proto dhcp src 192.168.10.30 metric 200
10.99.99.0/24 dev vlan99 proto kernel scope link src 10.99.99.30
192.168.10.0/24 dev br1 proto dhcp scope link src 192.168.10.30 metric 200
192.168.11.0/24 dev br0 proto dhcp scope link src 192.168.11.39 metric 100
192.168.30.0/24 dev vlan30 proto dhcp scope link src 192.168.30.249 metric 2000
192.168.40.0/24 dev vlan40 proto dhcp scope link src 192.168.40.248 metric 2000

@bogorad, I worked through a bunch of similar setups in parallel to your efforts.

I tried:

  • openvswitches
  • bridges
  • macvlan/macvtap

I built a few templating helper functions to keep systemd.network config under control. It all kinda worked and then…

I started trying to write it up to post on this thread and I realised it was madness. WET not DRY.
The hoops I was jumping through to get systemd.network to be consistent were absurd. So I switched it all off and came back to scripted networking.

If you want me to I can come back with the systemd.network stuff but, honestly, it is a beast.

The macvtap/macvlan is the solution I ended up with. The bridge needs a physical interface as it does all the tap magic in the hardware.

My use case was a lab to evaluate a virtualised firewall appliance. Physical NICs on the box are all i226v and named:

trunk0  # this is the interface being bound to the macvtap/macvlan mode="bridge"
wan1    # bound for exclusive use by macvtap mode="passthrough"
opt2
opt3
opt4
enp7s0  # fallback physical connection onto production network

Production VLANS exist on 10, 30 and 40. The evaluation VLAN is 24.

The trick to getting DHCP on the VLANed bridge was for the VM interface to get defined with trustGuestRxFilters="yes". If not, broadcast traffic on bridged VLANS is dropped.
e.g.

    <interface type="direct" trustGuestRxFilters="yes">
      <mac address="aa:bb:cc:xx:yy:zz"/>
      <source dev="trunk0" mode="bridge"/>
      <target dev="private"/>
      <target dev="firewall"/>
      <model type="virtio"/>
      ...
    </interface>

It is the combination of type="direct" and mode="bridge" that gives us macvtap in bridge mode on the VM side.

It is sensible for the host to have a static address rather than rely on the guest VM for DHCP services, however I wanted to prove out DHCP for a host accessible connection up to guest. I put the static addressing on the (production) VLAN 10 interface and used DHCP for VLAN 24. I also threw DHCP onto the native (untagged) interface just to see if that would work as well.

# networking.nix
{ config, lib, pkgs, ... }:

{
  networking = {
    # Enable networking
    hostName = "nixos";
    networkmanager.enable = false;
    useNetworkd = false;
    useDHCP = false;
    hostId = lib.mkDefault (builtins.substring 0 8 (builtins.hashString "md5" config.networking.hostName));  # thank you @bogorad
    firewall.enable = false;
    enableIPv6 = false;

    macvlans.host0 = {       #  The host interface onto the bridge gets called "host0"
      interface = "trunk0";  #  ... and hooks onto the trunk0 interface
      mode = "bridge";
    };
    vlans.host0v010 = { interface = "host0"; id = 10; }; # production MGMT
    vlans.host0v030 = { interface = "host0"; id = 30; }; # production IOT
    vlans.host0v040 = { interface = "host0"; id = 40; }; # production LAN
    vlans.host0v024 = { interface = "host0"; id = 24; }; # for test
    interfaces = {
      host0v010.ipv4.addresses = [{address = "10.0.10.190"; prefixLength = 24;}];
      host0v030.useDHCP = true;  # gets DHCP from existing over trunk
      host0v040.useDHCP = true;  # gets DHCP from existing
      host0v024.useDHCP = true;  # gets DHCP from the guest firewall
      host0.useDHCP = true;      # native untagged gets DHCP from the guest; the wired switch blackholes untagged packets
      enp7s0.useDHCP = true;
    };
    # BELOW IS FOR TEST SCENARIO ONLY
    # The following is a test scenario for sending all my traffic out via the evaluation firewall
    defaultGateway = {  
      address = "10.0.10.252";   # gateway address on the evaluation firewall subnet
      interface = "host0v010";   # the bridged VLAN 10 interface
    };
    nameservers = ["10.0.10.252"];  # use the evaluation firewall for DNS
  };
}

This all works like a charm (with no ipv6). private@trunk0 and public@wan1 show up as the VM attaching to the interfaces but the host doesn’t get to see the guest’s ip details.

For completeness and I didn’t mention it earlier, the WAN side of the firewall gets this libvirt defintion:

<interface type="direct">
  <mac address="aa:bb:cc:xx:yy:zz"/>
  <source dev="wan1" mode="passthrough"/>
  <target dev="public"/>
  <model type="virtio"/>
  ...
</interface>
1 Like

In case someone stumbles upon this and wants to use networkd directly, I have this configured on my home-server. Several VLANs and bridges on top of them, all tied to one physical interface. The config is here.

2 Likes

Thanks for your config!

I see that you are creating VLANs, then assigning a bridge to each VLAN, etc. Can you explain, why this approach?

I also see that you are assigning VLANs to a physical interface. I was looking for a way to assign them to a bridge, just like in my scripted-networking example above (i.e., for them to show up as vlan30@br0, vlan40@br0, etc).

    # Assign VLANs to physical ethernet device
    networks."20-vlan-to-phys" = {
      matchConfig.Name = "phys0";
      networkConfig.VLAN = [ "vlan-lan" "vlan-iot" "vlan-guest" "vlan-server" "vlan-vpn" ];
    };