Separate containers/services from host network

[Nyxos]

I’m trying to separate my containers/services from my host network, such that they aren’t routed through any potential VPNs running on the host. But I haven’t found a good way to go about this. I’ve tried out microvm.nix but I hit a roadblock with the networking. I found some guides on IP routing but any declarative approach seems to require systemd-networkd which is apparently not very suitable for desktop use (I use my PC as both a desktop and a 24/7 server). What’s the best way to go about this? I would strongly prefer a declarative solution so that I don’t have to run ip commands in the terminal if possible.

Right now I declare my containers through home manager (services.podman.containers.${name}) and it’s worked very well so far, the only problem remaining is the network separation.

Rather than doing the IP routing on the desktop, is it more reasonable to do it on the router? It’s OpenWRT.

Thanks!

[bitbloxhub]

Tried getting ChatGPT to search around for this and it looks like there are 2 ways to do this.
One way to do this is with some nftables rules and assigning the Podman containers to a systemd slice, then making the rules use an fwmark to route anything from the cgroup of the slice to the correct host interface. The problem is that nftables requires the cgroup to exist when the rules are created, so you need a more complex reconciler design.

The fwmark may also conflict with other software.

Raw ChatGPT ouput

The robust design is:

Selected Home Manager Podman units
        ↓
podman-novpn.slice
        ↓
nftables matches the slice cgroup and sets one reserved fwmark bit
        ↓
ip rule selects a dedicated routing table
        ↓
traffic leaves through the ordinary physical interface, bypassing VPN routes

It stays rootless, keeps NetworkManager, requires no VM/macvlan/router changes, and does not classify the entire user by UID.

1. Home Manager: dedicated slice and readiness marker

Replace podman-example.service with the actual units generated for the selected containers.

{ pkgs, ... }:

{
  systemd.user.slices.podman-novpn = {
    Unit.Description = "Rootless containers routed outside host VPNs";
  };

  # Keeps the slice instantiated even when no selected container is running.
  # The marker tells the system-level path unit that the cgroup now exists.
  systemd.user.services.podman-novpn-keeper = {
    Unit = {
      Description = "Keep podman-novpn.slice instantiated";
      Before = [
        "podman-example.service"
      ];
    };

    Service = {
      Type = "simple";
      Slice = "podman-novpn.slice";

      ExecStartPre =
        "${pkgs.coreutils}/bin/touch %t/podman-novpn.ready";

      ExecStart =
        "${pkgs.coreutils}/bin/sleep infinity";

      ExecStopPost =
        "${pkgs.coreutils}/bin/rm -f %t/podman-novpn.ready";

      Restart = "always";
      RestartSec = 1;
    };

    Install.WantedBy = [ "default.target" ];
  };

  # Apply this override to each selected rootless Podman service.
  systemd.user.services.podman-example.Service.Slice =
    "podman-novpn.slice";
}

Find the generated Podman unit names with:

systemctl --user list-unit-files 'podman-*.service'

Confirm placement with:

systemctl --user show podman-example.service \
  -p Slice \
  -p ControlGroup

2. NixOS: account and stable identifiers

Use the actual username and physical interface. Explicitly assigning the UID makes the runtime-directory path deterministic; it will commonly be 1000.

{
  users.users.alice = {
    uid = 1000;
    linger = true;

    # Preserve the rest of the user's normal account configuration.
  };
}

Lingering keeps the user systemd manager and rootless containers running without an interactive login.

Use named constants for the routing pieces:

let
  user = "alice";
  uid = config.users.users.${user}.uid;

  physicalInterface = "wlp82s0";

  routeTableId = 200;
  routeTableName = "podman-novpn";

  # One reserved bit, rather than replacing the whole packet mark.
  routeMark = "0x40000000";
  routeMarkMask = "0x40000000";

  # Must precede relevant VPN catch-all rules.
  rulePriority = 1000;
in

The values are arbitrary local allocations. Keep them centralized and check for conflicts with:

ip rule show
sudo nft list ruleset

Name the routing table:

environment.etc."iproute2/rt_tables.d/podman-novpn.conf".text = ''
  ${toString routeTableId} ${routeTableName}
'';

3. System reconciler

The reconciler:

1. Finds the real podman-novpn.slice cgroup at runtime.
2. Calculates the correct nftables cgroup level.
3. Reads the current non-VPN routes from the physical interface.
4. Rebuilds the dedicated routing table.
5. Replaces the owned nftables table.
6. Replaces the matching policy rule.

{ config, lib, pkgs, ... }:

let
  user = "alice";
  uid = config.users.users.${user}.uid;

  physicalInterface = "wlp82s0";

  routeTableId = 200;
  routeTableName = "podman-novpn";

  routeMark = "0x40000000";
  routeMarkMask = "0x40000000";
  rulePriority = 1000;

  cleanupScript = pkgs.writeShellApplication {
    name = "cleanup-podman-novpn";

    runtimeInputs = [
      pkgs.iproute2
      pkgs.nftables
    ];

    text = ''
      set -u

      nft delete table inet podman_novpn 2>/dev/null || true

      while ip -4 rule delete \
        priority ${toString rulePriority} \
        fwmark ${routeMark}/${routeMarkMask} \
        table ${toString routeTableId} 2>/dev/null
      do
        :
      done

      ip -4 route flush \
        table ${toString routeTableId} 2>/dev/null || true
    '';
  };

  syncScript = pkgs.writeShellApplication {
    name = "sync-podman-novpn";

    runtimeInputs = [
      pkgs.coreutils
      pkgs.findutils
      pkgs.gawk
      pkgs.gnugrep
      pkgs.iproute2
      pkgs.nftables
    ];

    text = ''
      set -euo pipefail

      readonly iface=${lib.escapeShellArg physicalInterface}
      readonly uid=${toString uid}

      readonly table_id=${toString routeTableId}
      readonly priority=${toString rulePriority}
      readonly mark=${lib.escapeShellArg routeMark}
      readonly mark_mask=${lib.escapeShellArg routeMarkMask}

      readonly nft_family=inet
      readonly nft_table=podman_novpn

      user_root="/sys/fs/cgroup/user.slice/user-''${uid}.slice/user@''${uid}.service"

      cgroup_absolute="$(
        find "$user_root" \
          -type d \
          -name podman-novpn.slice \
          -print \
          -quit 2>/dev/null || true
      )"

      if [[ -z "$cgroup_absolute" ]]; then
        echo "podman-novpn.slice does not currently exist"
        ${lib.getExe cleanupScript}
        exit 0
      fi

      cgroup_path="''${cgroup_absolute#/sys/fs/cgroup/}"
      cgroup_level="$(
        awk -F/ '{ print NF }' <<<"$cgroup_path"
      )"

      if [[ ! -e "/sys/class/net/$iface" ]]; then
        echo "Physical interface does not exist: $iface"
        ${lib.getExe cleanupScript}
        exit 0
      fi

      if ! ip -4 route show table main default dev "$iface" |
           grep -q '^default '; then
        echo "No ordinary default route through $iface"
        ${lib.getExe cleanupScript}
        exit 0
      fi

      echo "Using cgroup: $cgroup_path"
      echo "Using interface: $iface"

      # Copy the current IPv4 routes associated with the physical interface.
      # This picks up DHCP gateway, connected LAN route, preferred source, etc.
      ip -4 route flush table "$table_id"

      while IFS= read -r route; do
        [[ -n "$route" ]] || continue

        read -r -a route_argv <<<"$route"
        ip -4 route replace \
          table "$table_id" \
          "''${route_argv[@]}"
      done < <(
        ip -4 route show table main dev "$iface"
      )

      # Rebuild our dedicated nftables table.
      nft delete table "$nft_family" "$nft_table" \
        2>/dev/null || true

      nft -f - <<EOF
      table $nft_family $nft_table {
        chain mark_output {
          type route hook output priority mangle; policy accept;

          socket cgroupv2 level $cgroup_level "$cgroup_path" \
            meta mark set meta mark | $mark \
            counter \
            comment "Route podman-novpn.slice outside host VPNs"
        }
      }
      EOF

      # Replace only our exact policy-routing rule.
      while ip -4 rule delete \
        priority "$priority" \
        fwmark "$mark/$mark_mask" \
        table "$table_id" 2>/dev/null
      do
        :
      done

      ip -4 rule add \
        priority "$priority" \
        fwmark "$mark/$mark_mask" \
        table "$table_id"
    '';
  };
in
{
  assertions = [
    {
      assertion = uid != null;
      message = "users.users.${user}.uid must be explicitly configured";
    }
  ];

  networking.nftables.enable = true;

  environment.etc."iproute2/rt_tables.d/podman-novpn.conf".text = ''
    ${toString routeTableId} ${routeTableName}
  '';

  systemd.services.podman-novpn-sync = {
    description =
      "Reconcile non-VPN routing for selected rootless Podman services";

    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;

      ExecStart = lib.getExe syncScript;
      ExecReload = lib.getExe syncScript;
      ExecStop = lib.getExe cleanupScript;
    };
  };
}

The nftables chain must be a route output chain. That causes route selection to be reconsidered after the packet mark changes.

The mark operation preserves unrelated bits:

meta mark set meta mark | 0x40000000

The ip rule tests only that reserved bit:

fwmark 0x40000000/0x40000000

4. Event-driven cgroup activation

The Home Manager keeper creates:

/run/user/1000/podman-novpn.ready

Watch that exact marker using a system path unit:

systemd.paths.podman-novpn-sync = {
  description = "Watch for Podman non-VPN slice readiness";

  wantedBy = [ "paths.target" ];

  pathConfig = {
    PathExists =
      "/run/user/${toString uid}/podman-novpn.ready";

    Unit = "podman-novpn-sync.service";
  };
};

This avoids hard-coding the actual cgroup path or cgroup depth. The sync script discovers both after the marker appears.

Because the sync service uses RemainAfterExit = true, the persistent marker does not cause a repeated activation loop.

5. Event-driven NetworkManager updates

NetworkManager remains responsible for Wi-Fi, Ethernet, DHCP, roaming, and the desktop network.

Install a dispatcher that reconciles when the selected physical interface changes:

environment.etc."NetworkManager/dispatcher.d/90-podman-novpn".source =
  pkgs.writeShellScript "podman-novpn-networkmanager-dispatcher" ''
    interface="$1"
    action="$2"

    case "$action" in
      up|down|dhcp4-change|connectivity-change)
        if [[ "$interface" == ${lib.escapeShellArg physicalInterface} ]]; then
          ${pkgs.systemd}/bin/systemctl \
            reload-or-restart \
            --no-block \
            podman-novpn-sync.service
        fi
        ;;
    esac
  '';

There is no polling timer.

The relevant events are:

user slice becomes ready
    → systemd.path starts reconciler

physical route or DHCP changes
    → NetworkManager dispatcher reloads reconciler

NixOS configuration changes
    → systemd unit definitions are updated during rebuild

6. Activation and verification

After applying both NixOS and Home Manager configurations:

systemctl --user status podman-novpn-keeper.service
systemctl --user show podman-novpn.slice -p ControlGroup

Trigger or inspect the system configuration:

sudo systemctl reload-or-restart podman-novpn-sync.service

sudo nft list table inet podman_novpn
ip rule show
ip route show table podman-novpn

Check an individual container unit:

systemctl --user show podman-example.service \
  -p Slice \
  -p ControlGroup

It should report:

Slice=podman-novpn.slice

With a full-tunnel VPN active:

curl -4 https://ifconfig.me
podman exec example curl -4 https://ifconfig.me

The host request should use the VPN exit, while the selected container should use the ordinary physical-interface route.

Remaining caveat

A VPN kill switch may install firewall rules that drop all direct traffic leaving the physical interface. This setup controls routing, but cannot override a later firewall rejection.

If that happens, the VPN firewall must permit traffic carrying:

0x40000000/0x40000000

Also check ip rule show after connecting the VPN. The podman-novpn rule must appear before any relevant VPN catch-all rule; lower numerical priority values run first.

There’s also an alternative implementation using a user with JoinsNamespaceOf and PrivateNetwork-based network namespaces instead of an fwmark, although it still requires a reconciler service. If you can get this to work I recommend it.

Raw ChatGPT output

The cleanest design that preserves Home Manager’s rootless services.podman.containers is:

Let the user systemd manager own the network namespace, let the generated Podman units join it with JoinsNamespaceOf=, and let one NixOS root service configure that namespace through the anchor service’s PID.

Do not create a root-owned /run/netns/... namespace and try to make user services join it. Joining a root-owned namespace crosses privilege boundaries and is less reliable. JoinsNamespaceOf= is specifically designed to let systemd units share a private network namespace, but PrivateNetwork=true must be enabled on both the anchor and joining units. (Freedesktop)

Architecture

Home Manager user manager
│
├─ podman-novpn-netns.service
│    PrivateNetwork=true
│    owns the outer network namespace
│
├─ podman-foo.service
│    PrivateNetwork=true
│    JoinsNamespaceOf=podman-novpn-netns.service
│
└─ podman-bar.service
     PrivateNetwork=true
     JoinsNamespaceOf=podman-novpn-netns.service

NixOS root service
│
├─ finds anchor PID
├─ creates veth pair
├─ moves one end into anchor namespace
├─ configures addresses/default route
├─ installs `ip rule iif novpn-host`
└─ configures forwarding + NAT through physical NIC

Rootless Podman remains rootless. Its default pasta networking runs from inside the shared outer namespace, so pasta discovers that namespace’s route rather than the desktop namespace’s VPN route. Rootless Podman uses pasta by default for private networking. (Podman Documentation)

---

Home Manager

1. Namespace anchor

{ pkgs, ... }:

let
  anchorScript = pkgs.writeShellScript "podman-novpn-netns-anchor" ''
    set -euo pipefail

    pid_file="$XDG_RUNTIME_DIR/podman-novpn-netns.pid"

    printf '%s\n' "$$" > "$pid_file"

    cleanup() {
      rm -f \
        "$pid_file" \
        "$XDG_RUNTIME_DIR/podman-novpn-netns.ready"
    }

    trap cleanup EXIT INT TERM

    ${pkgs.coreutils}/bin/sleep infinity &
    wait "$!"
  '';
in {
  systemd.user.services.podman-novpn-netns = {
    Unit = {
      Description = "Non-VPN network namespace for rootless Podman";
    };

    Service = {
      Type = "simple";

      # This service owns the shared namespace.
      PrivateNetwork = true;

      ExecStart = anchorScript;

      Restart = "always";
      RestartSec = 1;
    };

    Install.WantedBy = [ "default.target" ];
  };
}

The PID file gives the root configurator a stable way to find:

/proc/$PID/ns/net

The anchor must remain alive because its process keeps the namespace alive.

2. Readiness gate

The namespace initially contains only loopback. The Podman services must wait until root has attached and configured the veth.

{
  systemd.user.paths.podman-novpn-netns-ready = {
    Unit.Description = "Watch for configured non-VPN namespace";

    Path = {
      PathExists = "%t/podman-novpn-netns.ready";
      Unit = "podman-novpn-netns-ready.service";
    };

    Install.WantedBy = [ "paths.target" ];
  };

  systemd.user.services.podman-novpn-netns-ready = {
    Unit.Description = "Non-VPN namespace is configured";

    Service = {
      Type = "oneshot";
      ExecStart = "${pkgs.coreutils}/bin/true";
      RemainAfterExit = true;
    };
  };
}

3. Override each selected Podman unit

For every container that should bypass the VPN:

{
  systemd.user.services.podman-example = {
    Unit = {
      Requires = [
        "podman-novpn-netns.service"
        "podman-novpn-netns-ready.service"
      ];

      After = [
        "podman-novpn-netns.service"
        "podman-novpn-netns-ready.service"
      ];

      JoinsNamespaceOf = [
        "podman-novpn-netns.service"
      ];
    };

    Service = {
      # Required on the joining unit too.
      PrivateNetwork = true;
    };
  };
}

That missing PrivateNetwork=true on the Podman units is important. JoinsNamespaceOf= only shares the network namespace when the relevant namespace isolation option is active for both units. (Freedesktop)

Home Manager may generate a slightly different service name. Check:

systemctl --user list-unit-files 'podman-*.service'

---

NixOS root configuration

Use fixed internal addresses; they exist only between the host and namespace:

let
  user = "alice";
  uid = config.users.users.${user}.uid;

  uplink = "wlp82s0";

  hostVeth = "novpn-host";
  namespaceVeth = "novpn-ns";

  hostAddress = "10.231.0.1";
  namespaceAddress = "10.231.0.2";
  namespaceSubnet = "10.231.0.0/24";

  routeTableId = 200;
  routeTableName = "podman-novpn";
  rulePriority = 1000;
in

Give the user an explicit UID and lingering:

{
  users.users.alice = {
    uid = 1000;
    linger = true;
  };

  environment.etc."iproute2/rt_tables.d/podman-novpn.conf".text = ''
    200 podman-novpn
  '';

  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
}

Root configuration script

The root service should be idempotent: every invocation validates the PID and rebuilds only the state it owns.

configureScript = pkgs.writeShellApplication {
  name = "configure-podman-novpn-netns";

  runtimeInputs = [
    pkgs.coreutils
    pkgs.gawk
    pkgs.gnugrep
    pkgs.iproute2
    pkgs.nftables
    pkgs.util-linux
  ];

  text = ''
    set -euo pipefail

    runtime_dir="/run/user/${toString uid}"
    pid_file="$runtime_dir/podman-novpn-netns.pid"
    ready_file="$runtime_dir/podman-novpn-netns.ready"

    iface=${lib.escapeShellArg uplink}
    host_veth=${lib.escapeShellArg hostVeth}
    ns_veth=${lib.escapeShellArg namespaceVeth}

    table=${toString routeTableId}
    priority=${toString rulePriority}

    rm -f "$ready_file"

    [[ -r "$pid_file" ]] || {
      echo "Namespace anchor PID file does not exist"
      exit 0
    }

    read -r pid < "$pid_file"

    [[ "$pid" =~ ^[0-9]+$ ]] || {
      echo "Invalid PID: $pid" >&2
      exit 1
    }

    [[ -e "/proc/$pid/ns/net" ]] || {
      echo "Namespace anchor no longer exists"
      exit 0
    }

    [[ -e "/sys/class/net/$iface" ]] || {
      echo "Uplink $iface does not exist"
      exit 0
    }

    default_route="$(
      ip -4 route show table main default dev "$iface" |
        head -n 1
    )"

    [[ -n "$default_route" ]] || {
      echo "No ordinary default route through $iface"
      exit 0
    }

    # Remove any stale veth left from an earlier anchor instance.
    ip link delete "$host_veth" 2>/dev/null || true

    ip link add "$host_veth" type veth peer name "$ns_veth"
    ip link set "$ns_veth" netns "$pid"

    ip address replace ${hostAddress}/24 dev "$host_veth"
    ip link set "$host_veth" up

    nsenter --target "$pid" --net \
      ip link set lo up

    nsenter --target "$pid" --net \
      ip address replace ${namespaceAddress}/24 dev "$ns_veth"

    nsenter --target "$pid" --net \
      ip link set "$ns_veth" up

    nsenter --target "$pid" --net \
      ip route replace default via ${hostAddress}

    # Rebuild the dedicated route table from the uplink's current main-table
    # routes, including its DHCP gateway and connected route.
    ip -4 route flush table "$table"

    while IFS= read -r route; do
      [[ -n "$route" ]] || continue
      read -r -a words <<<"$route"
      ip -4 route replace table "$table" "''${words[@]}"
    done < <(
      ip -4 route show table main dev "$iface"
    )

    while ip -4 rule delete \
      priority "$priority" \
      iif "$host_veth" \
      table "$table" 2>/dev/null
    do
      :
    done

    ip -4 rule add \
      priority "$priority" \
      iif "$host_veth" \
      table "$table"

    nft delete table inet podman_novpn 2>/dev/null || true

    nft -f - <<EOF
    table inet podman_novpn {
      chain forward {
        type filter hook forward priority filter; policy accept;

        iifname "$host_veth" oifname "$iface" accept
        iifname "$iface" oifname "$host_veth" \
          ct state established,related accept
      }

      chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;

        ip saddr ${namespaceSubnet} oifname "$iface" masquerade
      }
    }
    EOF

    touch "$ready_file"
  '';
};

Use an inet table for forwarding, but NAT remains IPv4 here because this design uses an RFC1918 IPv4 transit subnet.

Root service

systemd.services.podman-novpn-netns-configure = {
  description = "Configure rootless Podman non-VPN namespace";

  serviceConfig = {
    Type = "oneshot";
    ExecStart = lib.getExe configureScript;
  };
};

Event-driven start

Watch the anchor PID file:

systemd.paths.podman-novpn-netns-configure = {
  description = "Watch for rootless Podman namespace anchor";
  wantedBy = [ "paths.target" ];

  pathConfig = {
    PathExists =
      "/run/user/${toString uid}/podman-novpn-netns.pid";

    Unit = "podman-novpn-netns-configure.service";
  };
};

NetworkManager updates

Reconfigure when DHCP or the physical route changes:

environment.etc."NetworkManager/dispatcher.d/90-podman-novpn".source =
  pkgs.writeShellScript "podman-novpn-nm-dispatcher" ''
    interface="$1"
    action="$2"

    case "$action" in
      up|down|dhcp4-change|connectivity-change)
        if [[ "$interface" == ${lib.escapeShellArg uplink} ]]; then
          ${pkgs.systemd}/bin/systemctl start \
            --no-block \
            podman-novpn-netns-configure.service
        fi
        ;;
    esac
  '';

---

Port publishing caveat

This is the main operational difference.

Because Podman and pasta now run in the outer namespace, a normal published port such as:

-p 8080:8080

is exposed in the non-VPN namespace, not automatically on the desktop host’s ordinary namespace.

For outbound-only services, that is fine.

For services that must be reachable from the host or LAN, add explicit host forwarding, for example:

chain prerouting {
    type nat hook prerouting priority dstnat;

    tcp dport 8080 dnat to 10.231.0.2:8080
}

and permit the corresponding forwarding. Alternatively, run a host reverse proxy that connects to 10.231.0.2.

This is unavoidable: the namespace separation that prevents the VPN from controlling those sockets also separates their listening sockets from the host network namespace.

---

Why this is the cleanest netns version

It has:

• no fwmark allocation;
• no cgroup-path matching;
• no UID-based routing;
• no rootful Podman;
• no VM, macvlan, or router configuration;
• no polling timer;
• no attempt by a user service to enter a root-owned namespace;
• a direct ip rule iif novpn-host, which is unambiguous;
• event-driven setup from both namespace creation and NetworkManager changes.

The price is the real networking machinery required by a separate namespace:

namespace anchor + veth + forwarding + NAT + readiness ordering

That is more complex than the fwmark version, but it is the cleanest robust mark-free implementation while retaining HM-managed rootless Podman containers.

Sorry for dumping raw ChatGPT output at you, but the summaries above the details blocks are human-written and should at least help you out a bit.

Essentially the reconcilers are one-shot systemd services that are retriggerd on systemd paths and NetworkManager dispatcher scripts, and run shell scripts that use ip and nft commands with pkgs.writeShellApplication and ExecStart/ExecReload/ExecStop.