NixOS containers: how to allow Internet access but isolate containers?

Hello,

I am currently trying to use NixOS containers as an additional layer of isolation between applications. One thing I’m having trouble with is networking: I want containers to not be able to access eachother by default, yet can access the Internet. It manifests like this:

[root@container:~]# curl http://google.com
curl: (6) Could not resolve host: google.com

[root@container:~]# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11303ms

It seems like when using private networks with NixOS containers, they do not get internet access. I’ve tried the following things to get this working:

  1. I tried to enable network address translation for the ve-* interfaces:
  networking.nat = {
    enable = true;
    internalInterfaces = ["ve-+"];
    externalInterface = "enp0...";
  };

This did not seem to do anything. I already have ip_forward set to 1, in case that matters, since I do make use of port forwarding with containers:

  boot.kernel.sysctl = {
    "net.ipv4.ip_forward" = 1;
  };
  1. I had read that you need to add a MASQUERADE rule to get it to work, in addition to the above. I tried this:
iptables -t nat -A POSTROUTING -o enp0... -j MASQUERADE

…but as far as I could tell, it did nothing.

  1. I tried bridging the containers to the host. I am not 100% sure what this would do beyond what the default setup does, but I tried it just to see.

  networking.bridges = {
    br0 = {
      interfaces = ["enp..."];
    };
  };
  networking.interfaces = {
    br0 = {
      ipv4.addresses = [{ address = "10.10.10.0/24"; prefixLength = 24; }];
    };
  };
  containers.container = {
    ...
    hostBridge = "br0";
    ...

I would love to have fine grained deny-by-default control over the containers, but I am not the strongest in networking. Does anyone know the best way to go about isolating containers from eachother at least? I am guessing I am missing some iptables rules, but I am a bit of a n00b when it comes to iptables after all of this time, and I don’t really understand a lot of what’s going on there despite attempts.

2 Likes

Have you tried using macvlans? Something like this:

It should mean that each container has its own mac address, it’s own IP address, and its own firewall, so you just open the necessary ports in the container’s firewall and the host’s firewall shouldn’t interfere with that at all.

AFAIK, macvlans can’t be used on a wireless interface, but your enp0 doesn’t look like a wireless interface, so that shouldn’t affect you.

Thanks, this might be the right way to go; I avoided it because I am running this on a physical dedicated server, which only has one IPv4 address allocated with the provider. I am currently not sure how macvlan will interact with their network configuration; maybe it will fail to DHCP lease? I do have a /64 of IPv6 addresses, though, so perhaps I can use an IPv6-only stack for the containers. That seems like a promising avenue if it works.

I ran into issues using the macvlan solution. I suspect it would work if I could grab new routable IPs, but I could not figure out how I might configure this with a /64.

In the end, I figured out how to do this using NAT + bridging. It turns out I was getting close before, but had made some clerical errors.

Here’s what I ended up doing for the host configuration:

  networking = {
    ...
    bridges = {
      br0 = {
        interfaces = [];
      };
    };
    nat = {
      enable = true;
      internalInterfaces = [ "ve-+" "vb-+" containerBridge ];
      externalInterface = hostInterface;
    };
    ...

Then, for the container:

  containers.container = {
    ...
    privateNetwork = true;
    hostBridge = containerBridge;
    localAddress = "${containerAddress}/24";
    config = { config, pkgs, ... }: {
      networking = {
        defaultGateway = containerGateway;
        ...

Earlier I was not thinking about isolation very clearly. I was thinking I would have isolate things on the host side. That might be possible, but what’s a lot more intuitive is doing it on the container side. Just like in the macvlan setup, each container has its own interfaces. Therefore, in order to isolate the containers from each other but still poke a hole in the firewall for the host or another container, you can do something like this (inside of a container’s configuration!):

      networking = {
        defaultGateway = settings.containerGateway;
        firewall = {
          extraCommands = ''
            iptables -F INPUT # TODO: might not be necessary
            iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
            iptables -A INPUT -s ${containerGateway} -p tcp --dport 80 -j ACCEPT
            iptables -A INPUT -j DROP
          '';
        };
      };

However, this is a bit of a hack: the NixOS firewall rule comes after the DROP, which seems to work, but I am not sure if that’s guaranteed to go in that order. There may be security issues here as well, before copying and pasting this consider consulting someone who actually understands iptables :slight_smile:

I think I am understanding this all a bit better now. The macvlans solution is very cool, it is unfortunate I could not figure out how to make it work, but I think this might be more suited to my use case.

1 Like