Thanks for sharing, sounds interesting! Seems your GitHub repo is still private though, I’m getting a 404 error.
woops, fixed now! GitHub - bouk/nixos-containers: Minimal NixOS Containers · GitHub
This looks great! One thing I like to do with containers is instead of setting up veth and dhcp, I pass in a wireguard netdev. In general, I avoid systemd resolved like the plague because it can cross network namespaces and leak dns queries. My homelab is out of commission but when I find the time I might see if I can do something like that with this.
Would like to know more about how you’re using wireguard with containers… One thing I’m not entirely sure about yet is how to do secure networking with them. I run Tailscale on the host but I don’t want the containers to automatically gain access to the Tailscale network as well.
I’ve solved this with a bunch of iptables DROP rules but it seems to me there should be better solutions as well.
Like I said, it’s been awhile since I’ve had containers running on my homelab. But if I recall correctly, I made wireguard interfaces on my host machine using wirenix (but you can use plain old wireguard, wirenix is long overdue for a refactor but I just haven’t had the motivation) and passed them in as using containers.<name>.interfaces. One property of wirenix that I exploit is that, by default, all ips are ipv6 and generated from a hash of the node name, so there was never any need for me to worry about dhcp. And dns I think was a standard split instance of bind that routed requests for secured resources over the vpn or otherwise forwarded on to the outside world or provided nxdomain.
The whole thing was quite cool, but there were multiple layers of complexity that led to the whole thing not being maintainable by one guy. Mostly because I was messing with microvms as well which has nothing to do with this.
I really like this, I always thought something like this should exist.
One thing that would be nice is a way to use a less-privileged user to deploy the container, so that if I deploy it from a pipeline, i don’t need to give it root access to the whole machine. But that’s probably not that easy, because it needs to copy nix store paths to the host, which needs root. The only solution to this that comes to my mind is let the host build the derivation by itself (or by a remote builder) so that the host doesn’t have to trust the pipeline.
I actually had this exact idea today and managed to implement it: Allow deploying as non-root by bouk · Pull Request #1 · bouk/nixos-containers · GitHub
apparently there’s an RFC: rfcs/rfcs/0108-nixos-containers.md at c655bdaab40f7a467f75dbb5af4325d991874e44 · NixOS/rfcs · GitHub with a currently WIP PR: Add support for RFC108 imperative containers by m1cr0man · Pull Request #216025 · NixOS/nixpkgs · GitHub that rewrites the NixOS imperative containers to use systemd-nspawn like what I’m doing in my module, great to see
From the very end of your blog post:
As we grow I’m contemplating modifying nomad-driver-nspawn to run NixOS system closures directly, which would be an awesome NixOS-native way to orchestrate containers.
We did that at work for a while. It was years ago and probably very specific to our use case, but you might be interested: Comparing 462021eaa5ce00a8bac3ec4d20af7e203735b20d...main · input-output-hk/nomad-driver-nix · GitHub