We’re running Docker experimentally under NixOS and gathering some insight from actual customer usage.
The biggest pain point that we’ve found is the integration between Docker and the NixOS firewall. The docker daemon will manipulate the firewall at runtime and when someone comes along and runs nixos-rebuild switch then this may cause the firewall scripts to be run and kill all dynamic changes.
We’re still running this is on NixOS 15.09 but looking at 18.09 and master this has not changed in between. This appears to be a hard problem because
One option that would work cleanly but would be useless is restarting the docker daemon (and likely avoiding live restarts) whenever the firewall changes.
We started implementing a feature that would allow merging firewall configurations with ad-hoc + declarative states by tagging them markers, but those don’t guarantee order (although they are convergent) and feel brittle with the tagging.
Using host mode networks will work but quickly becomes a pain because the port namespace will collide quickly.
So, is anyone using Docker on NixOS productively at all? This seems to be a prohibitive issue but I haven’t found any conceptual good solution to this. I might be blind, though …