We’re running Docker experimentally under NixOS and gathering some insight from actual customer usage.
The biggest pain point that we’ve found is the integration between Docker and the NixOS firewall. The docker daemon will manipulate the firewall at runtime and when someone comes along and runs nixos-rebuild switch then this may cause the firewall scripts to be run and kill all dynamic changes.
We’re still running this is on NixOS 15.09 but looking at 18.09 and master this has not changed in between. This appears to be a hard problem because
One option that would work cleanly but would be useless is restarting the docker daemon (and likely avoiding live restarts) whenever the firewall changes.
We started implementing a feature that would allow merging firewall configurations with ad-hoc + declarative states by tagging them markers, but those don’t guarantee order (although they are convergent) and feel brittle with the tagging.
Using host mode networks will work but quickly becomes a pain because the port namespace will collide quickly.
So, is anyone using Docker on NixOS productively at all? This seems to be a prohibitive issue but I haven’t found any conceptual good solution to this. I might be blind, though …
Wildly off topic but this is at least the second time you’ve mentioned running 15.09 so I’m curious and would love to know how you manage software updates? Does your company maintain a nixpkgs fork based on 15.09 and backport all security fixes? It’s your company that’s responsible for reporting the CVE issue dumps on github so presumably you’re taking care of your own branch as these issues arise. If this is the case does your experience really suggest this was less work than tracking upstream NixOS and upgrading every 6 months? Or is it just how rapid nixpkgs moves to new software that is the barrier to keeping up with upstream for your clients?
Since @ckauhaus gave his talk at NixCon last year I’ve been really curious as to how you and your team operate as it sounds like a really cool place
Well, nothing is perfect. Getting 15.09 stable as platform for us took a lot of effort and we’re juggling many things anyway. We’re currently rolling out an 18.09 based update. We do maintain a fork and update selectively. It is quite a bit of work and given the new abilities with overlays in 18.09 we hope to update the base system more often. However, a more than 10 year experience tells me that large updates will be painful in some way all the time … My experience is that an update can easily - due to customer specific things - take half a year and I’d rather not be in continuous update mode …
And as sysadmin who is trying to motivate a team of developers to migrate 100+ internal PHP 5 applications from Debian 7 to Debian 9 I appreciate your position
Well you could do that but that disables a lot of functionality, like exposing ports, and you’d have to rebuild that. Another option would be to implement custom plugins for IPAM and bridging. Not sure whether that would be a moving target, though …
I was also wondering whether we could be using a separate namespace for docker and wrap all iptables commands that we expose to the docker daemon …
So actually, I’m wondering: couldn’t we try to run dockerd inside its own network namespace? Quick Googling tells me that will lead to its own issues as various mounting issues for /etc/netns and /sys may confuse the tooling …
