Some background: I have played with various ways of deploying containerized services on bare-metal and vps running NixOS:
- combination of
virtualisation.oci-containers.containers(similar to what’s explained here), with
--ipto make inter-container networking working without unnecessarily exposing ports
k3son bare-metal NixOS => this is a pain because usage of
metallbis cumbersome (ingress definitions are unnecessary complicated and error prone) and the only way that is “declarative enough” for my taste involves
helmfilewhich adds huge complexity, risk of breakage on upgrades, isn’t “popular enough” (limited online resources) and isn’t suitable on a multi-purpose host (e.g. doesn’t play nice with firewall etc.)
sops-nix and utility functions to pass the appropriate env to the
systemd units and/or generated start scripts with the necessary `–some-secret=“$(cat /run/secrets/…)” where needed.
I couldn’t really bring myself to jump on the “managed k8s instance” bandwagon, since it’s significantly more expensive than a VPS and gives me less flexibility in terms of “using the capacity I’m paying for” by running different deployments on the same host. And if I need (significantly) more persistent (and/or backup) storage in the future, it would limit the choice of the most cost-effective solution.
This whole thing is a pain as soon as some orchestration and inter-container-communication is required, so I was considering to use
podman-compose to support easy pod deployments and write a thin wrapper nixos-module around this taking care of a
systemd service, env, secrets and exec user transparently.
The fact that I didn’t find much “prior art” in this respect has got me wondering whether I’m embarking on an antipattern here?