Some background: I have played with various ways of deploying containerized services on bare-metal and vps running NixOS:
- combination of
systemd.services
andvirtualisation.oci-containers.containers
(similar to what’s explained here), with--add-host
and--ip
to make inter-container networking working without unnecessarily exposing ports -
k3s
on bare-metal NixOS => this is a pain because usage ofmetallb
is cumbersome (ingress definitions are unnecessary complicated and error prone) and the only way that is “declarative enough” for my taste involveshelmfile
which adds huge complexity, risk of breakage on upgrades, isn’t “popular enough” (limited online resources) and isn’t suitable on a multi-purpose host (e.g. doesn’t play nice with firewall etc.)
I use sops-nix
and utility functions to pass the appropriate env to the systemd
units and/or generated start scripts with the necessary `–some-secret=“$(cat /run/secrets/…)” where needed.
I couldn’t really bring myself to jump on the “managed k8s instance” bandwagon, since it’s significantly more expensive than a VPS and gives me less flexibility in terms of “using the capacity I’m paying for” by running different deployments on the same host. And if I need (significantly) more persistent (and/or backup) storage in the future, it would limit the choice of the most cost-effective solution.
This whole thing is a pain as soon as some orchestration and inter-container-communication is required, so I was considering to use podman-compose
to support easy pod deployments and write a thin wrapper nixos-module around this taking care of a systemd
service, env, secrets and exec user transparently.
The fact that I didn’t find much “prior art” in this respect has got me wondering whether I’m embarking on an antipattern here?