Brainstorm: a better, lean container orchestration method for nixos-vps deployments?

Help
1

Some background: I have played with various ways of deploying containerized services on bare-metal and vps running NixOS:

  • combination of systemd.services and virtualisation.oci-containers.containers (similar to what’s explained here), with --add-host and --ip to make inter-container networking working without unnecessarily exposing ports
  • k3s on bare-metal NixOS => this is a pain because usage of metallb is cumbersome (ingress definitions are unnecessary complicated and error prone) and the only way that is “declarative enough” for my taste involves helmfile which adds huge complexity, risk of breakage on upgrades, isn’t “popular enough” (limited online resources) and isn’t suitable on a multi-purpose host (e.g. doesn’t play nice with firewall etc.)

I use sops-nix and utility functions to pass the appropriate env to the systemd units and/or generated start scripts with the necessary `–some-secret=“$(cat /run/secrets/…)” where needed.

I couldn’t really bring myself to jump on the “managed k8s instance” bandwagon, since it’s significantly more expensive than a VPS and gives me less flexibility in terms of “using the capacity I’m paying for” by running different deployments on the same host. And if I need (significantly) more persistent (and/or backup) storage in the future, it would limit the choice of the most cost-effective solution.

This whole thing is a pain as soon as some orchestration and inter-container-communication is required, so I was considering to use podman-compose to support easy pod deployments and write a thin wrapper nixos-module around this taking care of a systemd service, env, secrets and exec user transparently.

The fact that I didn’t find much “prior art” in this respect has got me wondering whether I’m embarking on an antipattern here?

1 Like