I’m currently investigating the possibility to replace my kubernetes cluster with pure NixOS based solution. This is a single master node with some unreliable heavy workers. I have already migrated all the “have to be on master” as NixOS services.
I’m now searching for some clever way to migrate the heavy workloads. Although i would like to have them dynamically allocated and reallocated in case of node failure.
I was thinking about encapsulating them in NixOS containers and controlling with something like hashicorp nomad (at least by docs it seems to be an solution).
I’m not sure yet about networking. Kubernetes have nice advantage as it is able to dynamically assign addresses and resolve services by names. Wonder if something like that is possible with NixOS.
Have any of you explored such thing and may suggest some things?
I’ve thought about this somewhat and as far as I can tell I think it would be very awkward to do in pure NixOS. The issue is having workloads be distributed with some kind of scheduler (like k8s/nomad) you are maintaining mutable configuration state, which is difficult to consolidate with a “pure” NixOS setup. Depending on the nature of the workload you could declare & run it on multiple workers and have it be load balanced for redundancy but that doesn’t apply universally.
I’ve run a nomad cluster by declaring all the nomad infrastructure with NixOS and building containers with NixOS too, and it is somewhat nice but you’re still entangling yourself with the mutable state of the cluster which kind of goes against the benefits you get from NixOS in the first place. If it’s necessary to have failover placement, scheduling, rebalancing etc then you can indeed do this, but I would generally prefer just choosing where services should run declaratively (then you get things like service discovery for free!)
it sounds like what you want isn’t necessarily one or the other, but rather the advantages of both.
so technically, combining the two is possible, tho NixOS’s service modules weren’t necessarily written with Kubernetes in mind. Modular services are looking potentially more flexible in that regard, but the code still needs to be written.