We have been operating our cluster with NixOS and using our own custom scripts for job scheduling. To scale and better manage our scheduling logic, we are introducing K8s.
However, since we want to keep using our existing Nix-based build and deployment system, we’re mounting the host’s Nix Store into containers. For a rough example, you can refer to this configuration: k8s on NixOS · GitHub
This approach is possible because all nodes in our k8s cluster are NixOS machines.
We’re curious if there are others attempting similar approaches.
While we’re fairly confident about the methodology itself, it’s challenging to deal with areas that need new development and going through trial and error.
We’d like to know if there are well-developed solutions in the same direction
(we haven’t found any yet. While nix-snapshotter and arion are similar in some aspects and have influenced our approach, they don’t provide a complete solution for what we’re trying to achieve) or if there are others interested in this approach.