Let me just quickly get my opinion out there (obviously I also have one).
TLDR; I believe the in-tree nixpkgs module should be veeery simple, if even exist. And I think that any full-fledged, battery-included kubernetes module should live externally in a separate repo as a flake or just an external module like Simple NixOS Mailserver or similar.
Some of the pros of this approach:
It has the potential of reducing the number of NixOS options from currently ~145 to ~20, provided that each of the cluster components have only options for: enable and extraConfig.
It will become much easier to support many different versions of Kubernetes with the same module, since the module doesn’t really do anything other than letting you enable components with default options.
It gives the advantage of choice. Some people like kubeadm, some like kubeup, others like different kubernetes installations, even others like to do kubernetes-the-hard-way (either because they love the pain, or just because they have strange special requirements) ← that’s me by the way.
It comes with the freedom of a decoupled dev-cycle for each independent kubernetes flake/module. Perhaps you don’t have merge-access to nixpkgs (like me), and think it’s difficult to get people to review of merge your kubernetes PR’s in nixpkgs. Seperate repos can definitely help on that.
One disadvantage of course, is that it can be difficult for new users to find out-of-tree modules, sure; But I think we should definitely maintain a list somewhere with links to cool external flakes.
When we started using Kubernetes at my day job a couple of years ago, we deselected kubeadm because at that time it still had a: don’t use in production notice in the public README. Also, more importantly, it didn’t support multi-master (HA) setups. It looks like both of my concerns are no longer valid, which makes kubeadm more attractive for us, definitely.
Me too. Kubernetes should have native support for containerd as well. There are other choices to make as well, e.g: which CNI to use? Flannel is not… optimal IMHO. But as it turns out, many of those choices tend to be opinionated and subjective.
I don’t want to voice any opinion too strongly yet. I’ve seen support both for both hypothetical solutions: A completely minimal wrapper, and a kubeadm-based one. I think both sound potentially interesting and worth trying to flesh out to evaluate side-by-side.
One thing we should let be a guiding principle is the experience we have gained in maintaining k8s over the last several release. Simplicity from the perspective of Nix is important, otherwise we won’t be able to keep up and provide a good and, not least, flexible (eg. to CNI or container choice) user experience.
Good, I see multiple points here. Beside the discussion about “kubeadm vs something else“, wen can still add support for different runtimes and networking solutions.
I started to contribute a CRI-O module to NixOS these days, which could be a fist approach to configure something else beside docker as runtime. The same can be done for networks, like Cilium as Alternative to flannel.
In terms of the point “where and how to develop the Kubernetes module”:
On one hand I see the benefits developing it in within a dedicated project. On the other hand I guess it would be simply awesome if we could tag NixOS as native Kubernetes distribution some day without relying on external sources. Maybe we could ask the CNCF to put NixOS into their certified distribution portfolio some day.
The hardest part of Kubernetes is the fast paced upstream development, which we can follow as its best by sticking to the latest release (only).
I don’t think that we have to provide all of the different configurations we have now, but strong security defaults and major options like changing the runtime and network would be nice.
Since there are 3(?) kubernetes maintainers, it seems like a good idea to focus on one opinionated solution and do it well.
Recently I tried and enabled the NixOS module on my server. Things are running but kubectl is missing and not configured. The documentation doesn’t tell me how to connect to the single-node cluster. Just having a simple out-of-the-box single node experience would be quite cool IMO.
The documentation doesn’t tell me how to connect to the single-node cluster.
The module currently doesn’t include kubectl in environment.systemPackages by design. If easyCerts = true though, a cluster-admin keypair is generated and a designated kubeconfig is made available at services.kubernetes.pki.etcClusterAdminKubeconfig.
See the last paragraph of sec-kubernetes: NixOS Manual
Perhaps the docs could be made clearer.
