Kubernetes: The NixOS-module of the future

Since the merge of kubernetes: 1.14.3 -> 1.15.3 (with revert of module systemd dependencies) by johanot · Pull Request #67563 · NixOS/nixpkgs · GitHub we now at least have a stable kubernetes module (and package) for the 19.09. Now it’s time to look ahead and discuss what we want for the future, 20.03 in particular, before we get too close to March 2020. :slight_smile:

Let’s make this a thread to collect and continue the discussion started here: kubernetes: bootstrap process is hacked-up together and broken · Issue #61135 · NixOS/nixpkgs · GitHub and kubernetes: add generic builder and kops stable versions by yurrriq · Pull Request #66496 · NixOS/nixpkgs · GitHub and possibly other places.

6 Likes

Hey, thanks for the PR, it looks beautiful and I’m happy to see a recent version of Kubernetes in NixOS.

I’m now wondering if we still want to go ahead with kubeadm or not. I think we could drastically reduce the code by sticking to kubeadm, WDYT?

I also want to propose to have an alternative to docker, like CRI-O. Now it should be not that hard to configure.

1 Like

Let me just quickly get my opinion out there (obviously I also have one). :slight_smile:

TLDR; I believe the in-tree nixpkgs module should be veeery simple, if even exist. And I think that any full-fledged, battery-included kubernetes module should live externally in a separate repo as a flake or just an external module like Simple NixOS Mailserver or similar.

Some of the pros of this approach:

  • It has the potential of reducing the number of NixOS options from currently ~145 to ~20, provided that each of the cluster components have only options for: enable and extraConfig.
  • It will become much easier to support many different versions of Kubernetes with the same module, since the module doesn’t really do anything other than letting you enable components with default options.
  • It gives the advantage of choice. Some people like kubeadm, some like kubeup, others like different kubernetes installations, even others like to do kubernetes-the-hard-way (either because they love the pain, or just because they have strange special requirements) ← that’s me by the way.
  • It comes with the freedom of a decoupled dev-cycle for each independent kubernetes flake/module. Perhaps you don’t have merge-access to nixpkgs (like me), and think it’s difficult to get people to review of merge your kubernetes PR’s in nixpkgs. Seperate repos can definitely help on that.

One disadvantage of course, is that it can be difficult for new users to find out-of-tree modules, sure; But I think we should definitely maintain a list somewhere with links to cool external flakes.

2 Likes

When we started using Kubernetes at my day job a couple of years ago, we deselected kubeadm because at that time it still had a: don’t use in production notice in the public README. :slight_smile: Also, more importantly, it didn’t support multi-master (HA) setups. It looks like both of my concerns are no longer valid, which makes kubeadm more attractive for us, definitely.

Me too. Kubernetes should have native support for containerd as well. There are other choices to make as well, e.g: which CNI to use? Flannel is not… optimal IMHO. But as it turns out, many of those choices tend to be opinionated and subjective.

Thanks for opening this discussion.

I don’t want to voice any opinion too strongly yet. I’ve seen support both for both hypothetical solutions: A completely minimal wrapper, and a kubeadm-based one. I think both sound potentially interesting and worth trying to flesh out to evaluate side-by-side.

One thing we should let be a guiding principle is the experience we have gained in maintaining k8s over the last several release. Simplicity from the perspective of Nix is important, otherwise we won’t be able to keep up and provide a good and, not least, flexible (eg. to CNI or container choice) user experience.

1 Like

Good, I see multiple points here. Beside the discussion about “kubeadm vs something else“, wen can still add support for different runtimes and networking solutions.

I started to contribute a CRI-O module to NixOS these days, which could be a fist approach to configure something else beside docker as runtime. The same can be done for networks, like Cilium as Alternative to flannel.

In terms of the point “where and how to develop the Kubernetes module”:

On one hand I see the benefits developing it in within a dedicated project. On the other hand I guess it would be simply awesome if we could tag NixOS as native Kubernetes distribution some day without relying on external sources. Maybe we could ask the CNCF to put NixOS into their certified distribution portfolio some day.

The hardest part of Kubernetes is the fast paced upstream development, which we can follow as its best by sticking to the latest release (only).

I don’t think that we have to provide all of the different configurations we have now, but strong security defaults and major options like changing the runtime and network would be nice. :slight_smile:

I started GitHub - nix-community/awesome-nix: 😎 A curated list of the best resources in the Nix community [maintainer=@cyntheticfox] exactly for this purpose. So let me know if you end-up creating a new overlay! (or send a PR)

Since there are 3(?) kubernetes maintainers, it seems like a good idea to focus on one opinionated solution and do it well.

Recently I tried and enabled the NixOS module on my server. Things are running but kubectl is missing and not configured. The documentation doesn’t tell me how to connect to the single-node cluster. Just having a simple out-of-the-box single node experience would be quite cool IMO.

Slightly off-topic, but in terms of simplicity you cannot beat minikube when it comes to getting something up and running quickly and easily.

I prefer kind (Kubernetes in Docker) for development. It’s a bit faster to start and also doesn’t depend on VirtualBox.

The documentation doesn’t tell me how to connect to the single-node cluster.

The module currently doesn’t include kubectl in environment.systemPackages by design. If easyCerts = true though, a cluster-admin keypair is generated and a designated kubeconfig is made available at services.kubernetes.pki.etcClusterAdminKubeconfig.

See the last paragraph of sec-kubernetes: NixOS 23.11 manual | Nix & NixOS
Perhaps the docs could be made clearer.

minikube can use KVM.

Back to the topic: if we decide that we want to have it out of the nixpkgs tree, then I’d be happy to help you with that in any case. :slight_smile:

A benefit in having it inside a dedicated repository would be, that it makes it possible to use it on other distributions as well.