Use nixOS as single-node kubernetes cluster

Help
1

I posted this question already yesterday in the discord, but didn’t receive a response so far. Therefore I repost here with a slight reword:

I just tried the “1 master + 1 node” approach from the Wiki, but without adding a second computer to the mix…

{
  services.kubernetes.roles = ["master" "node"];
  services.kubernetes.masterAddress = "192.168.178.76";
  services.kubernetes.kubelet.extraOpts = "--fail-swap-on=false";
  services.kubernetes.easyCerts = true;
  services.kubernetes.apiserver = {
    securePort = 8443;
    advertiseAddress = "192.168.178.76";
  };
}

Though when I switch then etcd.service can’t start as some files are missing:

Sep 01 23:07:00 tux-nixos etcd[22865]: peerTLS: cert = /var/lib/kubernetes/secrets/etcd.pem, key = /var/lib/kubernetes/secrets/etcd-key.pem, ca = , trusted-ca = /var/lib/kubernetes/secrets/ca.pem, client-cert-auth = false, crl-file = 
Sep 01 23:07:00 tux-nixos etcd[22865]: open /var/lib/kubernetes/secrets/etcd.pem: no such file or directory

Someone has an idea how to get kubernetes running?

1 Like
2

Have you tried generating a certificate and put it into that path? Don’t know if NixOS is supposed to generate them or not …

3

I wouldn’t know how to do so.

What kind of certificate is even needed? I’m really new to that kind of stuff. Though I expected that enabling easyCert would deal with that stuff for me…

4

I just ran the test for kubernetes, it fails as well after trying ~300 times (~30 minutes) to restart some services…

Within those thousands of thousands of lines, I found some hints that it again can’t find some pem files.

So as even the tests fail, it seems to be a bug in nixos.

2 Likes
5

sorry for late replay on this, maybe this will be enough to generate such a key manually just for testing purposes?

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

I used it for some jupyter notebook configuration a while back.

6

Not direcly addressing your concern, but — if you can — you might try to use k3s it has good UX — generally speaking. We are on a similar journey at this moment, so in the days / weeks to come we might exchange eurekas and caveats (k3s bound).

7

I managed to get kubernetes up and running.

Well hidden in a github issue (which I can’t find now) I found that I have to delete some files and that I have to set masterAdress to the hostname, as the cert-deamon seems to not be able to generate a cert for an IP.

Also I learnt that apiserver.advertiseAddress has to be the IP (as I changed it to the hostname as well in first try).

So now the kubernetes is up and running.

Thanks for pointing me to k3s anyway, I might give it a shot.

1 Like
8

For the sake of completenes, I found the GitHub issue, which helped me identify the “broken” files: etcd not init etcd.pem with services.kubernetes.roles master · Issue #59364 · NixOS/nixpkgs · GitHub

2 Likes
9

@NobbZ

It would be great if you could share the final Nix config for single node k8s, including some comments about the various challenges you found would be amazing.

1 Like
10

It is probably burried in my git history. I dropped the whole setup after a very short time in favor of a k3os VM for the sake of simplicity, which again I dropped another month later, as I changed jobs and k8s didn’t play a role anymore.

1 Like
11

I’ve been trying to set up a single-node Kubernetes cluster following Kubernetes - NixOS Wiki, NixOS Manual and Appendix A. Configuration Options, but I’m running into persistent issues with the PKI setup and certificate management.

Despite having all the expected files in /var/lib/cfssl/ (including ca.pem, ca-key.pem, etc.), the certmgr service fails to start because /etc/certmgr/certmgr.json is missing and isn’t generated automatically by the NixOS modules. The logs show errors like “failed loading spec from /nix/store/…-certmgr” and “failed reading …-certmgr,” and as a result, services like kube-addon-manager don’t start.

I’ve tried cleaning up any old state, running nixos-rebuild switch, and even manually creating the certmgr config, but no luck so far. The cfssl service is running, but attempts to query it over TLS result in “unknown CA” errors, which I assume is because the cert isn’t trusted locally.

Has anyone managed to get this working recently? Is there something special needed to trigger the module to create /etc/certmgr/certmgr.json, or a recommended way to debug/fix certmgr and the Kubernetes PKI setup?

Any advice or recent success stories would be appreciated!

12

I’m planning to try this:

1 Like