Use nixOS as single-node kubernetes cluster

Help
1

I posted this question already yesterday in the discord, but didn’t receive a response so far. Therefore I repost here with a slight reword:

I just tried the “1 master + 1 node” approach from the Wiki, but without adding a second computer to the mix…

{
  services.kubernetes.roles = ["master" "node"];
  services.kubernetes.masterAddress = "192.168.178.76";
  services.kubernetes.kubelet.extraOpts = "--fail-swap-on=false";
  services.kubernetes.easyCerts = true;
  services.kubernetes.apiserver = {
    securePort = 8443;
    advertiseAddress = "192.168.178.76";
  };
}

Though when I switch then etcd.service can’t start as some files are missing:

Sep 01 23:07:00 tux-nixos etcd[22865]: peerTLS: cert = /var/lib/kubernetes/secrets/etcd.pem, key = /var/lib/kubernetes/secrets/etcd-key.pem, ca = , trusted-ca = /var/lib/kubernetes/secrets/ca.pem, client-cert-auth = false, crl-file = 
Sep 01 23:07:00 tux-nixos etcd[22865]: open /var/lib/kubernetes/secrets/etcd.pem: no such file or directory

Someone has an idea how to get kubernetes running?

1 Like
2

Have you tried generating a certificate and put it into that path? Don’t know if NixOS is supposed to generate them or not …

3

I wouldn’t know how to do so.

What kind of certificate is even needed? I’m really new to that kind of stuff. Though I expected that enabling easyCert would deal with that stuff for me…

4

I just ran the test for kubernetes, it fails as well after trying ~300 times (~30 minutes) to restart some services…

Within those thousands of thousands of lines, I found some hints that it again can’t find some pem files.

So as even the tests fail, it seems to be a bug in nixos.

2 Likes
5

sorry for late replay on this, maybe this will be enough to generate such a key manually just for testing purposes?

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

I used it for some jupyter notebook configuration a while back.

6

Not direcly addressing your concern, but — if you can — you might try to use k3s it has good UX — generally speaking. We are on a similar journey at this moment, so in the days / weeks to come we might exchange eurekas and caveats (k3s bound).

7

I managed to get kubernetes up and running.

Well hidden in a github issue (which I can’t find now) I found that I have to delete some files and that I have to set masterAdress to the hostname, as the cert-deamon seems to not be able to generate a cert for an IP.

Also I learnt that apiserver.advertiseAddress has to be the IP (as I changed it to the hostname as well in first try).

So now the kubernetes is up and running.

Thanks for pointing me to k3s anyway, I might give it a shot.

1 Like
8

For the sake of completenes, I found the GitHub issue, which helped me identify the “broken” files: https://github.com/NixOS/nixpkgs/issues/59364

1 Like