Kubernetes using multiple nodes with latest unstable

As I don’t think there’s much documentation out there on this, here is a short gist on how to get kube to work with the latest unstable channel,

On the master:


nix-channel --add https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz nixos
nix-channel --update

configuration.nix:

      kubernetes = {
        roles = ["master"];
        masterAddress = "nixos-k8s-master.core-vpn";
        apiserverAddress = "https://nixos-k8s-master.core-vpn:6443";
        easyCerts = true;
      };

then

nixos-rebuild boot
reboot

then fix kube-control-plane-online-pre-start:

systemctl status kube-control-plane-online.service

to get ^ the path to the systemd file,

and

mount -o remount,rw /nix/store

cat /nix/store/1ky3pama7byf28ypaqbcm0jjfijvpwiv-unit-script-kube-control-plane-online-pre-start
#! /nix/store/dax0d72jklvln5i3f3m2ikbp7zpij0z5-bash-4.4-p23/bin/bash -e
until curl -k -Ssf https://nixos-k8s-master.core-vpn:6443/healthz 
do
  echo curl -k -Ssf https://nixos-k8s-master.core-vpn:6443/healthz: exit status $?
  sleep 3
done

grab the apitoken on the master:

cat  /var/lib/kubernetes/secrets/apitoken.secret 
f3665d1e5a0b2706fa12c1807c28b48a

and reboot to make sure it’s all OK

on the nodes:

nix-channel --add https://github.com/NixOS/nixpkgs-channels/archive/nixos-unstable.tar.gz nixos
nix-channel --update

configuration.nix:

        kubernetes = {
        roles = ["node"];
        masterAddress = "nixos-k8s-master.core-vpn";
        apiserverAddress = "https://nixos-k8s-master.core-vpn:6443";
      };

then

nixos-rebuild boot
reboot

then fix kube-control-plane-online-pre-start:

systemctl status kube-control-plane-online.service

to get ^ the path to the systemd file,

mount -o remount,rw /nix/store


cat /nix/store/1ky3pama7byf28ypaqbcm0jjfijvpwiv-unit-script-kube-control-plane-online-pre-start
#! /nix/store/dax0d72jklvln5i3f3m2ikbp7zpij0z5-bash-4.4-p23/bin/bash -e
until curl -k -Ssf https://nixos-k8s-master.core-vpn:6443/healthz 
do
  echo "curl -k -Ssf https://nixos-k8s-master.core-vpn:6443/healthz: exit status $?"
  sleep 3
done


save the apitoken to the node:

echo f3665d1e5a0b2706fa12c1807c28b48a > /var/lib/kubernetes/secrets/apitoken.secret

and reboot to make sure it’s all OK

then on the master:

export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig 

kubectl run pause --image=google/pause --replicas=1



kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
pause-c4f4c8586-pgc55   1/1     Running   0          75s

1 Like

Need to look into submitting a PR to fix the broken kube-control-plane-online systemd script, but for now I’m really happy to have a multi-node k8s cluster on nixos.

Good to hear you like the module! This PR got merged to master today: https://github.com/NixOS/nixpkgs/pull/67563 . At least it should fix most of the systemd-dependency-related problems, including the control-plane-online issue. If not, please do tell. :slight_smile: