Trying to run kubernetes in a single node mode

Help
1

Hi,

I am trying to run kubernetes on nixos in a single node mode. I have added the following to my configuration.nix:

  services.kubernetes = {
    roles = ["master" "node"];
  };

when I run nixos-rebuild, I do get kubectl executable and I notice that a bunch of new systemd services have been loaded:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

However, I cannot create any persistent volumes or schedule any pods and my cluster seem to have no nodes:

kubectl get nodes
# No resources found.

Also I can notice that the kublet service has failed loading

 kubelet.service - Kubernetes Kubelet Service
   Loaded: loaded (/nix/store/wgd21w9frc9zs9x8905z3zcnhzz16yam-unit-kubelet.service/kubelet.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-02-17 09:13:34 GMT; 7min ago
  Process: 1760 ExecStart=/nix/store/bnrn04kn7bf3gjzp9di7nrgmn1y22j1c-kubernetes-1.13.3/bin/kubelet --kubeconfig=/nix/store/99xycfah58lrhimynja1dyqkgrylzygd-kubelet->
 Main PID: 1760 (code=exited, status=255)
      CPU: 94ms
[...]
Feb 17 09:13:34 antanix kubelet[1760]: I0217 09:13:34.311412    1760 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Feb 17 09:13:34 antanix kubelet[1760]: F0217 09:13:34.311580    1760 server.go:261] failed to run Kubelet: Running with swap on is not supported, please disable swap>
Feb 17 09:13:34 antanix systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Feb 17 09:13:34 antanix systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 17 09:13:34 antanix systemd[1]: kubelet.service: Consumed 94ms CPU time

Is there a nixos option to disable swap? Also, Am I supposed to do add any other configuration step in order to configure a single node setup (similar to minikube?)

1 Like
2

By investigating the error a bit more, I have found this issue is unrelated to nixos, and is due to my workstation setup not being fully supported e.g. see Why disable swap on Kubernetes.

However, I have also found through a related post that the error can be suppressed by setting the following kublet option

  services.kubernetes.kubelet.extraOpts = "--fail-swap-on=false";
3

I’d also ask/suggest testing with the new k8s refactor: https://github.com/NixOS/nixpkgs/pull/45670

It has been a big help for my setup.

2 Likes
4

I too was trying to use kubernetes on nixos multinode and I ran into some issues. It would be killer if it just “worked”. How simple is the basic single node setup? What besides enable = true needs to be set?

5

I run it like this (using johanot’s PR: https://github.com/johanot/nixpkgs/archive/b3905f0d224bc5c2da3572f7017f05a27df742cd.tar.gz;)

   networking.extraHosts = "${config.networking.privateIPv4} api.kube";
    services.kubernetes = {
      easyCerts = true;
      addons.dashboard.enable = true;
      roles = ["master" "node"];
      apiserver = {
        securePort = 443;
        advertiseAddress = config.networking.privateIPv4;
      };
      masterAddress = "api.kube";
    };
    services.dockerRegistry.enable = true;
    environment.systemPackages = with pkgs; [
      kompose kubectl
      vim
    ];
1 Like
6

@tom thanks for pointing my at the PR and sharing your setup. One quick question: how do you point nixos-rebuild to your local clone of the PR branch? I currently use the following for nixpkgs-unstable.

nixos-rebuild switch --upgrade -I nixpkgs=channel:nixpkgs-unstable

thanks!

7

I am using NixOps with nixpkgs pinned. I have a modified management nix-shell script:

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p nixopsUnstable which jq nix openssl openssh
#! nix-shell -p "(import ./nixpkgs.nix)"
...

I can replace the last line with:

#! nix-shell -I nixpkgs=some-path

to point to a local clone.

[1] https://github.com/tomberek/nixops-manager/blob/580eb161d7d331d3812ec9f69839549275524a81/manage

8

Thanks, will try to reproduce your setup on nixos. that easyCerts = true; option looks very handy!

2 Likes
9

Thank you for testing this! Please let me know if you have any questions or issues. I’m keen on getting this mergeable for 19.03.

2 Likes