How do I set up k3s on NixOS?

zeno · September 12, 2024, 8:56pm

NixOS version: latest (24.05)

I’ve followed the instructions over at the Wiki/GitHub but I can’t seem to get it working (mainly because the instructions seem to be incomplete).

The config:

{ pkgs, lib, config, ... }:
{
  networking.firewall.allowedTCPPorts = [
    6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
    # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
    # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
  ];
  networking.firewall.allowedUDPPorts = [
    # 8472 # k3s, flannel: required if using multi-node for inter-node networking
  ];
  services.k3s.enable = true;
  services.k3s.role = "server";
  services.k3s.extraFlags = toString [
    # "--kubelet-arg=v=4" # Optionally add additional args to k3s
  ];

  virtualisation.containerd = {
    enable = true;
    settings =
      let
        fullCNIPlugins = pkgs.buildEnv {
          name = "full-cni";
          paths = with pkgs;[
            cni-plugins
            cni-plugin-flannel
          ];
        };
      in {
        plugins."io.containerd.grpc.v1.cri".cni = {
          bin_dir = "${fullCNIPlugins}/bin";
          conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
        };
        # Optionally set private registry credentials here instead of using /etc/rancher/k3s/registries.yaml
        # plugins."io.containerd.grpc.v1.cri".registry.configs."registry.example.com".auth = {
        #  username = "";
        #  password = "";
        # };
      };
  };
  # TODO describe how to enable zfs snapshotter in containerd
  services.k3s.extraFlags = toString [
    "--container-runtime-endpoint unix:///run/containerd/containerd.sock"
  ];
}

systemctl status k3s.service :

● k3s.service - k3s service
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2024-09-12 22:34:28 CEST; 4s ago
 Invocation: 6144d17d1c7f424185174559147112fa
    Process: 618540 ExecStartPre=/nix/store/hacpf7kdpwcijjmawr6cqpr3v746l07b-activate-k3s-content (code=exited, status=0/SUCCESS)
    Process: 618541 ExecStart=/nix/store/g06lx0ia3g2yrgvasqygbn4l61baf14r-k3s-1.30.3+k3s1/bin/k3s server --container-runtime-endpoint unix:///run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 618541 (code=exited, status=1/FAILURE)
         IP: 767.7K in, 768K out
   Mem peak: 550.5M
        CPU: 6.713s

(Excerpt from) journalctl -xeu k3s:

...
sep. 12 22:51:18 homeserver k3s[632680]: I0912 22:51:18.762164  632680 event.go:389] "Event occurred" object="kube-system/aggregated-metrics-reader" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="AppliedManifest" message="Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
sep. 12 22:51:18 homeserver k3s[632680]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
sep. 12 22:51:18 homeserver k3s[632680]: W0912 22:51:18.766052  632680 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
sep. 12 22:51:18 homeserver k3s[632680]: W0912 22:51:18.766279  632680 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
sep. 12 22:51:18 homeserver k3s[632680]: I0912 22:51:18.767585  632680 server.go:479] "Kubelet version" kubeletVersion="v1.30.3+k3s1"
sep. 12 22:51:18 homeserver k3s[632680]: I0912 22:51:18.767598  632680 server.go:481] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
sep. 12 22:51:18 homeserver k3s[632680]: W0912 22:51:18.767632  632680 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
sep. 12 22:51:18 homeserver k3s[632680]: W0912 22:51:18.767687  632680 feature_gate.go:246] Setting GA feature gate CloudDualStackNodeIPs=true. It will be removed in a future release.
sep. 12 22:51:18 homeserver k3s[632680]: I0912 22:51:18.768874  632680 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
sep. 12 22:51:18 homeserver k3s[632680]: Error: failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
sep. 12 22:51:18 homeserver k3s[632680]: time="2024-09-12T22:51:18+02:00" level=error msg="kubelet exited: failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
sep. 12 22:51:18 homeserver systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
sep. 12 22:51:18 homeserver systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
sep. 12 22:51:18 homeserver systemd[1]: k3s.service: Consumed 8.683s CPU time, 489.4M memory peak, 1.3M incoming IP traffic, 1.3M outgoing IP traffic.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit k3s.service completed and consumed the indicated resources.

If you need something else, please let me know.

Any help is greatly appreciated!

ookhoi · September 15, 2024, 1:03pm

A year ago this was all I needed:

services.k3s = {
    enable      =  true;
    clusterInit =  true;
    extraFlags  = "--cluster-cidr=10.42.0.0/16,2a10:3781:25ac:2::/64 --service-cidr=10.43.0.0/16,2a10:3781:25ac:3::/112 --flannel-iface eno1";
};

azazel75 · September 15, 2024, 7:14pm

zeno:

{ pkgs, lib, config, ... }:
{
  networking.firewall.allowedTCPPorts = [
    6443 # k3s: required so that pods can reach the API server (running on port 6443 by default)
    # 2379 # k3s, etcd clients: required if using a "High Availability Embedded etcd" configuration
    # 2380 # k3s, etcd peers: required if using a "High Availability Embedded etcd" configuration
  ];
  networking.firewall.allowedUDPPorts = [
    # 8472 # k3s, flannel: required if using multi-node for inter-node networking
  ];
  services.k3s.enable = true;
  services.k3s.role = "server";
 }

If you just need a one node setup just use the configuration above. You need the external containerd only if you need a non stantard configuration , like for example having the /var on zfs and so on. Containerd is already integrated into the k3s executable.

Magicloud · November 8, 2024, 8:12am

The kubelet error indicated that “cri” plugin was not running. You can confirm it with ctr -a /run/containerd/containerd.sock plugin ls. Then try journalctl -e -u containerd.service and see what is the problem.

zeno · November 9, 2024, 7:08pm

@Magicloud @ookhoi @azazel75
Thank you all for the feedback; I forgot that I solved this: the issue was that I had to perform containerd config default > /etc/containerd/config.toml

@azazel75

That is exactly why I needed an external containerd I’m using ZFS.

As an addendum for anyone else stumbling across this thread: You will also need to edit the file that I mentioned. Find the line that says snapshotter under the category [plugins."io.containerd.grpc.v1.cri".containerd] and set it to "zfs". Like below:

[plugins."io.containerd.grpc.v1.cri".containerd]
# Other configuration options
snapshotter = "zfs"
# Other configuration options

Magicloud · November 10, 2024, 4:12pm

This part of configuration is mentioned indirectly in K3s - NixOS Wiki, following the link to Github, then the section about storage.

zeno · November 10, 2024, 8:46pm

Yeah, I noticed, but the information did seem incomplete. I come from Ubuntu with Microk8s, which is already configured (sort of), so I didn’t know how to perform that task manually.

By the way, can you tell me whether this is valid or not?

...
  virtualisation = {
    # k3s-related; more info at https://github.com/NixOS/nixpkgs/blob/34ed0c9cc1bb45c7cbfda050236282e0023d25fb/pkgs/applications/networking/cluster/k3s/docs/examples/STORAGE.md
    containerd = {
      enable = true;

      settings =
        let
          fullCNIPlugins = pkgs.buildEnv {
            name = "full-cni";
            paths = with pkgs; [
              cni-plugins
              cni-plugin-flannel
            ];
          };
        in
        {
          plugins."io.containerd.grpc.v1.cri" = {
            cni = {
              bin_dir = "${fullCNIPlugins}/bin";
              conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d/";
            };

            containerd = {
              snapshotter = "zfs";
            };
          };
        };
    };
  };
...

I tried nuking the configuration and test by adding the containerd = { snapshotter = "zfs"; }; but the file does not seem to get edited (opening the file shows that the line has not been edited).

Magicloud · November 11, 2024, 3:00pm

I do not have the snapshotter = "zfs" line. Containerd should detect it by itself. Do you have this ZFS filesystem, it must be at that place. “raid/k3s 89.6M 1.92T 240K /var/lib/containerd/io.containerd.snapshotter.v1.zfs”