Hi all!
The NixOS module for k3s has some nice options that allow to configure Kubernetes deployments directly in pure Nix! I would like to share some examples on how to use them and hope they help one or the other to build reproducible k3s nodes.
Deploy a Pod
Instead of setting up a cluster, writing YAML files and deploying them with kubectl apply, you
could write everything in your NixOS config and let the auto deploying manifests feature of k3s do the work. Rather than writing and applying the following YAML…
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
…you could use the following config and build the system.
services.k3s = {
enable = true;
manifests.nginx.content = {
apiVersion = "v1";
kind = "Pod";
metadata.name = "nginx";
spec.containers = [
{
name = "nginx";
image = "nginx:1.14.2";
ports = [ { containerPort = 80; } ];
}
];
};
};
This will install k3s and provision it with the nginx Pod. You should see the Pod running after a few seconds. Every Kubernetes resource, including CRDs, can be written in this style. However, this guide mainly uses Pods to keep it simple.
Multi document YAMLs
Multi-document YAML files are quite common in Kubernetes and can be effectively represented using a Nix list.
services.k3s.manifests.nginx.content = [
{
apiVersion = "v1";
kind = "Pod";
metadata = {
name = "nginx";
labels."app.kubernetes.io/name" = "proxy";
};
spec.containers = [
{
name = "nginx";
image = "nginx:stable";
ports = [
{
containerPort = 80;
name = "http-web-svc";
}
];
}
];
}
{
apiVersion = "v1";
kind = "Service";
metadata.name = "nginx-service";
spec = {
selector."app.kubernetes.io/name" = "proxy";
ports = [
{
name = "name-of-service-port";
protocol = "TCP";
port = 80;
targetPort = "http-web-svc";
}
];
};
}
];
Deploy big YAML files
If you have big YAML files and don’t want to translate them to Nix, you could also pass the path to the file.
services.k3s.manifests.nginx.source = ../nginx.yaml;
NB: The automated conversion from YAML over JSON to Nix works pretty well for big files.
Prefetch container images
All of the above will still pull container images at runtime, if they aren’t present on the node already. You can also embed images in your config and let k3s import them.
{ pkgs, ... }:
let
nginxImage = pkgs.dockerTools.pullImage {
imageName = "nginx";
imageDigest = "sha256:4ff102c5d78d254a6f0da062b3cf39eaf07f01eec0927fd21e219d0af8bc0591";
hash = "sha256-Fh9hWQWgY4g+Cu/0iER4cXAMvCc0JNiDwGCPa+V/FvA=";
finalImageTag = "1.27.4-alpine";
};
in
{
services.k3s = {
images = [ nginxImage ];
manifests.nginx.content = {
apiVersion = "v1";
kind = "Pod";
metadata.name = "nginx";
spec.containers = [
{
name = "nginx";
image = "${nginxImage.imageName}:${nginxImage.imageTag}";
ports = [ { containerPort = 80; } ];
}
];
};
};
}
The nix-prefetch-docker command is really helpful to get the information for pullImage.
nix run nixpkgs#nix-prefetch-docker -- --image-name nginx --image-tag 1.27.4-alpine
Deploy Helm charts
Similarly, you can deploy Helm charts.
services.k3s = {
autoDeployCharts.hello-world = {
name = "hello-world";
repo = "https://helm.github.io/examples";
version = "0.1.0";
hash = "sha256-U2XjNEWE82/Q3KbBvZLckXbtjsXugUbK6KdqT5kCccM=";
# configure the chart values like you would do in values.yaml
values = {
replicaCount = 3;
serviceAccount.create = false;
servcie.port = 8080;
};
};
};
NB: This only works on recent nixkgs revisions, previously the services.k3s.autoDeployCharts option didn’t exist. It is still possible to deploy Helm charts on older revisions, but you need to write more boilerplate, see this for an example.
Deploy secrets
The manifests you write will be placed in the Nix store, so you don’t want to write secrets directly into them. Instead, use a secret management tool like sops-nix.
{ config, ... }:
{
sops = {
secrets.super-secret-password = { };
templates.password = {
content = builtins.toJSON {
apiVersion = "v1";
kind = "Secret";
metadata.name = "password";
stringData.password = config.sops.placeholder.super-secret-password;
};
path = "/var/lib/rancher/k3s/server/manifests/password.json";
};
};
}
This will template a JSON manifest and replace the secret placeholder at activation time. K3s will then apply the secret like any other resource.
Airgap clusters
You can place all container images needed for your deployment on the node, however, k3s itself still needs some container images to work properly. Usually these will be pulled at runtime, but you can also provision nodes with the k3s images.
services.k3s.images = [ config.services.k3s.package.airgapImages ];
This will select the proper images based on the host system and the used k3s version. Being able to build air-gapped clusters is obviously handy if you want to run a cluster without an Internet connection, but it’s also really helpful for writing NixOS tests for Kubernetes services and clusters as a whole! In fact, this is done in nearly all the k3s tests.
Remove resources
You can remove resources that were previously installed by setting a corresponding --disable flag. Assuming you want to uninstall a manifest that was previously installed via services.k3s.manifests.nginx, you would set a --disable nginx flag.
services.k3s.extraFlags = [ "--disable nginx" ];
Note that this still needs you to specify services.k3s.manifests.nginx with the content you want to be uninstalled. Once the resources are uninstalled, you can remove the manifest together with the disable flag. This is unfortunately not very declarative and I suppose that this is required due to a bug in the k3s deploy controller, but that’s still uncertain.
A comprehensive example
This guide includes several small code snippets to demonstrate various possibilities. I’ve also created an example repository that puts everything together. It builds a cluster with a server and an agent node, each running a node exporter. The cluster also deploys a Prometheus and a Grafana instance pre-configured with datasources and dashboards. Additionally, it deploys the Grafana admin password via a secret and provisions the k3s token using sops-nix. The cluster runs within a NixOS test that can be accessed interactively. You can check it out here.