Declaring k3s with helm and cluster

Help
8

Would be nice to declare the helm charts in the nix config.

This is possible with the k3s module, however, you need to create the manifest for the k3s Helm controller manually. For example to deploy the bitnami nginx chart:

  services.k3s = {
    enable = true;
    manifests.nginx.content = {
      apiVersion = "helm.cattle.io/v1";
      kind = "HelmChart";
      metadata = {
        name = "nginx";
        namespace = "kube-system";
      };
      spec = {
        targetNamespace = "test";
        createNamespace = true;
        repo = "https://charts.bitnami.com/bitnami";
        chart = "nginx";
        version = "18.3.5";
        # configure the chart values like you would do in values.yaml
        valuesContent = ''
          replicaCount: 3
          tls:
            enabled: false
          metrics:
            enabled: true
        '';
      };
    };
  };

See Helm | K3s for more information.

The deployment can be updated by changing the configuration accordingly and doing nixos-rebuild, but be aware that removing the manifest from your config and rebuilding the system will not uninstall the chart. You need to have the chart in your configuration and additionally set services.k3s.extraFlags = [ "--disable nginx" ] to actively uninstall the chart.

NB: The above will download the chart at runtime. You can also prefetch the chart during build and place it on the filesystem so that k3s can pick it up:

  services.k3s =
    let
      nginxChart =
        pkgs.runCommand "nginx-chart"
          {
            nativeBuildInputs = with pkgs; [
              kubernetes-helm
              cacert
            ];
            outputHashAlgo = "sha256";
            outputHash = "sha256-e4zlCaK9mioU9A0Wr2YqCxwzlnS+ssPG46ixvFVXOqk=";
          }
          ''
            export HOME="$PWD"

            helm repo add repository https://charts.bitnami.com/bitnami
            helm pull repository/nginx --version 18.3.5
            mv ./*.tgz $out
          '';
    in
    {
      enable = true;
      charts.bitnamiNginx = nginxChart;
      manifests.nginx.content = {
        apiVersion = "helm.cattle.io/v1";
        kind = "HelmChart";
        metadata = {
          name = "nginx";
          namespace = "kube-system";
        };
        spec = {
          targetNamespace = "test";
          createNamespace = true;
          # the chart name (bitnamiNginx) has to match the key that is used in services.k3s.charts
          chart = "https://%{KUBERNETES_API}%/static/charts/bitnamiNginx.tgz";
          valuesContent = ''
            replicaCount: 3
            tls:
              enabled: false
            metrics:
              enabled: true
          '';
        };
      };
    };

Similarly, you could prefetch container images and place them in services.k3s.images for k3s to pick them up, so you only need to download things at build time.

Edit: Be careful to not place secrets directly in valuesContent as they will show up in the Nix store

2 Likes