Help with self-hosting k3s images

Help
1

I want to self-host the nginx image on k3s. I only have one box, so k3s’ role is server. The documentation describes services.k3s.images as the best starting place, and helpfully suggests enabling airgapImages (although the k3s docs leave me confused about what to do with airgap).

After learning about nix-prefetch-docker and reading the code for pullImage, I have this, which works! It builds the image in the nix store and links the tarball in /var/lib/rancher/k3s/agent/images/. So far, so good…

{ config, lib, pkgs, ... }:
{
  imports =
    [
      ./webapp.nix
    ];

  services.k3s = {
    enable = true;
    images = [
      (pkgs.dockerTools.pullImage {
         imageName = "nginx";
         imageDigest = "sha256:d2c11a1e63f200585d8225996fd666436277a54e8c0ba728fa9afff28f075bd7";
         sha256 = "0h90vhv9n0is30b95v96jh54f3krfnm5ia6952yq41gwdvf7if69";
         finalImageName = "nginx";
         finalImageTag = "local";
      })

      config.services.k3s.package.airgapImages
    ];
  };
}

Next, I have this manifest. This currently also works. It port-forwards port 80 from the load balancer to localhost, and directs traffic to nginx which returns a 404 as it’s not serving anything. If, however, I change image = "nginx:latest"; to image = "nginx:local";, the pod errors when pulling the image. From looking at the logs, I can see it’s still trying to pull the image from dockerhub.

{ config, lib, pkgs, ... }:
{
  services.k3s.manifests.webapp = {
    content = [
      {
        apiVersion = "v1";
        kind = "Pod";
        metadata = {
          name = "webapp-nginx";
          labels = {
            "app.kubernetes.io/name" = "webapp";
            "app.kubernetes.io/component" = "nginx";
          };
        };
        spec = {
          containers = [
            {
              name = "nginx";
              image = "nginx:latest";
              ports = [
                {
                  containerPort = 80;
                  name = "http-web-svc";
                }
              ];
            }
          ];
        };
      }
      {
        apiVersion = "v1";
        kind = "Service";
        metadata = {
          name = "webapp-service";
        };
        spec = {
          selector = {
            "app.kubernetes.io/name" = "webapp";
          };
          ports = [
            {
              name = "http";
              port = 80;
              targetPort = "http-web-svc";
            }
          ];
        };
      }
    ];
  };
}

I appear to be missing a step between saving images locally (working), and telling k3s to pull the images locally (not working). Can someone offer advice?

2

I needed to add this to the container config:

imagePullPolicy = "Never";

And run the following commands (because nixos-rebuild would not automatically correct the pod:

kubectl delete pod webapp-nginx
kubectl apply -f /var/lib/rancher/k3s/server/manifests/webapp.yaml
3

It shouldn’t be necessary to set imagePullPolicy explicitly. The default of IfNotPresent will only pull the image if it isn’t already present on the node. However, I think k3s only imports container images at startup so you would need to restart k3s.service to trigger the import of a newly added image (not sure if a rebuild restarts k3s).

And run the following commands (because nixos-rebuild would not automatically correct the pod:

k3s will check the manifests every 15 seconds (see here), so maybe you need to wait a few seconds until the pod is updated but it should happen automatically.

NB: If you want to make sure that the pod uses the right image, you could get name and tag directly from the imported image:

{ pkgs, ... }:
let
  nginxImage = pkgs.dockerTools.pullImage {
    imageName = "nginx";
    imageDigest = "sha256:d2c11a1e63f200585d8225996fd666436277a54e8c0ba728fa9afff28f075bd7";
    sha256 = "0h90vhv9n0is30b95v96jh54f3krfnm5ia6952yq41gwdvf7if69";
    finalImageName = "nginx";
    finalImageTag = "local";
  };
in
{
  services.k3s.images = [ nginxImage ];
  services.k3s.manifests.webapp = {
    content = [
      {
        apiVersion = "v1";
        kind = "Pod";
        metadata = {
          name = "webapp-nginx";
          labels = {
            "app.kubernetes.io/name" = "webapp";
            "app.kubernetes.io/component" = "nginx";
          };
        };
        spec = {
          containers = [
            {
              name = "nginx";
              image = "${nginxImage.imageName}:${nginxImage.imageTag}";
              ports = [
                {
                  containerPort = 80;
                  name = "http-web-svc";
                }
              ];
            }
          ];
        };
      }
      {
        apiVersion = "v1";
        kind = "Service";
        metadata = {
          name = "webapp-service";
        };
        spec = {
          selector = {
            "app.kubernetes.io/name" = "webapp";
          };
          ports = [
            {
              name = "http";
              port = 80;
              targetPort = "http-web-svc";
            }
          ];
        };
      }
    ];
  };
}