Nixifying Kubernetes with nix-csi, easykubenix and dinix

{
  pkgs ? import <nixpkgs> { },
}:
let
  sysMap = {
    "x86_64-linux" = "aarch64-linux";
    "aarch64-linux" = "x86_64-linux";
  };
  pkgsCrossish = import pkgs.path { system = sysMap.${builtins.currentSystem}; };

  # You can use flakes, npins, niv, fetchTree, fetchFromGitHub or whatever.
  easykubenix = builtins.fetchTree {
    type = "github";
    owner = "lillecarl";
    repo = "easykubenix";
  };
  ekn = import easykubenix {
    inherit pkgs;
    modules = [
      {
        kluctl = {
          discriminator = "demodeploy"; # Used for kluctl pruning (removing resources not in generated manifests)
          pushManifest = {
            enable = true; # Push manifest (which depends on pkgs.hello) before deploying
            to = "ssh://root@192.168.88.20"; # Shouldn't be root but here we are currently, maybe shouldn't be a module option either?
          };
        };
        kubernetes.resources.none.Pod.hello.spec = {
          containers = {
            _namedlist = true; # This is a meta thing to use attrsets instead of lists
            hello = {
              image = "quay.io/nix-csi/scratch:1.0.1"; # 1.0.1 sets PATH to /nix/var/result/bin
              command = [ "hello" ];
              volumeMounts = {
                _namedlist = true;
                nix.mountPath = "/nix";
              };
            };
          };
          volumes = {
            _namedlist = true;
            nix.csi = {
              driver = "nix.csi.store";
              volumeAttributes.${pkgs.system} = pkgs.hello; # this is stringified into a storepath,
              volumeAttributes.${pkgsCrossish.system} = pkgsCrossish.hello; # this is stringified into a storepath,
              # Now the manifest depends on pkgs.hello so when we push it we bring pkgs.hello and nix-csi can fetch it.
            };
          };
        };
      }
    ];
  };
in
ekn
nix run --file . deploymentScript
…n [☸ kubernetes-admin@shitbox]~/C/nix-csi/demo [🎋 main][!?][🗀 loaded/allowed][🐚fish]
[01:39:34]❯ nix run --file pod.nix deploymentScript
+ nix copy --to ssh://root@192.168.88.20 /nix/store/cr80hnrfl72nc943cr17v6fj2x0vpbaq-manifest.json
+ /nix/store/zpimjgx9k9drc2yvyx0v2kzbrds12y9z-kluctl-2.27.0/bin/kluctl deploy --no-update-check --target local --discriminator demodeploy --project-dir /nix/store/5jkji22hgsabx5qdrx19k7qzl2y8yp2r-kluctlProject
⚠ Failed to detect git project root. This might cause follow-up errors
✓ Initializing k8s client
✓ Rendering templates
✓ Rendering Helm Charts
✓ Building kustomize objects
✓ Postprocessing objects
✓ Writing rendered objects
✓ Getting remote objects by discriminator
✓ Getting 1 additional remote objects
✓ Getting namespaces
✓ prio-10: Nothing to apply.
✓ Finished waiting
✓ default: Applied 1 objects.

New objects:
  default/Pod/hello
✓ The diff succeeded, do you want to proceed? (y/N) y
✓ prio-10: Nothing to apply.
✓ Finished waiting
✓ default: Applied 1 objects.
✓ Writing command result

New objects:
  default/Pod/hello
…ubernetes-admin@shitbox]~/C/nix-csi/demo [🎋 main][!?][🗀 loaded/allowed][🐚fish][⏱ 2s]
[01:39:40]❯ kubectl logs pods/hello 
Hello, world!

There’s a bit of boilerplate and it’s still a moving target, if you have nix-csi installed this is how you’d deploy an application to your cluster in the current iteration of the project(s).
There’s more work to be done so it’s this easy for more usecases. This also ignores multi-arch clusters by only targeting pkgs.system but you could easily add a pkgsCross thingy. I want to make it easy to push to cachix, attic and S3 (with signing) and such too but this is the user experience using the builtin cache (exposed through Kubernetes as a loadbalancer on 192.168.88.20).

Edit: The deployment log is always quite verbose, kluctl isn’t really meant for “dump your manifests here” deployments, they wanna render kustomizations and helm charts and all the bits and bobs. I’ve discussed with the kluctl maintainer and he thinks it’s a good idea to separate the (great) deployment engine from the rendering bits, soon :tm:

3 Likes

Thanks @Lillecarl , this was quite insightful. Now I understand better

1 Like

The storepath rather than the expression example didn’t exist this Wednesday so I couldn’t have made that example until now :wink: Previously i was focused on building expressions in-cluster. It’s still a goal to have that featureset, now that both easykubenix and nix-csi are “maturing” from alpha1 into alpha2(ish) I realize it’s probably not going to be a widely used feature, building AOT is more important :smile:

Edit: I updated the example to show how to support a mixed arch cluster with this mode too :smile:

1 Like

I was able to run NixOS in unprivileged containers, though I had to mount /sys/fs/cgroup RW which is cursed. Luckily within a Kubernetes release or two we’ll have KEP5474 which should let the CRI setup a writable cgroupfs without security complications for us.

One big downside is still that it’s impossible to remount /nix/store RO in the guests so either the CSI gives RO mount and you won’t be able to run nix commands in the container, or you RW mount and /nix/store is writable for root applications :smile:

1 Like

Updates

Cache

nix-csi now bundles a StatefulSet that acts as a central cache. It’s just ssh-ng with the dumbest SSH key setup ever (Reusing the same keys like a madman).
WIP: A little patch to Lix that updates registrationTime in Nix database of packages as soon as they’re queried meaning we can garbage collect based on that registrationTime and keep the cache hot. (Kinda like attic but with just Nix and OpenSSH).

Builders

The cache maintains a list of all builders (CSI Pods) in /etc/nix/machines, with some SSH configuration on your client you’re now able to utilize all CSI Pods as your own builders, works with aarch64-linux and x86_64-linux so it can be your own little build cluster :smile:

CSI

Not much news here, it just works :tm:

Misc

There’s an undeploy option you can set in the easykubenix settings that’ll spawn a DaemonSet to clear the /var/lib/nix-csi off the host in case you want to get rid of the thing entirely, it is however important that you decommission any pods mounting nix-csi volumes before removing the CSI (unless you plan on reinstalling it) or Kubernetes will have stuck pods since there’s nothing to respond to NodeUnpublishVolume requests, you don’t want stuck pods :smile:

2 Likes