Nix-store-csi: Serve Nix closures to k8s pods

I’ve been working on a Nix CSI for kubernetes since NixCon when I realized that everyone seems to working towards killing OCI layers (and images) in Kubernetes deployments.

I’m hoping to get people to give it a spin and report back to me bugs, feature requests or general feedback. I don’t have a lot of data on how this scales and that makes it hard to decide whether to justify adding a feature or not (e.g. cache locality scheduling).

An LLM was used in the development of this project.

15 Likes

How does this compare to @Lillecarl’s nixkube (formerly nix-csi)?

Other relevant projects you may already be aware of:

(tied to containerd)

(tied to Podman/CRI-O and other projects based on container-libs)

1 Like

/cc @devusb, who at Planet Nix had given talks on this topic:

1 Like

I was unhappy with the design of nixkube (nix-csi). It’s written in Python and uses nix directly with a nix-daemon, and IMO I don’t think Nix belongs in the cluster.

  • I use Harmonia directly to do all the NAR stuff, which ends up being faster than nix, in addition to other clever optimizations I did with syncfs.
  • nix-store-csi assumes all the closures you will be fetching are available in the substituters you configured, unlike nixkube, which lets you eval/build on demand.
  • nix-store-csi does not connect to the cluster’s apiserver like nixkube does.
  • Pods get a read-only /nix/store with only their closure and no Nix database, while nixkube mounts /nix with a Nix database that can be writable.
  • Installing is one Helm chart, and the plugin runs as a DaemonSet. No cache StatefulSet, builders or SSH between pods.
  • When the closure is already on the node, a publish takes milliseconds and starts no processes. nixkube runs nix build on every publish.
  • Fetches never time out on kubelet. They keep running in the background, the pod’s events show progress, and kubelet’s retry joins the running fetch.
  • NAR unpacking runs in threads sandboxed with Landlock, so a bad NAR can’t write outside the plugin’s tmp/.
  • Every volume gets /nix/store/.roots/<name> links for each store path it lists, so one pod spec works on every architecture without hashes in args.

nix-store-csi has lots of other useful features like prefetching, node pools, a Grafana dashboard, and in the future I will support more storage models and kill NARs just like I killed OCI layers.

For nix-snapshotter, the README has a comparison: Why a CSI driver. (also nix-snapshotter is an unmaintained cargo-culted mess IMO)

cc. @Lillecarl plz correct me if I’m wrong

4 Likes

What if I want to run a builder as a pod?
E.g with nix-grpc-store.

I do think that not requiring nix on the cluster machines is a neat approach, especially if you aren’t running NixOS on your cluster machines (e.g. if you’re using Talos).
My main issue with these things is that I personally would like to run Nix-built containers on Podman, which AFAIK cannot use CSIs as easily as K8S/CRI-O/etc can. You can do manual unpack as shown in the README with this, but that’s not as convenient.

grpc-store is valid and it’s not the same as just nix with nixkube. grpc-store has nothing to do with the bullet point you’re quoting, it doesn’t work like that.

I personally have zero interest whatsoever in building images or podman, only running workloads on Kubernetes. CSI is k8s specific, snapshotters is containerd specific.

Nice! Does it do garbage collection on the nodes?

No, I was referring to running gRPC builders inside k8s as pods.
That requires the store to be writable AFAIK.

Yeah, feel free to go through the .md files in the repo and the helm chart.

But it does not require a nix csi and should not use nix csi for that probably.

You still need to get the gRPC server binaries in somehow though. Unless you skip the CSI stuff and just make a regular OCI image with all the store paths copied into the layers.

I just use upstream chart, but i see what you’re saying, maybe grpc store can have its store in a different directory? I still do not want to support s writeable store in the container had other applications in mind

I think you can actually use the CSI how it is now with grpc store, you just gotta mount the worker to --store 'local?root=/data' then create that writeable volume.
cc. @Mic92

1 Like

Honestly it’s probably an acceptable tradeoff that you can’t use this to make pod nix stores to run nix builds in, since not allowing writable stores probably simplifies the implementation a lot.
If we get good relocatable binaries, though, then it’s probably simple enough to just mount it somewhere else. Just pointing out the edge case here.

If you have a grpc store setup on the cluster could you try the solution I just thought of? I don’t think we need relocateable binaries for this.

I don’t actually run any of this setup, LOL. it was just a small edge case that I wanted to make sure you (were aware of/had any thoughts on). :wink:

Ok, gonna try it out, or get Mic92 to try it. But yeah I do think you probably don’t wanna use K8S for an build farm, you probably just want an autoscaler for longer-running VMs, you aren’t gaining that much by using pods.

I mostly agree on that. The nice thing about k8s is that you get a cloud-agnostic runtime, but if you have settled on 1 cloud you can just set up either the native autoscaling of your cloud, or kubevirt on your bare-metal setup.
Realistically that’s cleaner than running in pods anyhow.

1 Like

Okay, it does actually work, these are the main things you need to set:

  • nix-daemon: chmod 755 /data (nix-daemon refuses a chroot store whose root is world-writable) and store = local?root=/data
  • nix-grpc-daemon: --proxy-store=unix:///nix/var/nix/daemon-socket/socket?root=/data
  • both: an emptyDir at /nix/var, since the daemon socket stays there even with a chroot store

From docker.nix, list path and nixConf in storePaths and put its passwd and group in a ConfigMap. You can skip includeNixDB and gcRoot, since the tools aren’t in the store that builds write to. With the nix-store-csi runner image you can also skip Env and extraCommands, because that image already has a CA bundle, /tmp and /var/empty.

2 Likes

I forgot about chroot stores! Neat that this works.

1 Like

Out of curiosity: Maybe this is a newbie question but how do you make sure all store paths are actually present in the case (a) oci image built using nix (b) “regular” application referencing nix store paths, eg something that uses nix run/build in a container and needs /nix/store via the csi driver to actually work?