Are folks doing interesting things with Firecracker or Kata and NixOS? Surely someone out there is running a kata k8s cluster? Maybe someone is running Nix-powered Firecracker images, or are mounting /nix/store and using super-slim or empty firecracker images?
Firecracker doesn’t support virtio and doesn’t seem to plan to anytime soon. And I found that my old Kata work is still on GitHub, though indeed quite old.
So here’s what I’m after now:
nomad (via bitte?)
kata containers (a containerd-compatible runtime that runs workloads under various KVM VMMs)
a fork of the container nomad task driver
where fork of nomad-containerd-driver does a few different things:
reads the image tag, recognizes some special url format _NIX_:/nix/store/..... (or something)
don’t fetch any image
instead, fetch the store path
creates a GC root for that path, in some dir that containerd manages for the task
adds a /nix:/nix mount to the container configuration for the task
tells containerd to setup an empty root for the container, with the /nix+other mounts
This gets a multi-node workfload scheduler, allows us to re-use existing tooling for maintaining signed nix caches on various cloud blob providers, avoids any images, avoids snapshots, theoretically leverage virtio for solid isolation and virtio-fs for performant sharing of the /nix/store with the guests.
TODO:
update all of my pkg/modules for the latest kata stuff
figure out state of containerd in nixpkgs, see how hard it is to teach it about a new runtime
evaluate how to run nomad in nixos
bitte? (very “mature”/“complicated”, tied to AWS, maybe over-kill, but maybe great?)
hack on the nomad-containerd-driver to add the small nix features
ASK:
if you’re interested in this, please reach out. Even if you’re just a potential user, the biggest thing I need right now is motivation. Someone to bounce ideas off of, or be excited with, or just asking “hey, did you make any progress”, would go a long way.
if this seems dumb, useless, or inadvisable for some reason, also let me know .
Do you mean virtio-fs? Because it supports virtio fine. But yes katacontainer is the better choice if you go for that. They do a lot of performance optimization to improve virtiofs support in qemu. Also their 9p performance looks a lot better than what I am used to in NixOS tests. We should look into what they do there.
Yeah I did mean virtio-fs for sharing /nix in. I’m also hoping to use the cloud-hypervisor backend rather than qemu, but we’ll see how it turns out.
kernel built w/ some extra config (needs more extra config to not need modules)
containerd very hackily configured to know about the kata runtime
two temporary test derivations to test with, one of which calls ctr the way a real orchestrator might
Blocker:
qemu crashes with the args that kata-runtime passes to it
I’m currently looking into using cloud-hypervisor as the backend instead of qemu, but for some reason it requires an image rather than initrd and so is its own potential rabbit hole.