Firecracker or Kata on NixOS

#1

Hey,

Are folks doing interesting things with Firecracker or Kata and NixOS? Surely someone out there is running a kata k8s cluster? Maybe someone is running Nix-powered Firecracker images, or are mounting /nix/store and using super-slim or empty firecracker images?

Anyone want to share configs or ideas?

1 Like
#2

What I’d want is a join of Toolbox and Firecracker.

#3

I think @mic92 is working on something like that, but I can’t find it in the list of repos.

1 Like
#4

You can do that either with cntr using the containerd api: GitHub - Mic92/cntr: A container debugging tool based on FUSE
Or you need to wait until GitHub - Mic92/kvm-pirate: Attach to kvm-based VMs is functional

1 Like
#5

Soonish I want to fix the firecracker build so that we can build it from source.

#6

I’ve played around with setting up containerd + firecracker. But I don’t have a slick system for it yet. Here’s what I got so far: GitHub - MarcoPolo/firecracker-containerd-nix

1 Like
#7

I’m revisiting this. Was firecracker hard to build from source for some reason?

#8

Firecracker doesn’t support virtio and doesn’t seem to plan to anytime soon. And I found that my old Kata work is still on GitHub, though indeed quite old.

So here’s what I’m after now:

  • nomad (via bitte?)
  • kata containers (a containerd-compatible runtime that runs workloads under various KVM VMMs)
  • a fork of the container nomad task driver

where fork of nomad-containerd-driver does a few different things:

  • reads the image tag, recognizes some special url format _NIX_:/nix/store/..... (or something)
  • don’t fetch any image
  • instead, fetch the store path
  • creates a GC root for that path, in some dir that containerd manages for the task
  • adds a /nix:/nix mount to the container configuration for the task
  • tells containerd to setup an empty root for the container, with the /nix+other mounts

This gets a multi-node workfload scheduler, allows us to re-use existing tooling for maintaining signed nix caches on various cloud blob providers, avoids any images, avoids snapshots, theoretically leverage virtio for solid isolation and virtio-fs for performant sharing of the /nix/store with the guests.

TODO:

  • update all of my pkg/modules for the latest kata stuff
  • figure out state of containerd in nixpkgs, see how hard it is to teach it about a new runtime
  • evaluate how to run nomad in nixos
    • bitte? (very “mature”/“complicated”, tied to AWS, maybe over-kill, but maybe great?)
  • hack on the nomad-containerd-driver to add the small nix features

ASK:

  • if you’re interested in this, please reach out. Even if you’re just a potential user, the biggest thing I need right now is motivation. Someone to bounce ideas off of, or be excited with, or just asking “hey, did you make any progress”, would go a long way.
  • if this seems dumb, useless, or inadvisable for some reason, also let me know :slight_smile: :upside_down_face: .
1 Like
#9

Do you mean virtio-fs? Because it supports virtio fine. But yes katacontainer is the better choice if you go for that. They do a lot of performance optimization to improve virtiofs support in qemu. Also their 9p performance looks a lot better than what I am used to in NixOS tests. We should look into what they do there.

#10

Yeah I did mean virtio-fs for sharing /nix in. I’m also hoping to use the cloud-hypervisor backend rather than qemu, but we’ll see how it turns out.

#11

Update: https://github.com/colemickens/nixpkgs-kubernetes (ignore the name)

  • kata-agent built from source
  • kata-runtime built from source
  • initrd built from source
  • kernel built w/ some extra config (needs more extra config to not need modules)
  • containerd very hackily configured to know about the kata runtime
  • two temporary test derivations to test with, one of which calls ctr the way a real orchestrator might

Blocker:

  • qemu crashes with the args that kata-runtime passes to it

I’m currently looking into using cloud-hypervisor as the backend instead of qemu, but for some reason it requires an image rather than initrd and so is its own potential rabbit hole.

cc: @Mic92

edit: thanks to one of my favorite nix-ers, clever/cleverca22 for the stage-1 bits lifted from not-os.

3 Likes
#12

I rather cleanly packaged kata-containers + qemu now. I use the stock kernel + rootfs images. Configuration is mostly hard-coded but the setup works:

This is simple enough to be upstreamed to nixpkgs for people that just want the upstream kata-container experience.

1 Like