Nix-snapshotter: Native understanding of Nix packages for containerd

Announcements
1

Hello! This is Edgar and Robbie, and we’re excited to share with you the public release of nix-snapshotter! nix-snapshotter brings native understanding of Nix packages to containerd, the industry standard container runtime.

We built this because Nix is a great fit for making efficient containers. They don’t need an OS because Nix captures all dependencies exactly. However, the current process of creating Nix images is subpar because one needs to transform Nix packages into a format that container runtimes understand.

Using nix-snapshotter, instead of downloading image layers, packages come directly from the Nix store. Packages can be fetched from a binary cache or built on the fly if necessary. All existing non-Nix images continue to be supported, and Nix layers can be interleaved with normal layers.

nix-snapshotter also provides a CRI image service, which allows Kubernetes to resolve image manifests from Nix directly too. This enables for the first time, fully declarative Kubernetes resources, all the way down to the image specification and its contents. With this, you can even run pure Nix images without a Docker Registry at all, if you wish.

We’d love for you to try it out, there is a one-liner for Nix users to boot a VM with everything pre-configured: GitHub - pdtpartners/nix-snapshotter: Brings native understanding of Nix packages to containerd

nix run "github:pdtpartners/nix-snapshotter#vm"
nixos login: root # (Ctrl-a then x to quit)
Password: root

# Running `pkgs.hello` image with nix-snapshotter
nerdctl run ghcr.io/pdtpartners/hello

# Running `pkgs.redis` image with kubernetes & nix-snapshotter
kubectl apply -f /etc/kubernetes/redis/

# Wait a few seconds... 
watch kubectl get pods

# And a kubernetes service will be ready to forward port 30000 to the redis
# pod, so you can test it out with a `ping` command
redis-cli -p 30000 ping

See corresponding HackerNews discussion.

42 Likes
2

The link that probably everybody in here is looking for:

This is great stuff!

Have you been in contact with the OCI committee? About a year ago similar ideas where tossed around. If not, it might be worthwhile to present there. Folks seemed pretty interested and open to those ideas.

A long term goal could be to make this a recommended extension of the runtime spec.

Edit: I also think nlewo/nix2container may deserve some honorary mention in the “how is this different” section. That project move the goal post of more efficient OCI workflows by one important marginal step for ~2 years now.

3 Likes
3

Thank you! That’s a good idea, we’ll see if they are interested in a presentation. We have been to containerd meetings to give them a heads up and made several upstream contributions but that is separate from the OCI committee.

I think it will be beneficial to have it documented/recommended, but I don’t think we should modify the runtime spec. Adding a new media type suggested in your issue will require native support in all the container runtimes, whereas in our design it works with off the shelf containerd/Kubernetes.

I’ve opened a PR to add a FAQ entry for nlewo/nix2container, appreciate your feedback! Add FAQ entry for nix2container by elpdt852 · Pull Request #82 · pdtpartners/nix-snapshotter · GitHub

1 Like
4

Can I use this without Kubernetes? I mean running a container on nixos with my configuration.nix.

5

containerd is the default backend for docker, so yes via virtualisation.oci-containers.containers.

1 Like
6

I just tried with virtualisation.oci-containers.containers and I got docker: no matching manifest for linux/amd64 in the manifest list entries..

Note that I’m building the image on another computer and I push it to a docker registry.

nerdctl run image seems to fetch the image, but doesn’t find the paths in the nix store (but that’s for my next question).

7

Yes, the NixOS and home-manager modules both set up containerd with nix-snapshotter without Kubernetes, see installation steps. For example with the NixOS module, all you need is services.nix-snapshotter.enable = true and it’ll configure containerd for you as well.

I don’t recommend virtualisation.oci-containers.containers since it only works with docker or podman backend, and we don’t have support for either atm. See tracking issue

3 Likes
8

Beside virtualisation.oci-containers.containers, is there anything I can put in my configuration.nix to run a container with containerd?

9

Have you read through the installation steps? You need to set the following:

          services.nix-snapshotter = {
            enable = true;
            # Sets CONTAINERD_SNAPSHOTTER=nix
            setContainerdSnapshotter = true;
          };

          # A CLI for containerd, so you can run `nerdctl run --rm ghcr.io/pdtpartners/hello`
          environment.systemPackages = [ pkgs.nerdctl ];
10

Yes I already have that.

My understanding is that I can only start a container with nerdctl.

I was wondering if I was able to “start” a container declaratively using my config, like with virtualisation.oci-containers.containers.

11

Ah sorry, I misunderstood. We don’t have declarative containers yet, but please do file an issue we can track this. Welcome any ideas or contributions.

For this feature in particular, it’s more about containerd than nix-snapshotter. Perhaps this just involves adding containerd support to the virtualisation.oci-containers.containers backends.

1 Like
12

Gotcha, thanks. Yeah that makes sense.

Does running nerdctl run <image> automatically fetch nix store path from my configured substituters? It didn’t seem to be the case, but maybe I needed to restart something. (note that I built the image on another computer, so the files are missing from the nix store).

13

Yes, it’s meant to run nix build <store-path> under the hood, so it should use your configured substituter. You can check journalctl -u nix-snapshotter to see the logs if something is going wrong.

14

First off, incredible work! The Nix store always should have meshed with containers like this.

Is support for CRI-O / Podman feasible? The closest work I could find is containers/podman#4739: Remote snapshotter in podman, which supports the Stargz Store CRI-O plugin. I’d absolutely love to use nix-snapshotter with Quadlets for local, rootless .container and .kube systemd units. (I don’t know of an equivalent systemd generator for k3s.)

EDIT: I’ve opened an issue for this: pdtpartners/nix-snapshotter#103: [Feature request] CRI-O support (CRI-O driver)

15

CRI-O support is definitely feasible like how it is for stargz-snapshotter, we could definitely use the help to gain this capability.

1 Like
16

We just opened a PR to package nix-snapshotter in nixpkgs. The plan is to upstream the NixOS / Home Manager modules to make it dead easy to get started with nix-snapshotter: nix-snapshotter: init at 0.2.0 by elpdt852 · Pull Request #289682 · NixOS/nixpkgs · GitHub

5 Likes