Adding flake protocol support to nix-snapshotter

Development Deployments
1

I’ve been working on adding flake URL support to nix-snapshotter and opened a PR upstream, but it’s been sitting without review for a while. Figured I’d share it here and see what people think.

The existing nix:0 prefix requires the image to already be built in the local Nix store, which is fine for single-node setups but doesn’t really work in multi-node K8s clusters — you’d have to distribute the build result to every node somehow. So I added a way to encode flake URLs into OCI image references, letting each node fetch and build the image on its own.

In a pod spec it looks like:

containers:
  - name: redis
    image: flake-git-ssh:0/git--at--github.com/user/redis-image--q--ref--eq--main

This gets decoded to git+ssh://git@github.com/user/redis-image?ref=main internally. OCI refs only allow [a-z0-9._-], so special characters need escaping — I went with --at-- for @, --q-- for ?, --eq-- for =, etc. It’s a bit ugly, admittedly.

There’s also a small CLI to do the conversion:

bash

nix run github:bglgwyng/nix-snapshotter#encode-flake-ref -- 'github:user/repo'
flake-github:0/user/repo

Design doc: architecture.md — Flake Protocols

Any feedback welcome — especially on whether the encoding scheme makes sense or if there’s a less awkward way to do it.

1 Like
2

The way to distribute the stora paths is to have a central cache, nix-snapshotter will first check all substituters. I’m not sure whether flake refs in images is a good idea, kubernetes is already and adding eval time onto that seems like a horrible idea

3

I assumed flake eval cost would be acceptable, and your concern is fair. But the deeper issue is that a bare nix:0 store path can’t guarantee a successful pull — even on a single node, a nix store gc while a container is down means the kubelet can’t bring it back, and an opaque store hash gives no hint on how to recover. It’s a rare scenario, but I wanted a reliable answer for it. One idea: keep nix:0/ as-is but allow a flake ref in pod metadata as a recovery path. Curious if you have other thoughts.

4

What im planning to do with hook GC roots onto kubernetes somehow. So it doesnt GC anything declared in deployment, sts and such. Dont ask me specifics yet, but that wasy idea.

1 Like