Bring nix build to moby/buildkit (or leap-frog to a CRI shim for a rootfs composer drawing from a SAN-attached nix store)

Since seeing @grahamc 's talk at NYLUG, I was wondering together with @DavHau for ways how to bring the advantages of nix container builds to and things like GitHub - GoogleContainerTools/kaniko: Build Container Images In Kubernetes.

Maybe GitHub - moby/buildkit: concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit can be leveraged?

Repeating the talk’s title “sneaking in nix” - would it be supported through buildkit a majority section of the container ecosystem would gain immediate access to the advantages of nix builds over some more dumb container specifications, such as dockerfiles. (watch the talk!)

@grahamc What do you think? Is this possible and worth trying?

See also:

EDIT: As @DavHau brought up: some packages have dependencies on VMs. :confused: - unclear how this fits into a kaniko or windows containerized build.

1 Like

Not sure if you’ve seen GitHub - tazjin/nixery: Container registry which transparently builds images using the Nix package manager. Canonical repository is
It’s a really cool merger between Nix & “Docker” (OCI Images)

Thanks for the link, I had it in the back of my mind but didn’t remenber the name!

You may also be interested in the talk I gave about Nixery, which goes into some detail about the layering strategy used (and why it’s better than what Docker & friends do).

My current plan (barring time constraints, such as working on my other projects, or working on my actual job :slightly_smiling_face:) is to extract the Nixery functionality into a CLI tool. (See nixery#73).

There are also ways of significantly reducing the dependency on the actual Nix binary (also a good idea for security reasons), if we restrict this to packages that have already been built - that would make it much easier to integrate into other systems, too.

One other change that stands in the way of this is moving the generation of manifests and other metadata to ggcr.

1 Like

This probably isn’t quite on-topic. I don’t have the best handle on the tendrils of the virtualization ecosystem, but I spent some time recently on a frustrating search for any sane cross-platform abstraction layer I could use to sandbox a simple shell-script ~fuzzing tool.

I’d roughly prefer not to use Docker, but the gravity of anything usable cross-platform kept pulling me back there. At one point I stumbled on an interesting project (GitHub - ottomatica/slim: Build and run tiny vms from Dockerfiles. Small and sleek.) for converting Docker containers into a VM that can supposedly run on hyperkit, kvm, and virtualbox. Here’s the description:

slim will build a micro-vm from a Dockerfile. Slim works by building and extracting a rootfs from a Dockerfile, and then merging that filesystem with a small minimal kernel that runs in RAM.

This results in a real VM that can boot instantly, while using very limited resources. If done properly, slim can allow you to design and build immutable unikernels for running services, or build tiny and embedded development environments.

It seemed like a dead-end for my case (on first read I assumed this was an arbitrary Docker container–but after getting an example working it seems that there are some specific demands on the container/Dockerfile), but it still felt like a novel/interesting approach so I’ve been keeping my eyes peeled for chances to mention it…

RE: @abathur : or maybe better GitHub - weaveworks/ignite: Ignite a Firecracker microVM


With Ignite, you pick an OCI-compliant image (Docker image) that you want to run as a VM, and then just execute ignite run instead of docker run . There’s no need to use VM-specific tools to build .vdi , .vmdk , or .qcow2 images, just do a docker build from any base image you want (e.g. ubuntu:18.04 from Docker Hub), and add your preferred contents.

@tazjin I’m catching up with the directions your pointing at, thanks!

In my eyes, it looks like moby/buildkit required not only plugable frontends but plugable builders, too. I wouldn’t want to give up on the possibility to build custom stuff.

I wonder if how buildkit’s llb and nix’s drv compare conceptually - might look into that on occasion.

As for caching, do you avoid in nixery the problem of buildLayeredImage, that although it caches the nix builds, it does not cache the created tar resulting in rebuilds / reloads like this:

uilding '/nix/store/wk5wbzcij03d4gxh8ghwi5qp4hhhsmna-stream-ae-dir-web2ldap.drv'...
building '/nix/store/8y0kfgdwg7z9d799m407zsn8n15brbc8-ae-dir-web2ldap.tar.gz.drv'...
Creating layer 0 from paths: ['/nix/store/r2nywq3ziag55zi6dqcxkpb6yla044kq-libunistring-0.9.10']
Creating layer 1 from paths: ['/nix/store/arb8311fjm1dsrbsy8j7pdanwnz1qwxv-libidn2-2.3.0']
Creating layer 2 from paths: ['/nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31']
Creating layer 3 from paths: ['/nix/store/lxbxwg609cw3zby7ba78h0992hp9rcfy-zlib-1.2.11']
Creating layer 4 from paths: ['/nix/store/s2kyldk2s42n5z2ijjj1v5ns78n9wzr3-ncurses-6.2']
Creating layer 5 from paths: ['/nix/store/f8lgjhnkd140789llhcgp21l800gw7a4-libXau-1.0.9']
Creating layer 6 from paths: ['/nix/store/1226byrsysrk7bc58amxhgxryshpqhvi-libXdmcp-1.1.3']
Creating layer 7 from paths: ['/nix/store/j10bd2d07crrnnpx0d9x2xbp3z4m86ql-libffi-3.3']
Creating layer 8 from paths: ['/nix/store/a0jxdf0wc13l4270fgq9n1f42x45pq69-openssl-1.1.1g']
Creating layer 9 from paths: ['/nix/store/6737cq9nvp4k5r70qcgf61004r0l2g3v-bash-4.4-p23']
Creating layer 10 from paths: ['/nix/store/jgfa1mslsw5ag1mgr5bggi1f30nn5zam-bzip2-']
Creating layer 11 from paths: ['/nix/store/pg71kfxk5wh3g9djymgm7v55b3hs29bx-expat-2.2.8']
Creating layer 12 from paths: ['/nix/store/vfx4kcld2gfv5svzcfap8syqg7zdlcyb-gdbm-1.18.1']
Creating layer 13 from paths: ['/nix/store/v7b326xn4wq9n4wqjdxar3xadz73mi8f-readline-6.3p08']
Creating layer 14 from paths: ['/nix/store/f7lws879jarjs40qznw95pwz4ngc7ib0-sqlite-3.32.3']
Creating layer 15 from paths: ['/nix/store/5nq3a165axi47y1w11c71ks4cndr9gkm-xz-5.2.5']
Creating layer 16 from paths: ['/nix/store/rzh0rwc1fdw6jlip7hhlxy3z5rhywv1y-libxcb-1.13.1']
Creating layer 17 from paths: ['/nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5']
Creating layer 18 from paths: ['/nix/store/5r6j0lmhgyn3gf7hz6lv18174l3q3hzl-libX11-1.6.8']
Creating layer 19 from paths: ['/nix/store/28dwzmv1nmj15s0qk336am74qd03r2x2-libXext-1.3.4']
Creating layer 20 from paths: ['/nix/store/5gdc4wn4nx74dx5mv2k3gqx8hh4q72mg-pcre-8.44']
Creating layer 21 from paths: ['/nix/store/pbiv2l9lpfnynp6cn9d2v5cwm0ms8ncy-util-linux-2.35.2']
Creating layer 22 from paths: ['/nix/store/ms1cwiwzrzi3q6bmmsd8q5z6acsv999x-libICE-1.0.10']
Creating layer 23 from paths: ['/nix/store/v589pqjhvxrj73g3r0xb41yr84z5pwb7-gcc-9.3.0-lib']
Creating layer 24 from paths: ['/nix/store/bn4hydrywh61v2n7ql7madahvg258mfk-libglvnd-1.3.2']
Creating layer 25 from paths: ['/nix/store/fx2z753q11pr201pfmn1v83d94px91l0-libselinux-2.9']
Creating layer 26 from paths: ['/nix/store/8ffpsn7k69i94yhss6bnk70wqwsz2xzr-python3.8-pycparser-2.20']
Creating layer 27 from paths: ['/nix/store/s93vnglfz8zc75i5i4pqz721g2jkzvw5-ncurses-6.2-abi5-compat']
Creating layer 28 from paths: ['/nix/store/mazsps24z01540n3g01m8yf52wm6avkd-glib-2.64.4']
Creating layer 29 from paths: ['/nix/store/zb51anmsww742hsdy36np8wxr2iapr29-libGL-1.3.2']
Creating layer 30 from paths: ['/nix/store/l70mpys52kgkr6xn3rw65bshbsl341sc-libSM-1.2.3']
Creating layer 31 from paths: ['/nix/store/fqsnky9lac96al4189ws9vyn3k5w65sf-libXrender-0.9.10']
Creating layer 32 from paths: ['/nix/store/qjs7rs88flsyhjq1yjlqvipg9i0xw1bz-python3.8-cffi-1.14.2']
Creating layer 33 from paths: ['/nix/store/n4zhi9fx42390ba2f9hhvh6cf3a6b6ng-python3.8-six-1.15.0']
Creating layer 34 from paths: ['/nix/store/743yww4grp6622cfgmg9mpzh8li04s8m-python3.8-pyasn1-0.4.8']
Creating layer 35 from paths: ['/nix/store/hmkzi186wfsn89cqvhcx6k8hl3qhsq10-db-5.3.28']
Creating layer 36 from paths: ['/nix/store/j9cfsz1zr44sfc8bnv8f4xjrlb8bla6f-keyutils-1.6.1-lib']
Creating layer 37 from paths: ['/nix/store/ll5wz6bxr7s7cjgkrv4kc8vwmfix8bcw-libsodium-1.0.18']
Creating layer 38 from paths: ['/nix/store/digkwchx8xgmsks92b86cll659yzhfh5-libkrb5-1.18']
Creating layer 39 from paths: ['/nix/store/4nrmshrb6ch33671nphkqwf1qisy3iya-python3.8-cryptography-3.0']
Creating layer 40 from paths: ['/nix/store/rpgik653k5r1y9526nm2cgl6514rdjgx-python3.8-setuptools-47.3.1']
Creating layer 41 from paths: ['/nix/store/mn6n2yxayszqf5ynxyqik4jx4g9508wi-cyrus-sasl-2.1.27']
Creating layer 42 from paths: ['/nix/store/6r4q34d8ra1dr77w3ich0wk27pls5kyg-libtool-2.4.6-lib']
Creating layer 43 from paths: ['/nix/store/gg3c5plbaz2n6mfba14v5p6ry3iap1l2-python3.8-bcrypt-3.2.0']
Creating layer 44 from paths: ['/nix/store/75wh7jcc9gl2hikqq7lgs32fa53vd3kg-python3.8-pyasn1-modules-0.2.8']
Creating layer 45 from paths: ['/nix/store/s5kha4gfpj5f833lxbrq4x6nqi61c9c7-python3.8-pynacl-1.4.0']
Creating layer 46 from paths: ['/nix/store/r5zff1swk044mjcx69k141i7f28rkn4s-openldap-2.4.50']
Creating layer 47 from paths: ['/nix/store/015gqh818gsv5x9wmcpkzf7l9psxw900-python3.8-cryptography-3.0-dev']
Creating layer 48 from paths: ['/nix/store/vl70smqx22pxna273bxf5h9gdfl81zpa-python3.8-asn1crypto-1.4.0']
Creating layer 49 from paths: ['/nix/store/8h22lf49zcaazxal70qz80adzjiy8z8v-python3.8-dnspython-2.0.0']
Creating layer 50 from paths: ['/nix/store/1iv7c0cckjza7vhqr0zrcwygv817dqwq-python3.8-ldap0-1.1.0']
Creating layer 51 from paths: ['/nix/store/r63gighjp7xpli6m7gbd3np4na381ssw-python3.8-paramiko-2.7.1']
Creating layer 52 from paths: ['/nix/store/7laard0jfj3mqdkyfjhvnjf1mhf3ch8j-python3.8-xlwt-1.3.0']
Creating layer 53 from paths: ['/nix/store/c1jb82hb49iij9fn7d7kycbcrkv3r4if-python3.8-gunicorn-20.0.4']
Creating layer 54 from paths: ['/nix/store/bz9sma4ywvjrgawpynz64gycw8avsd7b-python3.8-web2ldap-1.5.94']
Creating layer 55 from paths: ['/nix/store/ifnmhjrvk3f0hbz3f25s3izlb9yk8x0f-iana-etc-20200729']
Creating layer 56 from paths: ['/nix/store/r2wvgnr54vmwnjvzyqdixv8xbn362jgh-mailcap-2.1.48']
Creating layer 57 from paths: ['/nix/store/gr6rjrscqn0ldhrkq937l5b0qvx5adi1-tzdata-2019c']
Creating layer 58 from paths: ['/nix/store/7ssa36laxnbx6v1v0fanmqfq1v6jq6dz-python3-3.8.5-env']
Creating layer 59 from paths: ['/nix/store/bkj0v1lfgybrqf3p9h13sam55kls6wfc-spiff-helper-0.5']
Creating layer 60 from paths: ['/nix/store/']
Creating layer 61 from paths: ['/nix/store/z30ppq6wzvwip7gzcjkk40rzdwwia79f-helper.conf']
Creating the customisation layer...
Adding manifests...
84c89ae303a7: Loading layer [==================================================>]  1.649MB/1.649MB
f4ab361769f9: Loading layer [==================================================>]    256kB/256kB
e844f01cf994: Loading layer [==================================================>]  31.63MB/31.63MB
bcb45912eb0e: Loading layer [==================================================>]  133.1kB/133.1kB
88b9b539a752: Loading layer [==================================================>]  4.925MB/4.925MB
8987bd8f6293: Loading layer [==================================================>]  40.96kB/40.96kB
7a51e1ac92be: Loading layer [==================================================>]  40.96kB/40.96kB
a6807477e83d: Loading layer [==================================================>]  61.44kB/61.44kB
79684c04a116: Loading layer [==================================================>]  4.188MB/4.188MB
08b80549b51c: Loading layer [==================================================>]  1.352MB/1.352MB
936eb31f9fdd: Loading layer [==================================================>]  92.16kB/92.16kB
8018cbe06ed0: Loading layer [==================================================>]  245.8kB/245.8kB
b49d3847ff83: Loading layer [==================================================>]  696.3kB/696.3kB
ba5a0e76ebda: Loading layer [==================================================>]  399.4kB/399.4kB
c62d197ab3f3: Loading layer [==================================================>]  1.311MB/1.311MB
2fe8ddf5301b: Loading layer [==================================================>]    512kB/512kB
9bc739433b09: Loading layer [==================================================>]  1.434MB/1.434MB
d25b3d9b6d1b: Loading layer [==================================================>]  70.78MB/70.78MB
9aacaff1f0d3: Loading layer [==================================================>]  2.929MB/2.929MB
bdab357a6fc6: Loading layer [==================================================>]  102.4kB/102.4kB
f189502ca909: Loading layer [==================================================>]  501.8kB/501.8kB
81ef144fa564: Loading layer [==================================================>]  1.659MB/1.659MB
1720abcd6703: Loading layer [==================================================>]  122.9kB/122.9kB
f34a60c7e476: Loading layer [==================================================>]  6.328MB/6.328MB
b58d21489a40: Loading layer [==================================================>]  2.632MB/2.632MB
34b296916c41: Loading layer [==================================================>]  593.9kB/593.9kB
b6dafa17d261: Loading layer [==================================================>]  1.044MB/1.044MB
88d620eac9e5: Loading layer [==================================================>]  4.915MB/4.915MB
9c9af477b21d: Loading layer [==================================================>]  13.13MB/13.13MB
4caa64ff6b18: Loading layer [==================================================>]  10.24kB/10.24kB
07f04c90f21e: Loading layer [==================================================>]   51.2kB/51.2kB
da2c22787734: Loading layer [==================================================>]  61.44kB/61.44kB
9d85f97875d2: Loading layer [==================================================>]  1.669MB/1.669MB
d5f33fa3fa48: Loading layer [==================================================>]  92.16kB/92.16kB
6433d9dab6c7: Loading layer [==================================================>]  819.2kB/819.2kB
7c8c49cc7348: Loading layer [==================================================>]  4.178MB/4.178MB
1c7885702c4f: Loading layer [==================================================>]   51.2kB/51.2kB
6f0220b5584e: Loading layer [==================================================>]  430.1kB/430.1kB
4f020908772a: Loading layer [==================================================>]  2.386MB/2.386MB
48f453da26f2: Loading layer [==================================================>]  2.724MB/2.724MB
a189eb500985: Loading layer [==================================================>]  2.796MB/2.796MB
e2a121786f05: Loading layer [==================================================>]  645.1kB/645.1kB
86d3c5c90f91: Loading layer [==================================================>]  71.68kB/71.68kB
d018247d60cb: Loading layer [==================================================>]  122.9kB/122.9kB
a7b080eb35a0: Loading layer [==================================================>]  1.556MB/1.556MB
fe668b051d18: Loading layer [==================================================>]  716.8kB/716.8kB
99863d5cd3cc: Loading layer [==================================================>]  4.321MB/4.321MB
0e42a978fd5b: Loading layer [==================================================>]  10.24kB/10.24kB
5ef0c0e7e032: Loading layer [==================================================>]  993.3kB/993.3kB
44c4fc80d177: Loading layer [==================================================>]  1.536MB/1.536MB
85035a0ceced: Loading layer [==================================================>]  1.147MB/1.147MB
aa38af16b0a6: Loading layer [==================================================>]  1.475MB/1.475MB
287eb5060b23: Loading layer [==================================================>]  860.2kB/860.2kB
b9fe6db46678: Loading layer [==================================================>]  624.6kB/624.6kB
09542d7fb87d: Loading layer [==================================================>]  4.516MB/4.516MB
b0d1527cdadf: Loading layer [==================================================>]  573.4kB/573.4kB
00f9555aa61b: Loading layer [==================================================>]  112.6kB/112.6kB
f702a39bb8ec: Loading layer [==================================================>]  2.888MB/2.888MB
56ae713dae45: Loading layer [==================================================>]  204.8kB/204.8kB
98597cf4b312: Loading layer [==================================================>]  8.253MB/8.253MB
dc7118d021b3: Loading layer [==================================================>]  10.24kB/10.24kB
648cb592f81a: Loading layer [==================================================>]  10.24kB/10.24kB
280bf81781e0: Loading layer [==================================================>]  194.6kB/194.6kB

Wow, somebody lent a hand to the nix cause …

Na, I just photo-shopped it. But it looks neat, don’t you think? :smile:

As for caching, do you avoid in nixery the problem of buildLayeredImage , that although it caches the nix builds, it does not cache the created tar resulting in rebuilds / reloads like this

Yes, Nixery caches entire builds - that is if the build is eligible for caching .

Or maybe it’s enough/better to implement nixc nixd , a runner daemon (or rather containerd plugin!) that skaffolds containers upon reception of a manifest from something like ipfs-based nix store. See Nix, Containerization, and SquashFS - with some crfs features. One step further than what nixery does - just throwing in loose ideas - but since k8s is seeing RuntimeClass - why not take the shortcut? - would solve most of the interesting use cases in kind of a nix-native way. and smells a ton like “innovation”. CI output could be just used as “registry”. - Had I only the skills to do it :wink:

I just wonder what would have to go into the image: field (or instead of it?). A handle to a CRD-induced nix build which nixd knows how to intercept?

/cc @burke for the sqashfs input

Sure, if buildkit does the nix builds instead of nix itself that woudn’t hurt - as it would have previsously forced the nix instantiate and build implementations to become more composable.

EDIT: CRI protobuf api allows arbitrary metadata (read nix derivations or expressions) in the imageSpec. Checkpot!

I feel we’re drifting from the original thread title, but I think you might find this interesting :slight_smile:

and for context: #nixos on 2018-01-10 — irc logs


I think it’s a fun idea. Wrap CRI, skip the container image process (almost?) entirely.

Very interesting, looks like nix-cri-imageservice did not come into existence, though. From the chat logs:

02:47 <dhess> shlevy: will this nix-cri-imageservice (grab what’s needed from the binary cache|build it) on demand?
02:47 <shlevy> The design I had in mind was grab from the binary cache
02:48 <dhess> sounds cool
02:48 <shlevy> In principle though you could have it build on demand
02:48 <shlevy> But I doubt that’s really desirable :smiley:
02:48 <dhess> probably not
02:48 <shlevy> It would fetch from the cache when kubelet pulls the relevant image, and delete the gc root when kubelet deltes the image

Probably a better idea would be to make the existence of a SAN-based nix store a collateral of the CI process, so that the shim only composes the root filesystem on demand.

@shlevy What’s your today’s stance on this?

Sorry, I should’ve mentioned – I reached out to Shea – in case Shea doesn’t see your ping, or doesn’t want to repeat themselves:

from the brief email thread, it sounds like it didn’t make it past the idea phase. The gist was an ImageService implementation/wrapper that ensures nix store paths are realized in the store and gcrooted for as long as ImageSvc needs it. And then either images that request the Nix store be mounted, or a wrapper runtime service that injects just the closure paths.

1 Like

Just found those docs — think they are relevant:

FYI, a nix-cri-imageservice has been implemented as part of nix-snapshotter:

Though the image reference looks a bit weird due to what’s considered a valid reference:

Essentially Kubernetes configured with nix-snapshotter can resolve store paths pointing to OCI image archives. Nix-snapshotter then configures the container rootfs from the host Nix store (creating gcroots & thereby substituting from a Nix binary cache if necessary).

Check out what’s possible, now you can define a pod spec without any external dependency (without a Docker Registry):

redis = pkgs.nix-snapshotter.buildImage {
  name = "redis";
  resolvedByNix = true; # passthru.image is set to `nix:0/nix/store/...`
  config = {
    entrypoint = [ "${pkgs.redis}/bin/redis-server" ];

redisPod = {
  apiVersion = "v1";
  kind = "Pod";
  metadata = {
    name = "redis"; = "redis";
  spec.containers = [{
    inherit (redis) name image;
    args = ["--protected-mode" "no"];
    ports = [{
      name = "client";
      containerPort = 6379;

You can then define a flake app to do something like nix run .#deploy-k8s which essentially wraps kubectl apply -f ${pkgs.writeText "pod.json" (builtins.toJSON redisPod)}.