# A faster dockerTools.buildImage prototype

**URL:** https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922
**Category:** Announcements
**Created:** [January 1, 2022, 6:04pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922 "2022-01-01T18:04:24Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![lewo](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lewo/32/4913_2.png) [@lewo](https://discourse.nixos.org/u/lewo)
#### Post date: [January 1, 2022, 6:04pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/1 "2022-01-01T18:04:25Z")

</div>

I prototyped new tools to build container images with Nix. The main idea is to stop writing images or layers to the Nix store, in order to speed up container build times: Nix only builds an image description (a small JSON file) which is then used by Skopeo to push or load the image.

More details can be found in this [blog post](https://lewo.abesis.fr/posts/nix-build-container-image/), which could be discussed here.

The current implementation is a PoC, so it would still require work to be upstreamed, especially since some patches in Skopeo are required.

So, i would like to know what you think about this.  
Maybe some of you would even be interested in helping on making this upstream!

---

<div class="post-metadata">

### Author: ![roberth](https://discourse.nixos.org/user_avatar/discourse.nixos.org/roberth/32/5488_2.png) [@roberth](https://discourse.nixos.org/u/roberth)
#### Post date: [January 1, 2022, 6:47pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/2 "2022-01-01T18:47:43Z")

</div>

This is a lot like `streamLayeredImage`, with differences being a new layer splitting strategy and a `system`-independent representation.

Some work on pluggable layering strategies has been done here [pkgs.dockertools.buildLayeredImage: customisable layering strategy by adrian-gierakowski · Pull Request #122608 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/pull/122608)

`streamLayeredImage` already uses a tarball-free, `system`-independent representation of images, but always wraps it in a script. For upstreaming, you could unify these representations and refactor `streamLayeredImage` to expose the json representation, if you need it to be `system`-independent.

---

<div class="post-metadata">

### Author: ![lewo](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lewo/32/4913_2.png) [@lewo](https://discourse.nixos.org/u/lewo)
#### Post date: [January 2, 2022, 9:49am UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/3 "2022-01-02T09:49:33Z")

</div>

> This is a lot like `streamLayeredImage` , with differences being a new layer splitting strategy and a `system` -independent representation.

Yep, there are several similarities and `streamLayeredImage` is a nice thing.

Correct me if i’m wrong, but each time you load an image with `streamLayeredImage`, you need to recompute the digest of all layers. With the presented prototype, digests are computed at build time (almost per layer) so when Skopeo pushes an image, it can immediately skips already pushed layers: if you develop with containers, containers need to be rebuilt/reloaded as fast as possible (cc Arion 😉).

In terms of user experience, i’m not sure streaming a tar on stdout is really convenient. I think having intermediate files is pretty useful to understand what is happening.

Also, i would like to reduce the amount of container custom code in nixpkgs: Skopeo has all stuffs to push/load images: it would be nice to reuse this instead of rewriting it. Moreover, the proposed Skopeo transport could used by other projects (Guix or Bazel for instance).

---

<div class="post-metadata">

### Author: ![roberth](https://discourse.nixos.org/user_avatar/discourse.nixos.org/roberth/32/5488_2.png) [@roberth](https://discourse.nixos.org/u/roberth)
#### Post date: [January 2, 2022, 12:37pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/4 "2022-01-02T12:37:30Z")

</div>

> Correct me if i’m wrong, but each time you load an image with streamLayeredImage, you need to recompute the digest of all layers.

This is correct and it is a solution for the non-reproducable hash problem. So the three possible solutions are:  
a. Just fail when the image paths aren’t reproducible  
b. Compute the digest at the last moment. Representation is compact. Image is always loadable. Slight unreproduciblities can creep in when a different build of any path is used.  
c. Compute the digest in advance and store the tar. Representation is inefficient. Image is always loadable. Unreproducibilities can creep in, but only when the image representation (json) is built more than once.

`streamLayeredImage` is (b). Your prototype is (a) and you’ve suggested (c) as opt-in.

> so when Skopeo pushes an image, it can immediately skips already pushed layers

This is nice when using a remote `DOCKER_HOST`.

> if you develop with containers, containers need to be rebuilt/reloaded as fast as possible (cc Arion 😉 ).

It’s hard to beat using the host store for development. You only have to load the “customization layer” ie non-store paths.

> In terms of user experience, i’m not sure streaming a tar on stdout is really convenient.

I have had no issue with this, but that could be because arion tracks the metadata for me.

> Also, i would like to reduce the amount of container custom code in nixpkgs: Skopeo has all stuffs to push/load images: it would be nice to reuse this instead of rewriting it.

Absolutely.

> > you could unify these representations and refactor `streamLayeredImage` to expose the json representation

… or you could do it the other way around, rebase `streamLayeredImage` onto your code. That would be a small wrapper and you’ll get a pretty good test suite for free. Also [upcoming](https://github.com/NixOS/nixpkgs/pull/148456) stuff like using NixOS modules to create images.  
Maintaining `dockerTools` has been 80% “please add a test” 😉

---

<div class="post-metadata">

### Author: ![zimbatm](https://discourse.nixos.org/user_avatar/discourse.nixos.org/zimbatm/32/4_2.png) [@zimbatm](https://discourse.nixos.org/u/zimbatm)
#### Post date: [January 3, 2022, 11:16am UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/5 "2022-01-03T11:16:47Z")

</div>

Nice work! I like how elegant the design is, you clearly thought about the various use-cases.

What kind of feedback are you looking for, are there areas where you are not so sure about the design?

In terms of store usage, the killer is mostly the surrounding tarball that is loaded with `docker load -i` because it duplicates the same content over and over again. The individual layer tarballs are less of an issue as it only doubles store usage. One thing I wanted to investigate is if skopeo can copy the unpacked image? The main advantage is that then it wouldn’t have to be patched.

---

<div class="post-metadata">

### Author: ![lewo](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lewo/32/4913_2.png) [@lewo](https://discourse.nixos.org/u/lewo)
#### Post date: [January 4, 2022, 6:56pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/6 "2022-01-04T18:56:52Z")

</div>

> `streamLayeredImage` is (b). Your prototype is (a) and you’ve suggested (c) as opt-in.

Thanks for this enumeration. It’s pretty clear!  
Actually, I think Skopeo could easily generate the digest if it is not specified in the json file. This could be a way to cover `(b)`.

> This is nice when using a remote `DOCKER_HOST` .

I didn’t mention it in my blog post, but unfortunately, Skopeo is not able to skip already pushed layers when the destination is the Docker deamon:/  
But I hope that would be possible with this [buildkit-nix project](https://github.com/AkihiroSuda/buildkit-nix) (and having a [Go library](https://github.com/nlewo/containers-image-nix) is a big plus for an integration in this project…).

---

<div class="post-metadata">

### Author: ![lewo](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lewo/32/4913_2.png) [@lewo](https://discourse.nixos.org/u/lewo)
#### Post date: [January 4, 2022, 6:59pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/7 "2022-01-04T18:59:20Z")

</div>

> What kind of feedback are you looking for, are there areas where you are not so sure about the design?

I’m mainly interested in finding people that could be interested  
in helping me on this topic! I actually don’t think I will have  
enough bandwidth to make it alone (this was a spare time  
project and I don’t have anything to deliver, excepting having  
fun with it!)

> One thing I wanted to investigate is if skopeo can copy the unpacked image?

Skopeo has a `dir` transport which IIRC is basically a unpacked  
Docker archive. So, we could imagine to only store layers (as  
tars) and use this transport instead of a whole Docker  
archive (but in this case, we would still have to write tarball  
in the store).

---

<div class="post-metadata">

### Author: ![lewo](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lewo/32/4913_2.png) [@lewo](https://discourse.nixos.org/u/lewo)
#### Post date: [January 16, 2022, 4:20pm UTC](https://discourse.nixos.org/t/a-faster-dockertools-buildimage-prototype/16922/8 "2022-01-16T16:20:33Z")

</div>

FYI, I created the [nix2container](https://github.com/nlewo/nix2container) repository and moved almost everything in this repository.

I also added the [non reproducible storepaths support](https://github.com/nlewo/nix2container/blob/98b0986a4ffeae5be9db83f55a6f119d65a34f57/examples/non-reproducible.nix) and “path rewriting” (to support `buildImage.contents`).
