A faster dockerTools.buildImage prototype

I prototyped new tools to build container images with Nix. The main idea is to stop writing images or layers to the Nix store, in order to speed up container build times: Nix only builds an image description (a small JSON file) which is then used by Skopeo to push or load the image.

More details can be found in this blog post, which could be discussed here.

The current implementation is a PoC, so it would still require work to be upstreamed, especially since some patches in Skopeo are required.

So, i would like to know what you think about this.
Maybe some of you would even be interested in helping on making this upstream!

8 Likes

This is a lot like streamLayeredImage, with differences being a new layer splitting strategy and a system-independent representation.

Some work on pluggable layering strategies has been done here pkgs.dockertools.buildLayeredImage: customisable layering strategy by adrian-gierakowski · Pull Request #122608 · NixOS/nixpkgs · GitHub

streamLayeredImage already uses a tarball-free, system-independent representation of images, but always wraps it in a script. For upstreaming, you could unify these representations and refactor streamLayeredImage to expose the json representation, if you need it to be system-independent.

1 Like

This is a lot like streamLayeredImage , with differences being a new layer splitting strategy and a system -independent representation.

Yep, there are several similarities and streamLayeredImage is a nice thing.

Correct me if i’m wrong, but each time you load an image with streamLayeredImage, you need to recompute the digest of all layers. With the presented prototype, digests are computed at build time (almost per layer) so when Skopeo pushes an image, it can immediately skips already pushed layers: if you develop with containers, containers need to be rebuilt/reloaded as fast as possible (cc Arion :wink:).

In terms of user experience, i’m not sure streaming a tar on stdout is really convenient. I think having intermediate files is pretty useful to understand what is happening.

Also, i would like to reduce the amount of container custom code in nixpkgs: Skopeo has all stuffs to push/load images: it would be nice to reuse this instead of rewriting it. Moreover, the proposed Skopeo transport could used by other projects (Guix or Bazel for instance).

Correct me if i’m wrong, but each time you load an image with streamLayeredImage, you need to recompute the digest of all layers.

This is correct and it is a solution for the non-reproducable hash problem. So the three possible solutions are:
a. Just fail when the image paths aren’t reproducible
b. Compute the digest at the last moment. Representation is compact. Image is always loadable. Slight unreproduciblities can creep in when a different build of any path is used.
c. Compute the digest in advance and store the tar. Representation is inefficient. Image is always loadable. Unreproducibilities can creep in, but only when the image representation (json) is built more than once.

streamLayeredImage is (b). Your prototype is (a) and you’ve suggested (c) as opt-in.

so when Skopeo pushes an image, it can immediately skips already pushed layers

This is nice when using a remote DOCKER_HOST.

if you develop with containers, containers need to be rebuilt/reloaded as fast as possible (cc Arion :wink: ).

It’s hard to beat using the host store for development. You only have to load the “customization layer” ie non-store paths.

In terms of user experience, i’m not sure streaming a tar on stdout is really convenient.

I have had no issue with this, but that could be because arion tracks the metadata for me.

Also, i would like to reduce the amount of container custom code in nixpkgs: Skopeo has all stuffs to push/load images: it would be nice to reuse this instead of rewriting it.

Absolutely.

you could unify these representations and refactor streamLayeredImage to expose the json representation

… or you could do it the other way around, rebase streamLayeredImage onto your code. That would be a small wrapper and you’ll get a pretty good test suite for free. Also upcoming stuff like using NixOS modules to create images.
Maintaining dockerTools has been 80% “please add a test” :wink:

1 Like

Nice work! I like how elegant the design is, you clearly thought about the various use-cases.

What kind of feedback are you looking for, are there areas where you are not so sure about the design?

In terms of store usage, the killer is mostly the surrounding tarball that is loaded with docker load -i because it duplicates the same content over and over again. The individual layer tarballs are less of an issue as it only doubles store usage. One thing I wanted to investigate is if skopeo can copy the unpacked image? The main advantage is that then it wouldn’t have to be patched.

1 Like

streamLayeredImage is (b). Your prototype is (a) and you’ve suggested (c) as opt-in.

Thanks for this enumeration. It’s pretty clear!
Actually, I think Skopeo could easily generate the digest if it is not specified in the json file. This could be a way to cover (b).

This is nice when using a remote DOCKER_HOST .

I didn’t mention it in my blog post, but unfortunately, Skopeo is not able to skip already pushed layers when the destination is the Docker deamon:/
But I hope that would be possible with this buildkit-nix project (and having a Go library is a big plus for an integration in this project…).

What kind of feedback are you looking for, are there areas where you are not so sure about the design?

I’m mainly interested in finding people that could be interested
in helping me on this topic! I actually don’t think I will have
enough bandwidth to make it alone (this was a spare time
project and I don’t have anything to deliver, excepting having
fun with it!)

One thing I wanted to investigate is if skopeo can copy the unpacked image?

Skopeo has a dir transport which IIRC is basically a unpacked
Docker archive. So, we could imagine to only store layers (as
tars) and use this transport instead of a whole Docker
archive (but in this case, we would still have to write tarball
in the store).

FYI, I created the nix2container repository and moved almost everything in this repository.

I also added the non reproducible storepaths support and “path rewriting” (to support buildImage.contents).

2 Likes