dockerTools unable to export large-ish (concourse-git-resource) image

Help
1

I’m working on a source build for the Concourse CI system. I’ve made good progress on most of the system, but one place where I’ve gotten stuck is the bundled resource images. (Ideally, I suppose, I could rebuild those too instead of downloading the binaries from dockerhub, but one thing at a time). Some of them work correctly out-of-the-box, but that’s not true of the git one:

let
  pkgs = import <nixos-unstable> {};
  inherit (pkgs) dockerTools;
  # Retrieved with: nix-prefetch-docker --image-name concourse/git-resource --image-tag 1.17.0 --arch arm64
  concourseGitResource = {
    imageName = "concourse/git-resource";
    imageDigest = "sha256:d2395b9ba889eda03f2c782e2ad8fe31920dd72a4fd49a11f4bd42b813999674";
    sha256 = "15xdzlxjg28bky9fcyxphc15asdh1958m9mjfzxnglbr12kmcqjs";
    finalImageName = "concourse/git-resource";
    finalImageTag = "1.17.0";
  };
  upstreamImage = dockerTools.pullImage concourseGitResource;
in dockerTools.exportImage {
  name = "concourse-git-resource";
  fromImage = upstreamImage;
  fromImageName = concourseGitResource.finalImageName;
  fromImageTag = concourseGitResource.finalImageTag;
  diskSize = 2048; # runs out of space with the default 1024
}

…consistently fails:

Formatting './image/disk-image.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2147483648 lazy_refcounts=off refcount_bits=16
SeaBIOS (version rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org)


iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+1EFD0C20+1EF30C20 CA00
Press Ctrl-B to configure iPXE (PCI 00:03.0)...^M                                                                               


Booting from ROM...
Probing EDD (edd=off to disable)... o[    0.208076] sgx: There are zero EPC sections.
loading kernel modules...
mounting Nix store...
mounting host's temporary directory...
starting stage 2 (/nix/store/ryfhaf1pfj4xg3hidsls0b8dh0yq7ic0-vm-run-stage2)
mke2fs 1.47.1 (20-May-2024)
Discarding device blocks: done                            
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 0b8de3a9-8681-4f7f-adef-3904d818a903
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

Unpacking base image...
Unpacking layer 270a1170e7e398434ff1b31e17e233f7d7b71aa99a40473615860068e86720af.tar
Unpacking layer 69e7e9143c29e5bb6533dd6d139a74bc8debb7a588217c38eb0b0b792056e0ea.tar
Unpacking layer 1f5a07f2fc66d9e8121aea9102bbb98072d21bad0c3ee6ef02aa8afa4d0cf85b.tar
Unpacking layer 1ff5e081e9bc7b11362171e0ec53e158ed6d1d3ccb06256eb3ab7e455f48f70f.tar
Unpacking layer cbe38ea56691af23ea5e50d415bad855b91b58db326081633ea0a63faf7334f5.tar
Unpacking layer 04baac603add8de04ad0b2cfdb320ea85999ddbe37de014628ec69777b6211ac.tar
Unpacking layer 94e1c4ba341be7c232d5dbd3cd10b3f40226ee28becbb30cd2be815d379ecf75.tar
Unpacking layer 0c2c9dfc02ff119b6af7e8b6d82c7fab3165bac702cf5bc3bbe4f8f92828de5f.tar
Unpacking layer bf38d4c3de775a3a454fb864a5bfd1052d1ccf474722b391a224645bef4e16a3.tar
Unpacking layer b655a068a083fdbc3ffcfb06efbb6793d486306bf04252c5ed7277ee52bc421f.tar
Unpacking layer 75b04bbf0c6a9b50492d5c71a9afcc3441767fa1cf6cafab468af7145cd0549f.tar
Unpacking layer bc687e6b053ddd6d08c25dc5221ba74ea218a975c19f543ee260efcbfc2232e9.tar
Unpacking layer 083e1570a27310835abaf82e880a3a6d1540b6a9ae46e1a884fa5eaf359f8151.tar
Unpacking layer 9cf21bd3fed7f0b65dad84f83e3abfe85def46a557b8669379586bbe65c45ceb.tar
Unpacking layer bbf7029d408e691ac08fe63357121dc6d0f7ae9a7e3ed4e61c48e419cd6e3270.tar
Unpacking layer fc9621bcf70d5b2f524b46a03deef5dce7cb2ea0c0172f8839886c007d49a535.tar
Unpacking layer 76d3e64ce93b5c169dd0ac61f2e62156a312fdc0ba96f8ae0c49e4131b6ebb0f.tar
Unpacking layer 44de41253c6f4d39420315933aae8b41fb924711a60a1274dfb791f10351c308.tar
Unpacking layer 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef.tar
mount: /tmp/disk/mnt: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
[   14.299532] reboot: Power down

Any pointers? I’m having trouble even getting a set -x into place to get the full command line of the mount that fails, as exportImage doesn’t support overrideAttrs or such.

2

I ended up running against a patched nixpkgs running with xtrace, and discovered that it’s the mount -t overlay that’s failing. The image in question has 19 layers; splitting this up into 16 layers at a time solved the problem.

PR filed as dockerTools: Support mounting images with more than 16 layers (#392421) by charles-dyfis-net · Pull Request #392422 · NixOS/nixpkgs · GitHub