Building a CI docker image

Help
1

Hey there,

I’m trying to build a CI image for a rust project which uses nix flakes and crane.
My current efforts are here: gitblick/flake.nix

The idea was to include all needed tools and the crate dependencies in the image, so that building the project’s binary in the pipeline would only rebuild the source code, without fetching the toolchain or building the rust dependencies.

My current attempt looks like this:

        gitblick-ci-image = pkgs.dockerTools.buildLayeredImage {
          name = "registry.gitlab.com/manuel2258/gitblick/gitblick-ci";
          tag = "latest";
          created = "now";
          enableFakechroot = true;
          fakeRootCommands = ''
            mkdir -p /tmp
            chmod 1777 /tmp

            ${pkgs.dockerTools.shadowSetup}

            echo "nixbld:x:999:" > /etc/group
            echo "root:x:0:" >> /etc/group

            for n in $(seq 1 10); do
              useradd -c "Nix build user $n" \
                -d /var/empty \
                -g nixbld \
                -G nixbld \
                -M \
                -N \
                -r \
                -s "$(command -v nologin)" \
                "nixbld$n"
            done
          '';
          contents = [
            (pkgs.writeTextFile {
              name = "nix.conf";
              destination = "/etc/nix/nix.conf";
              text = ''
                accept-flake-config = true
                experimental-features = nix-command flakes
              '';
            })

            rust-tools
            gitblick-crate-deps

            pkgs.bash
            pkgs.gnugrep
            pkgs.coreutils
            pkgs.nix
            pkgs.cacert
            pkgs.gitMinimal
            pkgs.crane
            pkgs.gzip
          ];

          config = {
            Cmd = [ "/bin/bash" ];
            Env = [
              "SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
            ];
          };
        };

However building the target in the image using nix build, refetches the rust toolchain as well as rebuilds the dependencies.

The store in the container looks like this:

ls /nix/store 
054lcfzbgh10590l7107nhv6znphnr29-openssl-3.0.13-bin
0881va25p2fs3zj31d4m6y3xji25dfqm-nix.conf
0ngqw18chwixfqw76q3f2pn50kksidbk-s2n-tls-1.4.12
15lzdmjwrvkml0xxms4a0q91mpianf8g-gettext-0.21.1
18aly31g886ddfx2hkg9hhj68085iair-openssl-3.0.13
18f0b2bnxpjq15j1ibwwppygwwhpg9r0-nix-2.18.2
1a4y0rc03bckd5mh4kz11wdlcp31zzdi-tcb-1.2
1nz68l013a5l289k8wfargqs8lv71lq5-aws-crt-cpp-0.26.8
20a0qpw4xw1f7lvvs71yigc0ngmnicib-iana-etc-20240318
22r6157kw2ncqikr1jyryd3sbgvvc4s4-rust-nightly-complete-with-components-2024-05-05
39q8358aad2h98s1k3pwmqafjqng3k1w-git-minimal-2.44.0
3mri6pgjkvci7yc1msb90kxqxdg9qb4h-keyutils-1.6.3-lib
4cps736z7in3d37qc801lwv9z0ib67ps-gnused-4.9
5a0xhcjclygjx14y4nl06sa57in6jx2z-linux-pam-1.6.1
5f9g4hc3snzgdx638s8bbdmi2gsk34zi-aws-sdk-cpp-1.11.309
5gckmydg2v3dlgcz81l98d62zlbvzr67-libkrb5-1.21.2
5gzyfqc5vzm4gyijkyxw8bl8i91zfk77-gmp-with-cxx-6.3.0
5m7d3z1xcwy0gagkginzqvhcrxifzwqf-openssl-3.0.13-dev
61s8cmgwj8kslqkqs80a2whwysqd354k-audit-3.1.2
78sr2vzp80nbvygvawm4zffmnq0vhw8k-publicsuffix-list-0-unstable-2024-01-07
7ci4z26j61yi9z4w9cn90fb67jiq49j8-aws-c-sdkutils-0.1.15
7lzfzr3xsdgscccnfl9rykiwncwwvhbi-tzdata-2024a
7qpqgvadz00yv8snbdz2dbin4mxwhc7n-libssh2-1.11.0
880p5hg6vwm7sswmhpcgcy02wl8rbm81-rustc-nightly-complete-2024-05-05
93zgiql3hzy7nf5q1y4nvg25nnaaxijg-dummy.rs
9g3hgzyxzn17x8yy6sh642gmdbjjjj3l-brotli-1.1.0-lib
afdwfcrnin96bwx0z0s21a73z4l947fp-nix-2.18.2-man
apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31
bfb1ayjwa1xy6mhm8rgrrcic7lnwh5zr-libseccomp-2.5.5-lib
bmq12snmn3dr98my7krdi9fp3qbahdph-zstd-1.5.6
cnxb8cwvk8gvg192b4jj7cqqv3jigcir-aws-checksums-0.1.18
ddfzjdykw67s20c35i7a6624by3iz5jv-libidn2-2.3.7
fjxs9vk02ldpz0xyd3yh5pnzvhq9gmby-libxcrypt-4.4.36
fnpnm49r786myrw4451waqrp471rz831-expat-2.6.2
ghw2grp548f6s885bbz3c779pszr4h2q-aws-c-auth-0.7.17
gm61h1y42pqyl6178g90x8zm22n6pyy5-libunistring-1.1
h3bhzvz9ipglcybbcvkxvm4vg9lwvqg4-bash-5.2p26
hq8765g3p1i7qbargnqli5mn0jpsdbfl-coreutils-9.5
hragmx5p3izmwinv4441hj5706sxy4xs-gzip-1.13
hzc5ypaxp663vnf0fzn5kg41w1x61485-libxml2-2.12.6
i0npczxkj074b9r8rg4vvlgrrik5wz7z-sqlite-3.45.2
k281507f4w0zjh98vw0d1pn85v0xjhky-libarchive-3.7.3-lib
k927dryij6k8ga8slxs39pjdsmsny8c8-gcc-13.2.0-libgcc
kps2wa2z3wx2qlmz5hggc0f8rvrr6fdk-rust-std-nightly-complete-2024-05-05
lg84fz9bqa2l0n9vb3aw88sr6vpfh89f-busybox-static-x86_64-unknown-linux-musl-1.36.1
lpsa98iwl9q7fb2hdiqhqcryyg2kvgnm-libcpuid-0.6.5
m5lri8cxiknksbzsw5d1i4bgk8ydqfcq-libmd-1.1.0
mdy1mz2vyiv09syppck82janhzjjmr4r-nghttp2-1.61.0-lib
n87dnp3cw99gssql0xk9yycrhrg50gir-xz-5.4.6
ni2x5a5x9j72l3s4wc7vs1r27b5hqzxi-shadow-4.14.6
nicj237cryrpyw2vg7aip1r6yg9ixsdx-libsodium-1.0.19
nkdgysx90pln2l562l7lsk021hbxjdg8-glibc-2.39-31-bin
nrwkk6ak3rgkrxbqhsscb01jpzmslf2r-xgcc-13.2.0-libgcc
nw1bnjadmzk9hnd4nbvxmqlh074vr16j-zlib-1.3.1
p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib
pfafl4fcfcmz4229c0pgdlg2x8dkxpwq-aws-c-cal-0.6.10
pfjgs4yf0v0sy8ic1cg0sq9ib41nd3wr-cargo-nightly-complete-2024-05-05
pk8a8hsabhk0z27q4y5qmf7rzvmra9ls-go-containerregistry-0.19.1-crane
q4k8gmw2gpncglwdvz9fmqqwnynsb143-bzip2-1.0.8
q8v0qhb36p32qilawhkllvw8l832xv31-aws-c-http-0.8.1
qn2sjk219k4vay7ajxnxqgx26rfy6flm-aws-c-compression-0.2.18
qv1qp23si0prgzcx6b8522f0g7fipaph-curl-8.7.1
rdz18jsmjv3n13zm2k12bi93wzf1mdzm-aws-c-s3-0.5.7
si81szrjym2wksdxf8m6yqz8y383idlx-libbsd-0.11.8
sik7a8dic5lxhm4cp9npibk97x9jcxwf-pcre2-10.43
v1fj4nqw9qdzjbb3xafrz6am91lyfz1d-aws-c-mqtt-0.10.3
v3ih684f3lhmzwm12hrchsvafsv2xn61-aws-c-common-0.9.15
vwmxv4d6wshi78s88jcm4j516sj13njm-boehm-gc-8.2.6
w2kvnj69k4r2f1rq8ahn1i530z6dspgz-libpsl-0.21.5
w86w7jgg8d172b2g66jj4sc77nxgi25k-aws-c-event-stream-0.4.2
whkfih9b9z6hxygc132yfl4rdp7s25dc-db-4.8.30
wy37jk2hirzqzx0666w1849kjdgzdam6-gnugrep-3.11
xjbls5ijpal21648185yj2z6am69x7n1-editline-1.17.1
y33c98w41jnpyq5kqq7q5akz62594q02-nss-cacert-3.98
y4psv76i8pllhnsd1dg7zbqsj8a2sz64-acl-2.3.2
yb39i0a63bx8kx2bgmx1p4dx1hcwmnzh-attr-2.5.2
ympcs8qs617iagzh1zm52rliqp3r3cin-aws-c-io-0.14.7
ywn2s30g3dzxf2qfyy1gpg28gqq684zc-gitblick-deps-0.1.0
yx6l9m0myffa8hcbiwwb1gh429fnl5h4-lowdown-1.1.0-lib
zm68r67a05xl4mmvv8vn774zsx3si7ym-mailcap-2.1.53

I would guess that nix also requires the .drv files as well as maybe some database, in order to accept the derivations in the store?
Is that correct?
How can I include them in the docker image?

Thanks already for your time and help!

1 Like
2

It looks like you try to use nix within the docker image. Is there a specific reason why you want that?

https://nixos.org/manual/nix/stable/installation/installing-docker

Usually i’d assume you want to build your rust package

And then build an image that includes the rust binary.

So here is what I’d typically do:

  1. Run nix flake init -t github:ipetkov/crane#quick-start to initialize crane in your rust project.

  2. Make sure your build works locally nix flake show (displays all exported packages)
    nix build .#<pkgName>

  3. Include the attribute into buildLayeredImage and add it to the flake output

  4. Create a .github-ci.yml .gitlab-ci.yml, with nix bootstrapping.

  5. run nix build .#<ImagePkgName> in the CI file.

If your dependencies don’t change crane re-uses them Artifact reuse - crane

You’ll probably want to setup a build cache like cachix for your ci to store your nix/store artifacts.

3

My thinking was that I would like to only define once how to build, test and check my project using nix flake and then use that very same environment and steps in the pipeline aswell.

Of course I could simply invoke cargo and for example clippy in the pipeline manually, but that would not be that ‘clean’.

So Ideally I would really like to get a docker image, where I can just call ‘nix build’ and only my package derivation will be rebuilded. That image is fully for CI / pipeline purpose.
For the ‘release’ I have a separate image which only contains the binary, just like you suggested.

In the end what I want to achieve is to have fast CI builds, do you think my best bet for that is to setup a cache using for example cachix?

4

By default the built docker images do not include the Nix database, so nix does not know which packages are already in the store and it will re-download/build everything.

You can use dockerTools.buildImageWithNixDb to include the DB, then it should not rebuild (or buildLayeredImageWithNixDb). It can be a bit tricky unfortunately to include all the inputs you need to build without triggering a re-download/re-build.

It looks like those functions are currently not documented, they work the same as the equivalents without the WithNixDb suffix. Documentation: missing documentation for certain `dockerTools` functions · Issue #283715 · NixOS/nixpkgs · GitHub

5

I already tried the withNixDb (both layered and not layered) functions, but all the packages within the container using that image, where still rebuilded everytime.

What files / entries in files are required for a derivation folder present in the store to be used?

And what are nice ways of debugging this / finding out why nix decides on when to rebuild?

6

While I was not able to solve this issue using my original idea, I just employed cachix now, which works perfectly for my use case and is free for open source projects, which I didn’t know …

I guess I will still keep this open, as the original question was not solved?

7

I do something very similar and using dockerTools.buildImageWithNixDb works for me.

Maybe something is preventing nix from using the DB in the resulting container?