Recommendations for introducing a shared nix store or cache for CI/CD and development

Help
21

Can you elaborate on how you do that? I’m looking for a similar solution but I ended up using shell runner to be able to use the local nix store but its really bad for reproducibility…

22

My build machine runs NixOS and so I mount /nix, /run/current-system, /etc/ssl/certs/ca-bundle.crt:/etc/ssl/certs/ca-bundle.crt and /etc/ssl/certs/ca-certificates.crt from the host in the runner and then use a small alpine docker image built from this Dockerfile:

FROM alpine:edge
RUN apk add bash curl
ENV \
    ENV=/etc/profile \
    TMP=/tmp \
    NIX_REMOTE=daemon \
    PATH=/run/current-system/sw/bin:/bin:/usr/bin \
    GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt \
    NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
CMD ["/bin/bash", "-l"]

This is the part of my Gitlab Runner config.toml that configures docker for this setup:

[runners.docker]
disable_cache = false
disable_entrypoint_overwrite = false
image = "lisberg/ci-host-nix"
oom_kill_disable = false
privileged = false
shm_size = 0
tls_verify = false
volumes = ["/nix:/nix", "/run/current-system:/run/current-system", "/etc/ssl/certs/ca-bundle.crt:/etc/ssl/certs/ca-bundle.crt", "/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt", "/cache"]

In my setup I also have a custom helper image but I don’t actually think that is needed any more so I have left it out. If you try to use this setup and you have problems with permissions you might need the custom helper image bit but I am pretty confident that it is no longer needed.

3 Likes
24

I’ve been lately exploring buildkite.com (free for open source) and I’m very happy with the results

  • You can launch machines on an autoscaling group or in a single machine using NixOS
  • Each machine is configured like this so that N agents run inside N systemd nspawn containters, and by default, all N nsspawn containers share the host /nix/store so it’s very shared and fast
  • Builds are as fast as If I run them locally, the overhead of buildkite, the agents, and the nsspawn containers is imperceptible. I push and the jobs immediately start, all deps already cached, just delicious
2 Likes
25

@kamadorueda : Do you have an opensource reference how you use that?