I have a project where we’re using nix-shell to provide a standard development environment. It’s been pretty great! As a next step I’d like to set up CI so it uses the the same derivation we’re using for our development environment. I’m not sure what the best approach is here and was wondering if anyone had some tips for me.
Currently we run CI in an Ubuntu based Docker image, in Jenkins. In an effort to make small steps I’d ideally like to replace the docker image (currently created from a fairly massive Dockerfile) with either a nix-based or nix-generated docker file.
The first thing I tried was building a docker with nix, based on the pretty awesome blogpost Cheap Docker images with Nix. While I can get this to work, there’s two aspects of this approach I’m not very fond of:
The generated image is pretty large. Around 8 times larger than the ubuntu-based image we’re using now.
Using this approach I’m not responsible for caching this step of the build, and I’m not sure what the best way would be to set this up. Building the docker image takes a pretty long time, so I don’t want to run it on every build.
We use the Jenkins equivalent of docker build . to build our current ubuntu-based image in our CI process, which builds the Dockerfile only when it changes, and then only part of the layers.
Because nix isn’t installed on the Jenkins host, I need to run the command that sees nix building the ci container inside of a container itself. But that means that at the end of the build, I throw away the docker container used to run the build along with its nix store, meaning the next build will have to start from scratch. Is there an easy way to cache the nix store in the container at the end of a build?
Are there other approaches I could try? Should I stick with the current approach and, if so, are there things I might improve about it?
I think the best approach here would probably be to use one of the nix docker images rather than building docker images with nix.
This might be done by starting from https://hub.docker.com/r/nixorg/nix/, and then just using nix-shell for the build/test process like in local development. For better performance, you may also want to set up a binary cache of some sort and copy build results into it.
Awesome, thank you for your response, this is super useful!
In dev we noticed that the time it takes up to 20 seconds to start nix-shell, due to the size of the derivation it builds. Is there a way to cache this? Locally we use direnv to cache this, but I imagine this isn’t the best solution on CI. Is there an alternative way to cache nix-shell for CI? Or maybe the option to install the dependencies in shell.nix in the local user environment?
It depends on what’s taking so long. Does nix-instantiate shell.nix take long? If so, you can “cache” it by keeping the resulting .drv around and running nix-shell against that rather than on shell.nix itself. Note however that this won’t take into account when shell.nix or the nixpkgs it uses changes, so you’ll want to update it/invalidate the cache periodically. This might look something like this:
expr_mtime=$(stat -c %Y shell.nix)
drv_mtime=$(stat -c %Y shell.drv)
# Can't just use -nt because we want the shell.drv symlink's mtime,
# not that of the .drv itself which is 1
if [[ -z $drv_mtime ]] || [[ $expr_mtime -gt $drv_mtime) ]] ; then
nix-instantiate shell.nix --indirect --add-root shell.drv
fi
nix-shell shell.drv --run 'do-the-tests'
I think you’d need the git checkout and the nix store within the container to be on a volume so that it can persist across runs. I’m not sure how this would work as far as the initial contents (necessary to run nix) are concerned.