Getting unique ID from nix expression to avoid redundant build

Help
1

I’d like to build a new docker image when a new commit is pushed to my repository and there is a change to the nix expression from previous build.
I’m planning to use docker image as an efficient way to cache shell.nix for CI environment.

Using pre-built binary from official binary cache server solves half of the problem. nix-shell still needs to fetch those packages from the Internet to run. I want to avoid relying on internet connection when possible to reduce possible failure points. Due to how nix works, nix-shell fetches build-time packages even when binary cache is available, making cold start even slower.

So I’d like to cache packages excluding build-time one, but how could this be done?
First solution I came up with is packaging all tools written in shell.nix as a docker image, and use that on CI.
With that, docker image is reused as long as the same CI runner is available. I’m on AWS, so I could also set up lifecycle rules to discard older images. If runners are running on ECS or Kubernetes, older images also get discarded automatically on each machine.
I don’t want to version the image manually (increment version number every time nix expression changes), so being able to generate docker image tag from resulting nix expression would be nice.

I’m not sure the same thing could be done without docker. There are some tools that runs nix-shell with cached packages, could those tools run on CI? Probably I still need to generate unique id to be used as a cache key.

2

We use buildkite, and just have have the CI do nix-shell --run 'test command'. The buildkite agents are spot instances, so they eventually get cycled, so we don’t have to worry about disk space becoming too much of an issue.

With nix, having less purity is good. Assuming the build environments are leaking somehow.

I can’t speak about docker approaches.

3

I’m not totally understanding what you want to do, but maybe the inputDerivation of the shell is what you want.

https://github.com/NixOS/nixpkgs/pull/95536

4

Thanks, so there are at least a few other options available.

  • Let each CI runner cache nix packages locally, and recycle runners as needed.

Or…

  • Don’t worry too much about nixpkgs server reliability and fetch packages every time. It’s usually fast enough even if you fetch all packages every time, and it’s rare to go down.

My initial plan was

  1. Just use shell.nix on local machine for development environment
  2. For the use with production environment, build docker image with the same package combination as shell.nix using pkgs.dockerTools.buildImage. Using the same package combination could be done by referencing the same file (packages.nix?) from shell.nix and docker.nix The image should be built before running pipelines.

However, I’m not sure if there is a good way to make sure that packages of docker image built with step 2 is identical with shell.nix one.

5

First I’d like to ask why do you need a nix-shell in your CI?
If all your builds are done via nix, then there is no reason to load an environment via nix-shell first. Just trigger all your builds via nix build. You don’t need to laod your dev environment for building.

The only case that I can imagine, where you want do do that, is if your build requires other tools than nix, in which case I would recommend to port all your builds to nix instead of putting much effort into complicated workarounds via docker.

If you trigger your builds via nix build, nix won’t load any build time dependencies if your builds are already cached, except your build does IFD (import from derivation), which would be the case if you use tools like mach-nix.

But still, I think, its easier to just setup your own cache for quickly fetching build time dependencies or results. The simplest way of doing that is probably using https://cachix.org/.

I think the docker workaround increases the complexity of your setup unnecessarily.

If, for some reason, you still want to continue with that docker approach, then I have the following ideas.

  • Consider a nix shell, built via mkShell, is basically just a list of nix packages (buildInputs) + environment variables + shell commands (shellHook)
  • Those same packages + variables + commands can be used to put into a docker image
  • You could write a nix function that takes those inputs and then either calls mkShell or dockerTools.buildLayaerdImage with it, so you will have one interface for both outputs.

Also consider that a nix-shell is by default not pure (except you use the --pure flag) and inherits the environment of your current system. Therefore there will probably be a discrepancy between your nix shell and you docker image, as there are discrepancies between all your dev machines, if they don’t use the --pure flag.

Concerning the unique ID for you nix expression: If your nix expression returns a derivation (which it usually does), then the /nix/store/… path of this derivation is your unique ID. To get that store path, you need to evaluate the nix expression via nix. That’s the only reliable way of getting a unique ID if you don’t want to include dirty hacks which will most likely break nix’ fundamentals in a terrible way.

So, hacking your way around the nix evaluation to optimize things, kind of defeats the purpose of nix, doesn’t it?

6

What I want to prepare with nix-shell is set of tools for working with AWS environment (Terraform, Terragrunt, AWS CLI, kubernetes CLI tools, etc). Sorry for not being clear.
All tools are already available as binary cache, so there is no way to optimize this part further.

Terraform and other CLI tool’s purpose is not building a software, so I guess those tools can’t be integrated with nix-build.

We update those tools regularly not to introduce too much changes to our environment at once.
We don’t update tools that often, about once a month. Still, this might cause tool version inconsistencies between environments.

Generating unique ID from nix expression by just reading files seems impossible without ugly hacks, as you mentioned. So only available options are…

  • Just run nix-shell on CI, downloading dependencies from binary cache every time.
  • Give up using the same repo for everything, and create another git repository for baking docker image. You can test the image locally by temporary changing image tag of docker-compose.yml. After testing it locally, build the image on CI and push to Amazon ECR. Then change image tag of docker-compose.yml

I’ll try out more to find out which method best suits me, thank you.