How to use attic for all packages in CI

Help
1

Hi, I added attic to my CI run, and it’s avoiding rebuilds of the main project nicely.

The commands are:

nix run -I nixpkgs=channel:nixos-unstable nixpkgs#attic-client login myaccount https://attic.example.com ${ATTIC_SECRET}
nix run -I nixpkgs=channel:nixos-unstable nixpkgs#attic-client use myaccount:build
nix build .#mydockeimager
cp result /build/dockerimage
nix run -I nixpkgs=channel:nixos-unstable nixpkgs#attic-client push myaccount:build ./result

(sure, could add attic to env, doesn’t matter much for the question)

This works well. It checks the cache and re-uses any previous build. However, I would like it to cache /everything/ on my attic instance, that’s all the dependencies, and also everything copied from cache.nixos.org. Mostly to be not using so much bandwidth from nixos.org (I find it super impolite to download 100Mb from them every CI run), but also it misses some local derivations this way I think.

Any ideas how to make it push everything needed for the build? attic has a watch mode, but that doesn’t look useful in CI mode; no way to know when attic is done uploading.

I tried attic cache configure build --upstream-cache-key-name '' but that didn’t change much. Thoughts?

1 Like
2

Since this runs in a CI build the nix store should be pretty minimal, so I thought I could do:

attic push --ignore-upstream-cache-filter mycache $(ls -d /nix/store/*/ | grep -v fake_nixpkgs)

And then set the cache with attic cache --priority 1. That works! Builds now get their basic files from my attic, not from http://cache.nixos.org. However, pushing all the resources /to/ attic takes about 8 minutes, but it looks like it’s a one time thing so it’s not too bad. My attic machine is a crappy one (cloud VM), and it seems to do a lot of work somewhere. So, progress, not 100% happy with this yet. tbc.

3

And finally the build now uses a 3 line Dockerfile with FROM nixpkgs/nix-flakes, which adds nixpkgs#attic-client + grep. That means a CI run now only uses local resources: no download from Docker hub of the build image, and no download of attic from cache nixos org. I’ll let that Docker image auto build once a week or so.
This looks workable.