Docker + nix + gitlab ci: cache /nix/store per-machine

I’m trying to make gitlab ci with the following objectives:

  • cache /nix/store per machine
  • can run on any docker runner (runner can be installed on any os)
  • download remote packages

Here is the test repo Serhii Khoma / nix-docker-cached-store-reproduction · GitLab

Tests are running inside nixos/nix docker container

Gitlab allows to cache:

  • directories in project root $CI_PROJECT_DIR
  • /cache dir (created automatically)

Since it’s not possible to cache /nix/store directory, I’m trying to:

  • cache $CI_PROJECT_DIR/.mycache dir
  • custom nix store stored in $CI_PROJECT_DIR/.mycache/nix/store
  • specify custom store path echo "store = $(pwd)/.mycache" > /etc/nix/nix.conf
  • run tests with nix run --store $CI_PROJECT_DIR/.mycache/nix/store --file ./nix/pkgs.nix <dep1> <dep2> run-tests

I have stumbled on issue

nix run for some reason changes GID of $CI_PROJECT_DIR/.mycache/nix/store to 65534

then if I run
nix run --file ./nix/pkgs.nix nix --command nix-instantiate --eval --strict -E '(import <nixpkgs> {}).system'
it will fail with
error: changing ownership of path '/builds/srghma/nix-docker-cached-store-reproduction/.mycache/nix/store': Invalid argument

Question:
Why it changes GID? I expect it to leave GID of 30000

N.B. I have tried to make some debugging here test (#153809309) · Jobs · Serhii Khoma / nix-docker-cached-store-reproduction · GitLab

N.B.

$ id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
$ nix run --file ./nix/pkgs.nix nix --command id
uid=0(root) gid=0(root) groups=0(root),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody)
# ???
3 Likes

Maybe not quite the answer… Is it possible to move /nix to /cache after tests are run, and populate /nix from /cache at start?

1 Like

I would avoid that cause it takes some time

1 Like

It became clear for me that i can’t use nix-build inside nix run --store... (chroot) inside docker, so I decided to try nix-build --store ... inside docker

Now i can’t access network

https://github.com/NixOS/nix/issues/2663

2 Likes

Current state of research

1 Like

Did you make any further progress on this? I’m in the same situation and don’t really know what to do.

1 Like

Hi, in comment above you can find my solution that works without copying

Its a hack on how gitlab makes caching volume for docker runner

Cons:

  • gitlab knows nothing about /cache/nix, so you are not able to clean to on UI
  • it doesn’t remove stale packages once a week
1 Like

I ended up using https://cachix.org/ for caching. The setup is basically:

$ cat .gitlab-ci.yml
image: nixos/nix:latest

before_script:
  - nix-env -iA nixpkgs.cachix
  - cachix use insertyourcachixnamehere

build:
  stage: build
  script:
    - nix-build

Before I push build on CI, I do nix-build | cachix push insertyourcachixnamehere on my dev machine. It uploads store paths to my personal cache, and CI downloads them instead of rebuilding them. I could cut down the pipeline time from 30 minutes to 3 that way.

6 Likes

We also use Cachix on Gitlab, though we use a framework to avoid the boilerplate

The side benefit of cachix is that when you build locally it’ll also use the cache, and when you build locally you are also populating the cache for the CI. So win-win

The infrastructure is simply deployed with terraform:

The arch is this:

It runs thousand of pipelines a day where each pipeline has 100+ jobs. The arch auto-scales automatically, making it pretty fast and cost-effective

3 Likes

Anyone tried to mount read-only a network storage as /nix/store on the workers?

2 Likes

That’s so great! :slight_smile:

I’ve implemented 5min caching for narinfo files as Cachix is getting 1000 req/s at peaks, which has reduced the load significantly. There were a few short bursts of high load since Friday.

In other news, Nix 2.4 reduces number of requested narinfos quite a bit so this will all get better soon: Short-circuit querying substituters on success by Kha · Pull Request #5109 · NixOS/nix · GitHub

5 Likes
Hosted by Flying Circus.