Docker + nix + gitlab ci: cache /nix/store per-machine

I’m trying to make gitlab ci with the following objectives:

  • cache /nix/store per machine
  • can run on any docker runner (runner can be installed on any os)
  • download remote packages

Here is the test repo https://gitlab.com/srghma/nix-docker-cached-store-reproduction

Tests are running inside nixos/nix docker container

Gitlab allows to cache:

  • directories in project root $CI_PROJECT_DIR
  • /cache dir (created automatically)

Since it’s not possible to cache /nix/store directory, I’m trying to:

  • cache $CI_PROJECT_DIR/.mycache dir
  • custom nix store stored in $CI_PROJECT_DIR/.mycache/nix/store
  • specify custom store path echo "store = $(pwd)/.mycache" > /etc/nix/nix.conf
  • run tests with nix run --store $CI_PROJECT_DIR/.mycache/nix/store --file ./nix/pkgs.nix <dep1> <dep2> run-tests

I have stumbled on issue

nix run for some reason changes GID of $CI_PROJECT_DIR/.mycache/nix/store to 65534

then if I run
nix run --file ./nix/pkgs.nix nix --command nix-instantiate --eval --strict -E '(import <nixpkgs> {}).system'
it will fail with
error: changing ownership of path '/builds/srghma/nix-docker-cached-store-reproduction/.mycache/nix/store': Invalid argument

Question:
Why it changes GID? I expect it to leave GID of 30000

N.B. I have tried to make some debugging here https://gitlab.com/srghma/nix-docker-cached-store-reproduction/-/jobs/153809309

N.B.

$ id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
$ nix run --file ./nix/pkgs.nix nix --command id
uid=0(root) gid=0(root) groups=0(root),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody)
# ???
1 Like

Maybe not quite the answer… Is it possible to move /nix to /cache after tests are run, and populate /nix from /cache at start?

I would avoid that cause it takes some time

It became clear for me that i can’t use nix-build inside nix run --store... (chroot) inside docker, so I decided to try nix-build --store ... inside docker

Now i can’t access network

1 Like

Current state of research

Did you make any further progress on this? I’m in the same situation and don’t really know what to do.

Hi, in comment above you can find my solution that works without copying

Its a hack on how gitlab makes caching volume for docker runner

Cons:

  • gitlab knows nothing about /cache/nix, so you are not able to clean to on UI
  • it doesn’t remove stale packages once a week

I ended up using https://cachix.org/ for caching. The setup is basically:

$ cat .gitlab-ci.yml
image: nixos/nix:latest

before_script:
  - nix-env -iA nixpkgs.cachix
  - cachix use insertyourcachixnamehere

build:
  stage: build
  script:
    - nix-build

Before I push build on CI, I do nix-build | cachix push insertyourcachixnamehere on my dev machine. It uploads store paths to my personal cache, and CI downloads them instead of rebuilding them. I could cut down the pipeline time from 30 minutes to 3 that way.

2 Likes