Docker + nix + gitlab ci: cache /nix/store per-machine


#1

I’m trying to make gitlab ci with the following objectives:

  • cache /nix/store per machine
  • can run on any docker runner (runner can be installed on any os)
  • download remote packages

Here is the test repo https://gitlab.com/srghma/nix-docker-cached-store-reproduction

Tests are running inside nixos/nix docker container

Gitlab allows to cache:

  • directories in project root $CI_PROJECT_DIR
  • /cache dir (created automatically)

Since it’s not possible to cache /nix/store directory, I’m trying to:

  • cache $CI_PROJECT_DIR/.mycache dir
  • custom nix store stored in $CI_PROJECT_DIR/.mycache/nix/store
  • specify custom store path echo "store = $(pwd)/.mycache" > /etc/nix/nix.conf
  • run tests with nix run --store $CI_PROJECT_DIR/.mycache/nix/store --file ./nix/pkgs.nix <dep1> <dep2> run-tests

I have stumbled on issue

nix run for some reason changes GID of $CI_PROJECT_DIR/.mycache/nix/store to 65534

then if I run
nix run --file ./nix/pkgs.nix nix --command nix-instantiate --eval --strict -E '(import <nixpkgs> {}).system'
it will fail with
error: changing ownership of path '/builds/srghma/nix-docker-cached-store-reproduction/.mycache/nix/store': Invalid argument

Question:
Why it changes GID? I expect it to leave GID of 30000

N.B. I have tried to make some debugging here https://gitlab.com/srghma/nix-docker-cached-store-reproduction/-/jobs/153809309

N.B.

$ id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
$ nix run --file ./nix/pkgs.nix nix --command id
uid=0(root) gid=0(root) groups=0(root),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody)
# ???

#2

Maybe not quite the answer… Is it possible to move /nix to /cache after tests are run, and populate /nix from /cache at start?


#3

I would avoid that cause it takes some time


#4

It became clear for me that i can’t use nix-build inside nix run --store... (chroot) inside docker, so I decided to try nix-build --store ... inside docker

Now i can’t access network


#5

Current state of research