Docker + nix + gitlab ci: cache /nix/store per-machine

Development Deployments
1

I’m trying to make gitlab ci with the following objectives:

  • cache /nix/store per machine
  • can run on any docker runner (runner can be installed on any os)
  • download remote packages

Here is the test repo Serhii Khoma / nix-docker-cached-store-reproduction · GitLab

Tests are running inside nixos/nix docker container

Gitlab allows to cache:

  • directories in project root $CI_PROJECT_DIR
  • /cache dir (created automatically)

Since it’s not possible to cache /nix/store directory, I’m trying to:

  • cache $CI_PROJECT_DIR/.mycache dir
  • custom nix store stored in $CI_PROJECT_DIR/.mycache/nix/store
  • specify custom store path echo "store = $(pwd)/.mycache" > /etc/nix/nix.conf
  • run tests with nix run --store $CI_PROJECT_DIR/.mycache/nix/store --file ./nix/pkgs.nix <dep1> <dep2> run-tests

I have stumbled on issue

nix run for some reason changes GID of $CI_PROJECT_DIR/.mycache/nix/store to 65534

then if I run
nix run --file ./nix/pkgs.nix nix --command nix-instantiate --eval --strict -E '(import <nixpkgs> {}).system'
it will fail with
error: changing ownership of path '/builds/srghma/nix-docker-cached-store-reproduction/.mycache/nix/store': Invalid argument

Question:
Why it changes GID? I expect it to leave GID of 30000

N.B. I have tried to make some debugging here test (#153809309) · Jobs · Serhii Khoma / nix-docker-cached-store-reproduction · GitLab

N.B.

$ id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
$ nix run --file ./nix/pkgs.nix nix --command id
uid=0(root) gid=0(root) groups=0(root),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody),65534(nobody)
# ???
4 Likes
2

Maybe not quite the answer… Is it possible to move /nix to /cache after tests are run, and populate /nix from /cache at start?

1 Like
3

I would avoid that cause it takes some time

1 Like
4

It became clear for me that i can’t use nix-build inside nix run --store... (chroot) inside docker, so I decided to try nix-build --store ... inside docker

Now i can’t access network

https://github.com/NixOS/nix/issues/2663

2 Likes
5

Current state of research

1 Like
6

Did you make any further progress on this? I’m in the same situation and don’t really know what to do.

1 Like
7

Hi, in comment above you can find my solution that works without copying

Its a hack on how gitlab makes caching volume for docker runner

Cons:

  • gitlab knows nothing about /cache/nix, so you are not able to clean to on UI
  • it doesn’t remove stale packages once a week
1 Like
8

I ended up using https://cachix.org/ for caching. The setup is basically:

$ cat .gitlab-ci.yml
image: nixos/nix:latest

before_script:
  - nix-env -iA nixpkgs.cachix
  - cachix use insertyourcachixnamehere

build:
  stage: build
  script:
    - nix-build

Before I push build on CI, I do nix-build | cachix push insertyourcachixnamehere on my dev machine. It uploads store paths to my personal cache, and CI downloads them instead of rebuilding them. I could cut down the pipeline time from 30 minutes to 3 that way.

6 Likes
9

We also use Cachix on Gitlab, though we use a framework to avoid the boilerplate

The side benefit of cachix is that when you build locally it’ll also use the cache, and when you build locally you are also populating the cache for the CI. So win-win

The infrastructure is simply deployed with terraform:

The arch is this:

image
image1392×794 57.2 KB

It runs thousand of pipelines a day where each pipeline has 100+ jobs. The arch auto-scales automatically, making it pretty fast and cost-effective

5 Likes
10

Anyone tried to mount read-only a network storage as /nix/store on the workers?

2 Likes
11

That’s so great! :slight_smile:

I’ve implemented 5min caching for narinfo files as Cachix is getting 1000 req/s at peaks, which has reduced the load significantly. There were a few short bursts of high load since Friday.

In other news, Nix 2.4 reduces number of requested narinfos quite a bit so this will all get better soon: Short-circuit querying substituters on success by Kha · Pull Request #5109 · NixOS/nix · GitHub

4 Likes