I ended up using https://cachix.org/ for caching. The setup is basically:
$ cat .gitlab-ci.yml
image: nixos/nix:latest
before_script:
- nix-env -iA nixpkgs.cachix
- cachix use insertyourcachixnamehere
build:
stage: build
script:
- nix-build
Before I push build on CI, I do nix-build | cachix push insertyourcachixnamehere
on my dev machine. It uploads store paths to my personal cache, and CI downloads them instead of rebuilding them. I could cut down the pipeline time from 30 minutes to 3 that way.