I ended up using https://cachix.org/ for caching. The setup is basically:
$ cat .gitlab-ci.yml
image: nixos/nix:latest
before_script:
  - nix-env -iA nixpkgs.cachix
  - cachix use insertyourcachixnamehere
build:
  stage: build
  script:
    - nix-build
Before I push build on CI, I do nix-build | cachix push insertyourcachixnamehere on my dev machine. It uploads store paths to my personal cache, and CI downloads them instead of rebuilding them. I could cut down the pipeline time from 30 minutes to 3 that way.