Running native NixOS builds with caching in Gitlab CI

Announcements
1

I’ve been hacking on a nixos module that allows automatic setup and teardown of a gitlab-runner
that supports nix builds natively (i.e. they use the host nix-daemon). This means that you get caching of nix buils for free!

One caveat is that the /nix/store is mounted inside the gitlab-runner container so it’s perhaps
not wise to use this in a shared environment. But for your company or your toy project, this
might be perfect to get cheap rebuilds of your gitlab projects.

If you’re interested. Have a look here:

9 Likes
2

Do you know if I can use the gitlab-runner also on github? I would like to use it for NUR instead of travis ci.

3

Yes you can. Login to Gitlab with your GitHub account.

Then “create new project” and then “CI/CD” and then “connect to GitHub”.

See https://about.gitlab.com/features/github/

4

Does this also work in pull requests?

5

Only from PRs inside the same repo. Not PRs from forks: Backend: Allow CI/CD mirroring to handle forks on GitHub SCM (#5667) · Issues · GitLab.org / GitLab · GitLab

6

As soon as this is available as the nixos gitlab-runner I will definitely be using this.

7

@Mic92 if you have a spare machine we can also get a OSS license from BuildKite. It works a bit like the buildkite runner but is CI-agnostic (but they host the job scheduler / dashboard).

1 Like
8

In Coq, we use a bot to push PRs to branches on GitLab. I’m planning to make the code reusable but in the meantime the same bot could push some other projects’ PRs to GitLab as well. Let me know if you’re interested.

9

Sorry for the OT @Zimm_i48, can I ask you more in detail which kind of bot you have deployed?

10

@azazel75 The “bot” is a simple server listening to webhooks from GitHub and in the case a PR is opened / closed / synchronized it takes an action like force-pushing / deleting a branch named after the PR number to GitLab (the bot does many things besides). If you are interested or want to pursue the discussion, please comment on Support other projects that want to push PRs to GitLab · Issue #23 · coq/bot · GitHub.

11

Just an observation. Such a bot comes with security risks though. It allows anyone to execute code on your Gitlab runner.

On Gitlab, PRs of are built on the Fork’s CI, bypassing this security issue. Which is quite neat.

12

Indeed. This is something you need to be aware of which means you cannot put secrets in environment variables for instance, unless you use “protected environment variables” (a nice GitLab CI feature).

On Gitlab, PRs of are built on the Fork’s CI, bypassing this security issue. Which is quite neat.

I wasn’t aware of this, but this means that the PR is not tested fully if you need some specific runners (i.e. Windows runners) to use it. Trade-offs…

13

In the case of your nixos-gitlab-runner, when building something from an untrusted source, could there be a security risk with regards to what goes into the Nix store? Or is this prevented by the permission settings?

14

You can modify the nix store through all nix- related commands (through nix-daemon) but in no other way as /nix/store is mounted read-only. Builds are run sandboxed and can only touch their own nix store path.

This means builds can’t arbitrarily change already installed nix store paths as far as I know.

It’s the same risk as having a non-root user account on your server installing software through nix. (Nix multi-user mode)

15

More importantly. An arbitrary user has full read access to the nix store. So if you have secrets in there, this person will be able to access it. This is the biggest ‘concern’ but is resolved by not storing sensitive info in your nix store

1 Like
16

resolved by not storing sensitive info in your nix store

but where to store secrets?

17

In gitlab you can easily pass them to the job using custom variables.

2 Likes
18

It’s not the fastest, but I’ve been able to nix-store —export and —import small closures via GitLabs cache mechanism to speed up CI.

19

if you have different jobs running in parallel while using the host daemon, wont it be slow because of locking ?