Overview
OfBorg is a CI (Continuous Integration) system that runs important checks on each pull request. Here’s what it does:
- Verifies that the pull request doesn’t break the Nixpkgs evaluation.
- Labels the pull request based on the number of package rebuilds it causes.
- Builds packages and tests based on the commit messages.
OfBorg operates on infrastructure provided by the NixOS Foundation and is currently sponsored by Equinix Metal. Here’s an overview of the infrastructure setup:
- 1x c3.small.x86
- core.ofborg.org
- Loki (250GB)
- Prometheus/Grafana
- 5x m3.large.x86
- Evaluator/Builder
- 1x c3.large.arm64
- Builder
- also our Community-Builder
- 1x c3.medium.x86
- netboot-foundation
Furthermore it uses 4 mac minis running builds at mac stadium.
Issue: Ending of Equinix Metal Sponsorship
Equinix Metal’s sponsorship is set to end this year. As a result, we need to find alternative resources for running our builds. Our infrastructure team is small, and with our current workload of reducing binary cache sizes and maintaining Hydra, we have decided to no longer maintain OfBorg infrastructure on our own.
Solution: Github actions runner
We have 47 days left finish either of them.
When OfBorg was first introduced, we were using Travis CI, which had limited resources. Today, GitHub provides better infrastructure, including builders with 16GB RAM, which can efficiently handle the tasks OfBorg performs, such as evaluating Nixpkgs.
We’ve tested this in GitHub Pull Request #352808 and found it can evaluate Nixpkgs in under 4 minutes for all four core architectures (see).
This is even faster than what we see with ofBorg today! Lily Foster noted that this is not the whole evaluation that ofBorg runs today. In my benchmarks when using ofborg’s method it was ~16 minutes, which is still faster than the hours of queuing we experience in ofborg lately.
GitHub limits us to 20 concurrent builds per organization. To manage this, we can run GitHub Actions in contributors’ forks instead of directly on pull requests. This will distribute the load across contributors, giving us the capacity we need. Note that building actions on a forked repository is not against the terms of service. It is a documented feature in github’s documentation and used by millions of repository i.e. the nix repository
One downside is that build status won’t be immediately visible on pull requests. To address this, we can set up something as simple as letting the ci write a status message in the pull request or having a small web service that receives notifications when builds complete on a fork, allowing us to update the pull request with the build status.
Using github actions also could have other benefits for something ofborg cannot provide today:
- It allows contributors to debug builds using the tmate action.
- We could add support for allowing contributors to add their own binary cache in their own repository settings so that the build results could be also used.
Call for Help: Github runner taskforce
To meet the end-of-year migration deadline, we need your help! Here are the key tasks:
We need to replace OfBorg’s functions with GitHub Actions scripts:
- Running evaluation checks on Nixpkgs/Manual/NixOS.
- Identifying package rebuilds and adding appropriate labels to the repository.
- (Optional) Rebuilding selected packages for Linux/macOS.
Your contributions will help make this transition a success! Let us know if you can take on any of these tasks.
Update
We have our first jitsi meeting to coordinate the migration on the 14.11 (today) at 17:00 UTC (18:00 Berlin time) at Jitsi Meet
