Infrastructure Announcement: The future of OfBorg – Your Help Needed!

Overview

OfBorg is a CI (Continuous Integration) system that runs important checks on each pull request. Here’s what it does:

  • Verifies that the pull request doesn’t break the Nixpkgs evaluation.
  • Labels the pull request based on the number of package rebuilds it causes.
  • Builds packages and tests based on the commit messages.

OfBorg operates on infrastructure provided by the NixOS Foundation and is currently sponsored by Equinix Metal. Here’s an overview of the infrastructure setup:

  • 1x c3.small.x86
  • 5x m3.large.x86
    • Evaluator/Builder
  • 1x c3.large.arm64
    • Builder
    • also our Community-Builder
  • 1x c3.medium.x86
    • netboot-foundation

Furthermore it uses 4 mac minis running builds at mac stadium.

Issue: Ending of Equinix Metal Sponsorship

Equinix Metal’s sponsorship is set to end this year. As a result, we need to find alternative resources for running our builds. Our infrastructure team is small, and with our current workload of reducing binary cache sizes and maintaining Hydra, we have decided to no longer maintain OfBorg infrastructure on our own.

Solution: Github actions runner

We have 47 days left finish either of them.

When OfBorg was first introduced, we were using Travis CI, which had limited resources. Today, GitHub provides better infrastructure, including builders with 16GB RAM, which can efficiently handle the tasks OfBorg performs, such as evaluating Nixpkgs.

We’ve tested this in GitHub Pull Request #352808 and found it can evaluate Nixpkgs in under 4 minutes for all four core architectures (see).
This is even faster than what we see with ofBorg today! Lily Foster noted that this is not the whole evaluation that ofBorg runs today. In my benchmarks when using ofborg’s method it was ~16 minutes, which is still faster than the hours of queuing we experience in ofborg lately.

GitHub limits us to 20 concurrent builds per organization. To manage this, we can run GitHub Actions in contributors’ forks instead of directly on pull requests. This will distribute the load across contributors, giving us the capacity we need. Note that building actions on a forked repository is not against the terms of service. It is a documented feature in github’s documentation and used by millions of repository i.e. the nix repository

One downside is that build status won’t be immediately visible on pull requests. To address this, we can set up something as simple as letting the ci write a status message in the pull request or having a small web service that receives notifications when builds complete on a fork, allowing us to update the pull request with the build status.

Using github actions also could have other benefits for something ofborg cannot provide today:

  • It allows contributors to debug builds using the tmate action.
  • We could add support for allowing contributors to add their own binary cache in their own repository settings so that the build results could be also used.

Call for Help: Github runner taskforce

To meet the end-of-year migration deadline, we need your help! Here are the key tasks:

We need to replace OfBorg’s functions with GitHub Actions scripts:

  • Running evaluation checks on Nixpkgs/Manual/NixOS.
  • Identifying package rebuilds and adding appropriate labels to the repository.
  • (Optional) Rebuilding selected packages for Linux/macOS.

Your contributions will help make this transition a success! Let us know if you can take on any of these tasks.

Update

We have our first jitsi meeting to coordinate the migration on the 14.11 (today) at 17:00 UTC (18:00 Berlin time) at Jitsi Meet

52 Likes

Here is a tracking issue for the github action migration: Replacing ofBorg with Github actions · Issue #355847 · NixOS/nixpkgs · GitHub

4 Likes

One downside is that build status won’t be immediately visible on pull requests. To address this, we can set up something as simple as letting the ci write a status message in the pull request or having a small web service that receives notifications when builds complete on a fork, allowing us to update the pull request with the build status

Confused about this. Doesn’t GitHub Actions show build status? As soon as the action is queued?

It doesn’t show the github action checks coming from a fork in the pull request itself. You have to visit the forks github action tab for that.

I’m willing to lend a hand in this. I have some experience with CIs (Junior DevOps) but mainly with Jenkins, though I want to help.

Make sure to also close GitHub Actions to replace grahamofborg-eval · Issue #99722 · NixOS/nixpkgs · GitHub :slight_smile:

This is an idea I’ve had in the back of mind for a while, but not the focus to work towards. If we had this service, which could integrate with the Checks on the PRs, it could provide a similar (better!) experience to ofborg without the centralized building component. If we design the service correctly, we could receive reports from various sources including a forked repo’s Actions, but would allow us to not be fully coupled to GH for everything.

2 Likes

We can also run them on garnix (on their own infrastructure). We’ve we every once in a while have people accidentally enable garnix in their fork of Nixpkgs and then in addition enable automated syncing of the fork, so at various time we have done a couple hundreds evals (and the building of all the flake packages) a day of Nixpkgs a day already. Not the same as building everything continuously, but I’m sure we can pretty easily get its own infra running to manage that.

We are happy to all the dev work for free. We might also be able to potentially sponsor all the costs, but I’d have to check. Either way, with us covering dev costs, it’ll likely be less than 20% of the Equinix list price.

11 Likes

Would garnix work for macOS? IIRC there was an experimental offering at some point but I’m not sure if that’s still a thing. Darwin CI is a major pain point right now and something I’ve been planning to try and solve for the 25.05 release cycle (though of course the need to potentially rework ofborg entirely means I’m not yet sure the best way to go about that).

1 Like

For garnix we would likely need a way to dynamically generate what needs to be build. Is this something we can do with garnix?

We have aarch_64-darwin support already. We can do x86 as well I think

1 Like

Would you want to build this for us?

Generated from what? We have two new features that we haven’t announced yet that might work here, but we’d have to check.

We have the package attribute in the beginning of the commit title and we want to build that including passthru.tests.

Yeah that should be okay

That’s nice. Than we can already cover aarch64-{darwin,linux} and x86_64-linux. That build load is hopefully also not as bad as evaluating nixpkgs.

Covering aarch64-darwin covers x86_64-darwin. We have no native Intel Mac builders already.

2 Likes