ron
November 24, 2024, 5:39pm
1
Short TL;DR (More information can be found in Mic92s post): Infrastructure Announcement: The future of OfBorg โ Your Help Needed! - Announcements - NixOS Discourse
Equinix Metal has been providing in-kind sponsorship/funding to our needs for a while! Thank you to Edward Vielmetti and the entire team there. Equinix needs to roll back this funding going into next year and we are launching an effort to ensure continuity for the short term and sustainability for the longer term.
The board (aka me), @Ericson2314 , @fpletz , @tomberek , @hexa , @Mic92 , @winterqt and @infinisil recently met to discuss how to move forward with a replacement for the Equinix machines. In short:
Evaluation needs will be taken care of by GitHub Actions going forward. @infinisil and @Mic92 are tending to that, starting with Parallel GH actions workflow for Nixpkgs eval by infinisil ยท Pull Request #356023 ยท NixOS/nixpkgs ยท GitHub , but will need some follow-ups.
To take care of builds in the short-term future, @hexa from the infra team will make a funding proposal to the foundation for renting necessary machines, similar to the draft in Infrastructure Announcement: The future of OfBorg โ Your Help Needed! - #34 by Mic92 .
For the long-term future, we are looking into sponsorships of hardware and a place to host it. The board and infra team will follow up on some potential previous offers from Flying Circus, Ampere, Wobcom, Obsidian Systems. Appreciation to all of these teams!
As such, there is no immediate critical concern at the moment and we will keep updating.
If you have connections to companies that might also be able to help us out with 3., please reach out to foundation@nixos.org or infra@nixos.org , thank you!
18 Likes
hexa
December 3, 2024, 1:33am
2
The proposal is up and ready for review. PTAL
opened 04:53PM - 27 Nov 24 UTC
With the Equinix Metal sponsorship coming to and end around EOY we need to take โฆ steps to keep our build pipelines working, so that we can keep delivering package and channel updates to our users.
To that end, as a short-term measure, I propose to rent an adequate number of machines from Hetzner, which in a comparison with other popular hosters (e.g. OVH, Scaleway) seems to be among the most cost-effective options.
The plan is for this to be a transitional period, while we investigate going for our own bare metal machines, that we plan to host with various sponsors.
## Replace hydra builders
The proposal is to replace the two ARM64 builders with similar machines, so that we can keep the number of ARM64 builders at three. For x86_64 I propose we rent four machines, since we regularly provide x86_64-linux build capacity to test bulk updates for bigger changesets and package set updates.
- 2x [RX220](https://www.hetzner.com/dedicated-rootserver/rx220/configurator/#/) (Ampere Q80-30, 256 GB DDR4) ~270 EUR/mo
- we already have one of these at Hetzner
- 4x [AX162-R](https://www.hetzner.com/dedicated-rootserver/ax162-r/configurator/#/) (AMD Epyc 9454P, 48C/96T, 256 GB DDR5) ~240 EUR/mo
Total: 1500 EUR/mo + 600 EUR setup cost
## Upgrade for hydra.nixos.org
On top of this comes a required update for rhea.nixos.org, the machine that hosts hydra, which we already host at Hetzner. It is currently bottlenecked by the compression load of incoming build results and cannot go any faster. Additionally, it [requires more RAM](https://grafana.nixos.org/goto/jEdu9w7NR?orgId=1) to effectively deal with evaluations, for which we currently extensively rely on [`zramSwap`](https://github.com/NixOS/infra/commit/b349b94c650e803cce59cbf9f0fdfb935db4cef0).
- 1x [AX162-R](https://www.hetzner.com/dedicated-rootserver/ax162-r/configurator/#/) (upgrade from existing AX101) ~240 EUR/mo
- up from 32 to 96 parallel compression slots
- up from 128 to 256GB RAM for evals
Total: 240 EUR/mo (up from 107 EUR/mo) and 100 EUR setup cost
4 Likes