hexa
November 6, 2022, 11:03pm
1
Hey everyone!
Today we start the Zero Hydra Failures campaign for the upcoming 22.11
This is where we target our efforts towards stabilizing the master branch ahead of the branch-off on 2022-11-21, after which fixes continue to roll in, but will need to be backported.
Zero Hydra Failures ends with the planned release at the end of November.
Head over to GitHub and read the instructions to get started!
opened 10:43PM - 06 Nov 22 UTC
Hi, we are Janne Heß & Martin Weinelt, the release managers for NixOS 22.11 :rac… coon: ("Raccoon").
Today we want to invite everyone to participate in the Zero Hydra Failures Project, wherein we prepare the package set for the upcoming release, up until its public release at the end of november.
There are only two more upcoming dates that we'd like to mention in that context:
- 2022-11-21: Branch-off
- 2022-11-30: 22.11 Release
The complete timeline can be found in the
- #193585.
## The mission
Every time we plan to do a release, we take time to stabilize the master branch and later on the release branch. Our goal here is to reduce the number of failing jobs on the `nixpkgs:trunk` `nixos:trunk-combined` jobsets as much as possible before branch-off. We call this the "Zero Hydra Failure" campaign.
Besides aiming for zero failed jobs, we also strive to again provide all packages that were available in the previous release.
Changes need to always target the `master` branch. Take note that the branch-off will occur on 2022-11-21, after which ZHF changes will need to be tagged with the `backport: release-22.11` label to land in the stable release.
## Jobsets
The relevant jobsets to check for failing jobs are:
- [nixos:trunk-combined](https://hydra.nixos.org/jobset/nixos/trunk-combined) (Linux packages & NixOS tests)
- [nixpkgs:trunk](https://hydra.nixos.org/jobset/nixpkgs/trunk) (Linux & Darwin packages)
<!--
- [nixos:release-22.11](https://hydra.nixos.org/jobset/nixos/release-22.11) (Linux packages & NixOS tests)
- [nixpkgs:nixpkgs-22.11-darwin](https://hydra.nixos.org/jobset/nixpkgs/nixpkgs-22.11-darwin) (Darwin packages)
-->
## Workflow
### Finding broken packages
#### Eval reports
Evaluation reports provide a structural overview of the most impactful failing builds. They originated at https://github.com/nix-community/nix-review-tools and were automated over at https://github.com/malob/nix-review-tools-reports.
1. Navigate to https://malob.github.io/nix-review-tools-reports/
2. Open the relevant jobset
3. Browse the latest reports for build failures
4. Follow the links to the build failure on hydra
#### ZERO Hydra Failures
The platform automatically crawls Hydra and lists packages by maintainer and lists the most important dependencies (failing packages with the most dependants). It also graphs the general trend per platform.
1. Navigate to https://zh.fail
#### Check on packages you maintain
1. Clone nixpkgs and checkout the `master` branch
2. Run
```
nix-build maintainers/scripts/build.nix --argstr maintainer <name>
```
#### Hydra
Hydra is nixpkgs CI platform, where all active branches are built and pushed into the cache, after which channels can originate from its build results.
1. Open the [nixpkgs:trunk](https://hydra.nixos.org/jobset/nixpkgs/trunk) jobset
2. Select the latest evaluation
4. Directly failing jobs are marked with a red cross, while transitively failing ones are greyed out.
5. Use the search form to scope the package list to things relevant to you and that you can test.
### Submit fixes
6. Search through PRs to make sure noone provided a fix yet. If there is one, please take the time and help review the change.
7. If there is no open PR, troubleshoot why it's failing and fix it.
8. Pull Request the fix against the `master` branch and wait potential review & change requests
- If your PR causes more than ~500 rebuilds, it is generally preferred to target `staging` to avoid compute churn for users on `master`.
### Backporting
**After 2022-11-21**
9. Apply the relevant backport label to land the fix in the release branch
- Changes to `master` get backported into `release-22.11`
- Changes to `staging` get backported into `staging-22.11`
10. If the backport action fails, follow the [manual backporting steps](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#backporting-changes). Make sure to use `git cherry-pick -x <rev>` on all commits intended for backport.
- @jonringer created a [video covering the backport process](https://www.youtube.com/watch?v=4Zb3GpIc6vk).
---
Always link back to this issue by mentioning the issue number in the description of your pull request:
```
ZHF: #199919
```
If your PR receives no reviews or does not get merged, feel free to
- add the `0.kind: ZHF Fixes` label, so people can better browse these fixes
- request or mention @NixOS/nixos-release-managers on the PR
## Broken packages
Everything we cannot fix in time will need to be marked broken on the respective platforms, so that Hydra will not retry builds over and over, thereby wasting compute resources.
Set `meta.broken` and add a reference and/or explanation, like this:
```nix
meta = {
# ref to issue/explanation
broken = stdenv.isDarwin; # only broken on darwin
# broken = true; # broken on all platforms.
};
```
## Closing
This is a great way to help NixOS, and it is a great time for new contributors to start their nixpkgs adventure. :partying_face:
As with the [feature freeze issue](https://github.com/NixOS/nixpkgs/issues/194208), please keep the discussion here to a minimum so we don't ping all maintainers (although relevant comments can of course be added here if they are directly ZHF-related) and ping one of the release managers (@mweinelt, @dasJ) in the respective issues.
16 Likes
(“Raccoon”) release.