When are NixOS/nixpkgs changes visible to `nix-channel --update`?

Following up on the discussion here, when do commits on NixOS/nixpkgs end up visible to nix-channel --update? I just assumed that it would be immediate when you’re on unstable a la homebrew. But it seems this is not the case? What’s the schedule for this? How does it work?

There are several channels that take from master, in particular nixpkgs-unstable, nixos-unstable, and nixos-unstable-small. Each of these channels has a different Hydra job set that must succeed. Hydra attempts to build these at regular intervals. If the job for a channel succeeds, then the channel is moved to tho the corresponding commit.

The difference between the channels are their job sets. For instance, the nixos-unstable and nixos-unstable-small job sets a certain set of OS tests to pass. Otherwise, it could happen that a commit on master breaks something fundamental that could result in an unbootable system [1]. nixos-unstable-small has a smaller job set, so that channel generally moves faster, but it catches fewer regressions. Also, use of the small channel can require more local builds, since its coverage is smaller.

Channels are explained in more detail here:
https://nixos.wiki/wiki/Nix_channels

You can check the last time a channel was updated here:
https://status.nixos.org/

(As you can see, unstable was currently last updated 12 days ago, which is probably why you do not see the update.)

You can see what is built here:



[1] This is also the reason why you should probably never build NixOS directly against the master branch.

1 Like

The patch which was causing the failure was removed, not sure why nixos-unstable is so far behind.

This commit by @vcunat should fix the current error:

1 Like

:man_facepalming: I forgot to pull the latest master xD

The status.nixos.org page is nice for that. I know about one blocker issue and it’s shown there on top. (Naturally, we’ve had several different channel-blocking issues since the last nixos-unstable update, which is partially why it’s taking so long.)

2 Likes

@jonringer this bug is blocking nixos-unstable too? https://github.com/NixOS/nixpkgs/issues/96479

Correct, the nixos plasma test and a certain bios usb tests also failed recently. But this one is the most severe among which.

1 Like

Out of curiosity, why not just as a CI job on every commit? Nix should enable very aggressive caching of build artifacts such that only changed derivations require any rebuilding, no?

My understanding is that hydra is not building just a subset of the nixpkgs repository, but all of it, it might be overkill to start a new “build” for every PR, some debounce policy is quite interesting.

Yep, this jobset builds everything: https://hydra.nixos.org/jobset/nixpkgs/trunk

To my understanding, this is the job of OfBorg, the other CI going on. I think the difference is that OfBorg just rebuilds the derivation affected by the PR and is just a check, whereas Hydra will also rebuild its reverse (transitive) dependencies and is definitive. Someone else more familiar with the situation could probably correct me if I’m wrong or explain it better.

1 Like

nixpkg’s level of abstraction is a package. It doesn’t know how to split build artifacts beyond that, so we get 0 caching benefits between PR pushes. If the PR affects a lot of other packages, this will cause a lot of rebuilds as well, further diminishing the value of running hydra. The value to cost ratio becomes too low if you ran hydra on every PR commit change.

We even have the staging branch to avoid this kind of churn, where many “large rebuild” commits are consolidated into a single change set to avoid populating hydra resources with build artifacts that are not very likely to be used by anyone. The set of changes are then stabilized by this hydra job https://hydra.nixos.org/jobset/nixpkgs/staging-next where the regressions are analyzed to ensure that critical scenarios weren’t broken. Once it’s “stable enough”, it will land in master.

It specifically just tries to build and test the installable name of the last commit message of a given push https://github.com/NixOS/ofborg#automatic-building if you want it to build multiple packages, you have push many times.

Ofborg does a few other things, likes validates the meta field, determines the number of rebuilds, and does a rough sanity check that most expressions will succeed in evaluation.

Hydra is split into jobsets and each jobset will evaluate and build a set of packages. So it doesn’t really know so much about reverse dependencies as it knows what packages it should be building for a given evaluation. What makes hydra special is that it will cache the build results of a given .drv and so it “should” only need to attempt to build something once, ever.

The list of packages is defined by the hydra jobset. For example, trunk’s “nix expression” is pkgs/top-level/release.nix https://hydra.nixos.org/jobset/nixpkgs/trunk#tabs-configuration. If you look inside the file you can see that it knows about the target systems it’s going to build for and it lists the critical packages needed for a release. This is also why you see the packages having names such as <pkg>.<system>, as hydra will submit jobs to many platforms and architectures.

2 Likes