Why is nixos-unstable behind 26.05?

I stumbled over this because of ollama.
Version 30.5 is already in nixos-26.05 while it is still on version 24.0 on unstable.
Here are the corresponding PRs:
nixos-unstable: ollama: 0.24.0 → 0.30.4 and ollama: 0.30.4 → 0.30.5
nixos-26.05: [Backport release-26.05] ollama: 0.24.0 → 0.30.4 and [Backport release-26.05] ollama: 0.30.4 → 0.30.5

On https://status.nixos.org/ it looks like this:

As nixos-unstable is green i think that everything is ok but under “Last updated” it says 6 days ago which doesn’t sound right to me.

I found also this status page https://hydra.nixos.org/jobset/nixos/unstable


Are the eval errors the culprit that no new unstable version gets released?
If that is the reason why is it healthy (green) in the other status page?

Not directly. It is the tests that need to pass for the channel to be advanced. A failing package could indirectly cause a failing test, but it is not the only reason that a test can fail.

The test status for nixos-unstable is shown here:

See the “Internal channel update process” section here for the process:

1 Like

I could have been more precise i meant with green the “Hydra job for tests” (i imagine that it is a representation over time) and rolling.

I see. I misunderstood.

As far as I am aware, it will show as green provided that all tests have passed, but it will also show as green while pending tests exist, which could be a little misleading.

The 6 days ago “last update time” is correct, it is when the test suite last succeeded and the channel was updated and released.

I had a look and there is this section:

On nixos/unstable/tested is the last successful build from 1 day ago:

On Channel branches - Official NixOS Wiki is this section “When unstable lags behind master”:

As https://status.nixos.org shows, a downside of nixos-unstable is that when the channel is blocked due to hydra failures, other (security) fixes will also not get in. While of course we try to keep hydra green, it is expected that this happens every once in a while. When you want to upgrade or downgrade a single package while leaving the rest of your system on nixos-unstable, you could use this approach.

As https://status.nixos.org/ is at the moment green (the “Hydra job for tests”) i can’t seem to find why unstable is behind.

Ok so the current theory is that somewhere tests are still pending even if the last job on nixos/unstable/tested shows this build/331099469

which i interpret as no tests are pending.

Eval 331099469 will advance the channel once there are no remaining queued jobs any more.

I think this could happen in the next hour.

2 Likes

If you look at the constituents for that job, nixpkgs.mesa_i686.x86_64-linux only completed 53 minutes ago.

1 Like

I was wrong about this:

In addition to the test suite passing, from Channel branches - Official NixOS Wiki :

Success Conditions

For a channel update to succeed, two conditions need to be satisfied:

  • Particular jobset evaluation needs to be completely built ie. no more queued jobs, even if some jobs may fail

  • Particular jobset evaluation’s tested/unstable job needs to be built succesfully

So presently, the “tested/unstable job” has built successfully, but there is still a queued job[1] for the “jobset evaluation”. When this finishes, the channel should advance.

It is green because the nixos:unstable:tested job has succeeded. It is behind because the evaluation has not completed all jobs yet.


  1. This is nixpkgs.exo.aarch64-linux, which has been building for close to 2 hours and will likely fail and timeout as the tests are stuck. ↩︎

1 Like

The channel advance is tied to Latest successful build from a finished evaluation, currently pointing at https://hydra.nixos.org/build/330702919. It does not use Latest successful build, which you note is currently https://hydra.nixos.org/build/331099469.

Since nixos:unstable:tested is a constituent of nixos:unstable, I don’t expect the latest-finished URL will advance until the parent nixos:unstable jobset finishes. At this time, it still has a queued/running job for nixpkgs.exo.aarch64-linux.

In general, this makes sense, since advancing the channel before that binary cache is populated will cause a lot of pain and pointless compiling on user computers, and because the official https://github.com/NixOS/nixos-channel-scripts is also responsible for generating the command-not-found database.

EDIT: yep, exo.aarch64-linux timed out, thus finishing the last job in nixos:unstable, and now latest-finished points to 331099469, which had nixpkgs a799d3e3886d

3 Likes

Backported changes to stable do often get out faster than the corresponding changes to unstable. Sometimes that’s because of hydra failures on unstable, but it doesn’t have to be. Hydra also just doesn’t have as much work to do for stable as for unstable, simply because less is changing.

6 Likes

new nixos-unstable release:

2 Likes

One thing I noticed was that the release came out prior to the build for nixpkgs.exo.aarch64-linux timing out leading to its job being marked as completed.

The release date-time is stated as 2026-06-08 00:39:40 UTC here, while nixpkgs.exo.aarch64-linux is stated as finishing at 2026-06-08 02:20:47 UTC here.

Does anyone know why this might be?

As far as I’m aware, the automated infra lives in https://github.com/NixOS/infra and modules/hydra-mirror.nix defines systemd services which explicitly check the latest-finished endpoint, but that doesn’t preclude a manual update.

My best guess is someone SSH’ed into the server running the nixos-update-channels scripts and manually ran mirror-nixos-branch against the latest endpoint. If someone cared to look at the build logs for https://hydra.nixos.org/job/nixos/unstable/nixpkgs.exo.aarch64-linux, they would have noticed that the build seemed to be stalling in the same place where the last attempted build timed out. The build log was stalled at 91% on a test for at least the last hour of the build, the same spot where the last build stalled before timing out.

2 Likes

Why would release for nixos-unstable be tied to jobs on a different jobset? (nixpkgs/unstable in this case)

EDIT: I restarted that exo job now, too.

I am not sure what you mean, it says that it is part of nixos:unstable.

Build 331064971 of job nixos:unstable:nixpkgs.exo.aarch64-linux

It will be interesting to see if the job completes or times out again. 25 minutes in, and the build log shows it’s reached the same “91%” line where the last job stalled.

# from https://hydra.nixos.org/build/331064971/nixlog/2/tail

....sss..sssssssss.s......s.............s....ss.ssssssssssssssssssssssss [ 81%]
sssssssss.......................................s..................ss... [ 82%]
......s................................................................. [ 83%]
...............................s...ss..................s.s.ss........... [ 85%]
.s............s....ss...x............................................... [ 86%]
......................................................ss.sss..ss...sssss [ 88%]
.ss.s..s.s.ss...sss.s..s..ssssssss.s.s.ssss......x...................s.. [ 89%]
....................ssss................................................ [ 91%]

Unfortunately, restarting this job again actually reset the latest-finished endpoint so it’s back to pointing at the older evaluation haha. The channel release is already out so no big deal but it is kind of funny.

EDIT: Yep, 45 minutes in and the log is still at 91%. A nice reproducible stall :upside_down_face: Eh, hydra will time it out in a few hours and it doesn’t really hurt anything.

I’m sorry. I looked wrong, apparently, probably still half-asleep.

1 Like

Long ago it used to be a problem, but we already have checks, so the channels can’t move backwards anymore.

1 Like

That seems very unlikely. I don’t think we’ve ever done that. (I’m in the small set of people with this kind of access.)

I don’t have any explanation here. It looked like first attempt, so it doesn’t seem like restart after channel update could’ve caused this instance.

2 Likes

Well, if it wasn’t manual, that makes two of us. I suppose life is enriched by mystery.