Future of channels and channels.nixos.org in a Flakes world?

Development
1

As it stands, it seems that there’s a good chance that Flakes will be the future. Then where would channels and channels.nixos.org go? Here are some of my thoughts:

I feel that less reliance on GitHub for releases might be a good idea. If we decide to leave GitHub some day for some reason, it would be nice if we don’t suddenly leave countless mentions of github:NixOS/nixpkgs lying around.

So maybe we should add a way to refer to channels in a Flake URL? Here’s a possible way that I thought of:

  1. When a user specifies an http/https URL, and it leads to (possibly several) redirects, we instead record the final redirection destination in flake.lock.
  2. When a flake.lock is consulted to download the tarball, the URL in flake.lock is used.

This way specifying:

inputs.nixpkgs.url = "https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz";

Would actually just work, as instead of failing with an invalid hash whenever nixos-unstable updates, it saves the redirected URL which points to a stable version. flake.nix would look something like this:

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "narHash": "sha256-N1qI50AkeTSBp1ffUCHrcK2re52wrn6euFFGGvFa2iw=",
        "type": "tarball",
        "url": "https://releases.nixos.org/nixos/unstable/nixos-21.05pre269929.ff96a0fa563/nixexprs.tar.xz"
      },
      "original": {
        "type": "tarball",
        "url": "https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz"
      }
    },
    // ...
}

This way, older versions of Nix seeing this new lock file would just behave as if someone used --override-input, and newer versions of Nix seeing the old lock file with the pre-redirection URL would simply migrate it over when nix flake update --update-input is used.

A major concern would be whether this redirection is actually part of the intended interface of channels.nixos.org. That’s mainly why this is isn’t a feature request: It isn’t clear whether this even is a feature. I hope to generate some discussion here first.

In any case, I hope that channels.nixos.org doesn’t go away, if not just for a slightly independent place where nix expressions can be found.

4 Likes
2

On an unrelated note, it seems that channels.nixos.org redirects to the fixed version releases.nixos.org URL with an HTTP 301 instead of a 302. Is this right? There’s no Cache-Control header in the second response…

Trace is abridged from curl -IL https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz:

HTTP/2 302
cache-control: public, max-age=0, must-revalidate
content-length: 73
location: https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz

HTTP/2 301 
content-length: 0
location: https://releases.nixos.org/nixos/unstable/nixos-21.05pre269929.ff96a0fa563/nixexprs.tar.xz

HTTP/2 200 
content-type: application/octet-stream
content-length: 16904176
3

In case anyone is interested, here’s a proof of concept patch that implements this redirection behavior. No promises, no warranty, and in particular I have not tested throughly what would happen if you upgrade from good old nixUnstable to this.

4

I’m guessing the registry is supposed to replace channels, instead of github:NixOS/nixpkgs you can just use nixpkgs which currently points to the same github repo but can be pointed at any other repo without changing flake.nix (flake.lock would need to be updated though).

5

Oh that seems reasonable.

6

yes but channels are actually a subset of the git commits: only built and tested commits. So while I agree that flakes would be nicer than channels, actually the flakes would need to point the channel I think?

So maybe the nixpkgs flake should be pointing to https://channels.nixos.org/nixos-21.11/nixexprs.tar.xz? Does that work?

7

I’m using flake-based NixOS and Home Manager configurations, and I set the nixpkgs-like inputs from nixos-* branches to ensure that they are built and tested on Hydra and that the binary substitutions are available.

This way, I’m literally using the channels implicitly. Is that right?

8

Maybe for 301 redirects, but not 302.

1 Like
9

This was my thought too. A 301 is a permanent redirect and so is suitable for recording in the lockfile. A 302 is temporary, and should be treated as temporary. For example, it could be a CDN pointing you at the local edge node and so you want a different node if you’re fetching it on a different network. Or perhaps the edge node it’s pointing at today isn’t even around tomorrow and it needs to resolve to a different URL tomorrow even if you haven’t changed networks.

The downside of course is that https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz is in fact a 302 redirect, as it’s redirecting you temporarily to the current channel head, and will redirect to a different channel head later. So recording 301 redirects doesn’t solve the original issue.

I think the solution to the original issue is to have a flake input option that tells nix to lock the result of a single redirection (modulo 301 redirects) instead of the original URL in the lockfile. This way you can point at a dynamic URL that you expect to redirect to a permalink and explicitly ask Nix to store the permalink instead.

The detailed behavior here should be “follow all 301 redirects. Upon hitting a single 302 redirect, if the option is specified, follow that and resume following 301 redirects. A subsequent 302 redirect is not followed”.

10

If I understand how the channel process works, they do already. Hydra builds the channel, once the configured jobset passes the commit that was built is then pushed to the appropriate branch as well as published as a channel tarball. By that I mean that e.g. the nixos-21.11 branch is updated at the same time as the nixos-21.11 channel.

My guess is the source for each channel is the equivalently-named release-* branch, e.g. release-21.11 is presumably what feeds into the nixos-21.11* channels. So if you were to point at flake at the release branch you’d get the release channel equivalent of pointing at master (e.g. if you don’t want to wait for hydra).

Of course, given all that, it makes me wonder why the default flake registry doesn’t specify a branch for nixpkgs, thus having it default to master. I would have expected it to default to nixpkgs-unstable so users who don’t customize their flake registry will get the hydra-built packages by default. I think I’ll go file a bug on that. (Edit: Filed as `nixpkgs` entry should default to `nixpkgs-unstable` ref · Issue #16 · NixOS/flake-registry · GitHub, though given the PRs sitting open I’m not sure anyone’s paying much attention to this repo).

4 Likes
11

I thought this was strange as well. This inspired me to write:

1 Like
12

I’ve change the flake registry entry for nixpkgs to use nixpkgs-unstable by default instead of master. (nixpkgs: Use nixpkgs-unstable instead of master by edolstra · Pull Request #17 · NixOS/flake-registry · GitHub) This ensures that users get a revision of nixpkgs that has been built and tested by Hydra.

8 Likes
13

This topic came up again in the discussion around my PR to add an explicit nixos-x86_64-linux channel.

From my point of view, channels are also useful in a flake world, as they represent a tested snapshot of a given nixpkgs branch.

My original motivation for requesting nixos-x86_64-linux was, that in preparation of RFC 87 nixos-aarch64-linux had already been created. But the additional load on Hydra is something that we currently can’t manage.

My takeaway on this is that we probably should reorganize how channel builds are triggered (probably through an RFC). As far as I understand the current process, channel builds are triggered (independent of each other) by time and start from the head of the relevant branch in nixpkgs at the time the trigger occurs.

A simple but hopefully worthwhile change might be to use the -small channel as a smoke test. Then nixos-aarch64-linux and nixos-x86_64-linux could both follow nixos-small, so we would possible spare a lot of builds that never end up in a channel bump, freeing up resources for possibly faster iteration of -small or staging-.
In turn the nixos-YY.MM builds would only be triggered on revisions that succeeded for both, nixos-aarch64-linux and nixos-x86_64-linux. Given that on a successful evaluation the matching branches are advanced in a common repo, it is easy to select the head of the lagging branch.
Given that this setup can make good use of caching, I do not expect a significant increase in end-to-end latency from commit to the channel advancing.

2 Likes