Questions about `inputs.nixpkgs`

Help
21

Whoa that’s cool, nvm me, switching all my flake inputs over as we speak.

23

Yeah! We contributed that to Nix so that FlakeHub could implement semver solving, but still lock reliably :tada:

5 Likes
24

Could this be elaborated, and do other Nix proficient people agree?

What could be the potential tradeoff of doing that?

I think remembering over the vast majority of the nix flakes configuration I have seen, nixpkgs inputs were almost always using a version of a github:NixOS/nixpkgs URLs (and some using nixpkgs FlakeHub URLs)

I don’t think I have seen it recommended in the wiki either (but probably should if that is correct)

It seems like an interesting and bold claim

Also, about the benefits you are talking about:

Are those benefits not possible to be worked on and added for git urls?

I am thinking of a fallback to trusted artifacts sources in case of a rate-limit / censor

1 Like
25

The way I understand nix’ CI infrastructure (from reading these docs apparently) and branching strategy things go roughly as follows:

  1. A commit is pushed to the respective dev branch (master or release-YY.MM).
  2. Hydra eventually picks up the commit and starts building packages and tests.
  3. Hydra completes
  4. Upon completion, hydra:
    • Produces the release tarball
      • This contains the relevant command-not-found cache (and man cache?)
    • Hydra fast-forwards the respective non-dev branch (nix(os|pkgs)-unstable or nixos-YY.MM)
      • There are sub-points to the CI completion that trigger the *-small and *-darwin releases

As such, the release tarballs are more complete, aren’t subject to the same rate limits and censorship as GitHub (which have caused many folks headaches in the past), and mean actually using the nixpkgs infrastructure as intended. You can build from the branches, but those are downstream from the release tarballs and you get less complete artifacts.

Older versions of nix won’t be able to resolve the input correctly, though they should interpret the lockfile correctly I think? Other than that, none, you’re just using nixpkgs as intended, before the limitations of flakes resulted in people using the git repo directly instead of the release tarballs. I.e., you’re getting the same stuff nix-channel used to download.

CDN details change, too, I suppose, but the tarball is ~40MB, while the shallow git repo is ~80MB compressed, so you’d have to have a pretty bad connection to NixOS infra to have overall slower downloads than from GitHub.

This feature only came into existence a few months ago. You’re looking at half a decade’s worth of ecosystem. Of course not all of the above have updated, nor has your memory retroactively inserted a feature that didn’t exist. The fact that this is even possible hasn’t been remotely advertised beyond that PR either, so the vast majority of the community will be as unaware as I was.

It’s not exactly intuitive behavior, normally a url doesn’t magically resolve to a different location at flake resolution time, so I don’t think anyone would be looking for this either unless they scanned the nix release notes and understood what the feature name meant at a glance. I imagine this passed by most of us.

Not really, since the command-not-found (and man?) cache fundamentally has to be produced during the build, so it cannot reasonably be committed, and the cleanup steps involved in reducing the tarball size most likely involve removing dev-only cruft from the repo.

Using git for release versioning is anyway a bit of a hack, this is much nicer IMO.

Overall, really awesome work from detsys, thank you @grahamc & co for upstreaming this. It’s technically a small fix, but if we can get over the collective inertia I think this has significant impact. Even just the emission savings from cutting all nixpkgs update downloads in half matters.

10 Likes
26

Just to clarify: the tarball from Github (which is what a github: flake input would use) is also 43 or so MB.

1 Like
27

That just makes it sound like you upstreamed a feature to enable your paid product instead of upstreaming the features themselves instead

28

Nope! It’s not clear how version resolution could possibly happen in the Nix CLI itself with respect to a platform like GitHub, which doesn’t expose the proper hooks for such resolution.

What we did is we upstreamed a change to Nix that would make not just FlakeHub with SemVer possible but also an indefinite number of platforms and URL schemes. Set up an HTTP server that provides responses that the Nix CLI understands and boom, you’ve got yourself a flake resolution scheme of whatever complexity and serving whatever use cases you like. We liked how Rust and Cargo do things so we brought it to Nix, and people can use it for free (for public flakes). Nothing is stopping anyone from doing an entirely different thing.

4 Likes
29

This is starting to become out of scope of the original topic

30

Thanks for the explanation!

Then, it would certainly be nice to document that in Flakes - NixOS Wiki and Nixpkgs - NixOS Wiki

Unfortunately if I make that change myself I will surely document some wrong and stupid things :smile:

Also, does FlakeHub also ships tarball and thus has those benefits or are those only true from the upstream hydra tarballs?

31

FlakeHub serves statically generated tarballs. They’re created once, and then archived in read-only storage forever. That means that changes to tar creation or compression will never cause the hashes to change.

32

@TLATER would you suggest we update the official wiki on this topic?

I’ve only seen documentation pointing nixpkgs.url to GitHub, but arguments behind using:

  • https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz
  • instead of github:nixos/nixpkgs?ref=nixos-unstable

consists of a number of good arguments, like avoiding temporal cache misses from hydra.

1 Like
33

Go for it, there will be a lot of documentation out there that probably wants updating with this feature in place.

1 Like
34

I think work should be done directly in Nix to make it so inputing github:NixOS/nixpkgs* would redirect to hydra (or at least by default)

Do you know if there is an open issue about that?

I guess it shouldn’t require a big PR either

35

I disagree, as that breaks user expectations. Nix shouldn’t translate our inputs from one thing to another arbitrarily, that should be a conscious decision by the user.

5 Likes
36

What are there to expect about nixpkgs input which could make this enhancement a deal breaker?

The current github repository input is technically also breaking expectations because its redirecting to a tarball, which is a direct enhancement and also if I’m not mistaken was not present from day one and required “breaking user expectations” to implement

Given @TLATER’s downsides analysis, I’m quite optimistic

Alternatively, do you think an hydra: / nixpkgs: / nixos: / channel: input type or something similar would be preferable?

37

This was discussed here: Add a `channel:` flakeref · Issue #13168 · NixOS/nix · GitHub

38

FWIW, iirc determinate nix actually does support the channel: syntax. (edit: nevermind, that was for non-flakes only, and regular nix supports it too.) And the extracted tarballs are flakes, so it could be supported via flakes.

39

I think this is preferable. You might want to explicitly use the GitHub repo for one reason or another, and this magic would get in the way.

Plus, special handling just for nixpkgs is weird.

3 Likes
40

That’s reasonable indeed

However, that idea seems to have already been closed as not planned

41

I think a small paragraph to explain the tradeoff and benefits would be good

Have you updated the official wiki yet?