Continuation: Discussion about python infrastructure

This is a continuation of the python thread started in Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub. Quick summary:

@FRidh (Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub)

On a sidenote, considering you’re using Python here a lot, and in various ways (packages, envs, …) I am interested in knowing how you experienced the tooling. Any issues? Thinks you like to see different? Perhaps better to discuss is in another thread though.

@timokau (Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub)

Some things took a while to figure out (especially the problem that makes the empty env necessary as described above). Overall I’m pretty happy though and much prefer the “hand-curated” approach to the completely auto-generated package sets.

In the beginning I used to just set PYTHONPATH manually, but the performance of that is atrocious (adding around a full second to startup time). So I’m glad that python envs exist.

Minor nitpicks/questsions:

  • Its a bit irritating that some conventions are a tiny bit different than the rest of nixpkgs. Close enough to confuse, but not the same. pname, just using python.pkgs instead of passing everything explicitly as arguments and overridePythonAttrs come to mind. I kind of prefer the way things are done in python, but it does hurt consistency a bit.

  • I don’t quite understand why you do batch updates. Is there any practical reason for it? Why not handle python package updates like updates for any other package? For example the recent batch broke a few things for me and it is a bit hard to find the cause when everything is batched. Of course nobody can reasonably expect you to test every single update. So why not let the regular update bot create the updates and rely on the community to test and merge them?

  • I don’t understand why pypi (with fetchFromPypi) seems to be the preferred source for packages. Wouldn’t it be preferrable to pull things directly from source control if available? That cuts out the middle-man and makes it easier to quickly test with an unreleased version.

All in all I’m very happy with it, thank you for maintaining :slight_smile:

@7c6f434c (Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub)

@timokau re: batch updates: when upstream of the language ecosystem has a package manager, it makes sense to track it via batch updates because the outcome is easier to reason about when comparing to the upstream situation.

@timokau (Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub)

Why is that? Python packages can depend on specific version (ranges) and if a update builds in nixpkgs and doesn’t break any reverse-dependencies, I see no reason not to include it right then.

@dotlambda (Sage on nixos by timokau · Pull Request #39981 · NixOS/nixpkgs · GitHub)

It’s simply that just a small percentage of the python packages would be updated at all if we didn’t have batch upgrades. And since the batch upgrades only happen on the unstable channel, people encountering broken builds should expect that this happens from time to time. Before a new release, we try to stabilize the python package set.
Btw, @FRidh proposed https://groups.google.com/forum/#!topic/nix-devel/koLvsCVjQcA, which would allow maintainers to opt out of batch upgrades.

It’s simply that just a small percentage of the python packages would be updated at all if we didn’t have batch upgrades.

How is that different from all the other packages that are handled by the new(ish) update bot? That allows individual review of updates.

And since the batch upgrades only happen on the unstable channel, people encountering broken builds should expect that this happens from time to time.

I don’t fully agree with this. While build breakage on unstable might happen from time to time, it should still be avoided if possible. Probably that’s what you mean.

Btw, @FRidh proposed https://groups.google.com/forum/#!topic/nix-devel/koLvsCVjQcA, which would allow maintainers to opt out of batch upgrades.

For me that thread sounds like it is about @ryantm’s bot, not the batch updates.

I am not familiar with Python packaging but I know about Haskell packaging so I will try to be helpful here sharing the similarities.

This also happens in Haskell packaging. You could also have a derivation for every version of the package which has existed but this creates a very large number of derivations. The compromise is to keep the most up to date one. I think the best solution is to be specific about which version of nixpkgs you are using for each project if you don’t want things to break when you update nixpkgs. It should also be possible to selectively pin just the Python
package definitions using a suitable overlay but that could be quite fragile in general.

There is no way that someone could review every update to a Haskell package, there would just be too many difficult to analyse changes. The package set is automatically updated and the overall status is tracked to catch any major breakages.

Python packages are specified as an attribute set which is passed to the mkPythonDerivation function which is not the same as the mkDerivation function. They are quite different so it should be expected that they want different arguments. As another comparison, the Haskell specific mkDerivation function is in fact called… mkDerivation. It has been suggested to rename this to avoid confusion with the normal one as it accepts completely different arguments. This is the crux of why the overriding mechanism is also different.

Perhaps I am ignorant about how the Python ecosystem works but if the canonical place tools like pip get packages from is pypi then surely nix should as well. Are you not free to specify a different src which points to a github repo if you like? There are some examples in the manual where this happens.

Anyway, my answers read like someone who has been institutionalised. Hopefully I don’t get in the way of making the experience better than I can see possible!

I think you’re misunderstanding what I meant by batch updates. Its not about different versions of the same package but just about the process of updating (semi-automatically?) lots of different packages in one PR.

While it wouldn’t be perfect, one PR per update would at least give a better option to review and/or test with ofBorg.

I understand that it uses a different mechanism, that doesn’t mean it has to have a different interface though. Like I said I actually prefer the python choices and I’m not really criticizing, just responding to a request for feedback. Consitency is important.

Why? Fetching directly from source control has several benefits:

  • cuts out middle man that could potentially have modified the package (like sourceforge did)
  • speeds up time to distribution (I think pypi has some lag from the time the version was tagged)
  • makes it easier to test master versions (by simply replacing rev = "${version}" with a commit hash)

And I don’t see any disadvantages.

Not at all, thanks for your input!

Well, increasing the amount of review is expending the bottleneck resource of Nixpkgs — reviewers who are not currently burnt out on doing reviews.

True but I don’t think thats a valid reason not to review an arbitrary selection of packages. Most reviews of these updates should consist of checking the ofBorg output and glancing over a 2-line diff. The few reviews that get interesting would benefit from that approach.

True but I don’t think thats a valid reason not to review an arbitrary selection of packages.

Manual review of obviously-straightforward bumps of packages that are not obviously highly important should be assumed to be done 1) in a hurry b) in a manner that a script would do just as well.

Most reviews of these updates should consist of checking the ofBorg output and glancing over a 2-line diff.

Looking at a lot of two line diffs still takes a lot of time (and for zero benefit, if the update script cannot change anything but versions and hashes).

Some packages are better upgraded together anyway.

Also, a toposorted batch upgrade will often pass ofborg builds even if it breaks things: it is non-updated reverse-dependencies that break most, after all.

Many reverse-dependency problems indeed get missed in review; review should produce noticeably better results than scripts to be worth the effort.

The few reviews that get interesting would benefit from that approach.

How would anyone even separate them? If you have an automatable criterion for separating the few reviews with much higher chance of being interesting, please describe it so it can become an exception to batch-processing scripts.

But that would be the same for every one of r-ryantm’s PRs.

That’s a good point. I don’t know how often this happens in practice.

But at least individual PRs would make it easier to improve the process in the future. For example with a super-powered ofBorg that checks reverse dependencies.

If all the updates are really just version + hash, thats true. You could maybe infer something from the estimated number of rebuilds.

I don’t know if that is really the case and how much of @FRidth’s update process is automated.

But that would be the same for every one of r-ryantm’s PRs.

Yes. Reviewing these PR’s usually boils down to the question whether someone guesses a reverse-dependency to test; and also whether a macOS or aarch64 build failures (if any) are acceptable.

But at least individual PRs would make it easier to improve the process in the future. For example with a super-powered ofBorg that checks reverse dependencies.

That’s much more build power than Hydra! I mean, I think if we ever are even close to that, we can try adjusting the workflow to this newfound power.

If all the updates are really just version + hash, thats true. You could maybe infer something from the estimated number of rebuilds.

I don’t know if that is really the case and how much of @FRidth’s update process is automated.

As far as I understand the batch part is just pulling from pypi, there may be additional changes performed by @FRidth, but these are debilerate and done with conscious care.

So why are python updates different?

Yeah definitely not anytime soon. But with the sponsoring of the right company that uses nix, maybe sometime. Until then maybe some selective reverse-dependency testing. I think it was alreay talked about to keep a build-time log of packages and then only test a quick sample of the reverse-dependencies.

I imagine thats hard to do considering not all python packages have a standard format. Isn’t that duplication of effort with r-ryantm?

So why are python updates different?

Because here is already an upstream that does approximately the same level of looking at integration?

I imagine thats hard to do considering not all python packages have a standard format. Isn’t that duplication of effort with r-ryantm?

I think it is older than r-ryantm (just like multiple existing scripts), and uses a more direct source than repology.

For updates it is most critical that pypi has a standard version information format, I think.

First of all, let me elaborate on the process for batch upgrades:

The commits are made using https://github.com/NixOS/nixpkgs/blob/master/maintainers/scripts/update-python-libraries and are pushed to the python-unstable branch, for which we have a Hydra job. This job builds all Python packages and Python applications. Usually, the Python maintainers have a look at the new build failures and try to fix the major ones at least. After sitting around for some days, the python-unstable branch is merged into staging, which in turn is merged into master after some time.
This means that batch upgrades do indeed break some packages but the we try to reduce the breakage to a minimum.

r-ryantm

It’s not as extreme as with the Haskell packages but I think there are quite a lot of Python packages in Nixpkgs that are not packaged by other distros. This means they wouldn’t be updated by r-ryantm. I don’t have any numbers but if I find the time I might have a look at some Repology statistics to calculate the percentage of Nixpkgs-only python packages.
Of course, this problem could be overcome by teaching r-ryantm to use PyPI as a source for possible updates. However, there are still

Advantages of batch upgrades

  1. The sheer number of PRs would be too much for the current number of Nixpkgs maintainers.
  2. Most PRs would probably not be meaningful to just about any Nixpkgs maintainer: Most Python packages are rather unknown because they are just obscure libraries needed for some application.
  3. Human reviewers don’t have the computing power needed for reviewing mass-rebuild upgrades. Such upgrades are better handled by letting Hydra build all Python packages and having some human(s) look at the newly introduced build failures. If we’re doing the latter, why not just save time by doing it for all packages at once?
  4. Some packages need to be upgraded simultaneously, because e.g. versions need to match. Also, many times upgrades introduce build failures of depending packages. If we have larger intervals between batch upgrades, upstream maintainers are given a chance to discover these in their CI infrastructure or because other distros come across them. This way, the chance of even introducing new build failures is reduced.
  5. The current policy of Python batch upgrades is to update everything regardless of whether the new version builds or not (as mentioned above, most errors will be fixed manually). Having security and a general desire of being up-to-date in mind, this seems quite appealing to me: I prefer breaking some builds and having someone fix them at a later point instead of a conservative upgrade policy minimizing breakage but also leaving many packages outdated.
1 Like

Is there? I thought pypi basically just mirrors the source.

Semi-related: I researched what the status of repology getting its information directly from pypi is. For anyone interesetd: Fix PyPi support · Issue #278 · repology/repology-updater · GitHub

Wow, that script is a lot less complicated than I would’ve thought. Thanks for that insight! Is the process documented anywhere (well I guess it is here now :D)?

Repology issue 278 is again relevant here (but that doesn’t help in the moment of course).

Do you know how other distors solve this? Do they have that much more manpower?
I’m still not really convinced that individual PRs are very much overhead over one big PR in terms of review. That is assuming that the big PR gets reviewed.

Good points, I didn’t consider that.

This is a bit problematic because the batch updates hide that problem. Ideally such updates should be in a single commit, keeping both packages buildable at every revision.

But that nicity is probably not worth the effort here.

Again, good point.

I don’t 100% agree with that. I think if breaking updates are applied, the packages should at least be marked as broken.

I’m not totally convinced that batch updates are really better here, but I see that there are various advantages and you and @FRidth are definitely in a better position to weigh those against the disadvantages.

My main painpoint with the python infrastructure is the difficulty to override package (in the overlay). I would like to be able to just do pkg.override Nixpkgs 23.11 manual | Nix & NixOS

@dotlambda described very well why I started off with the batch upgrades. I would prefer not to break anything, but unfortunately it takes way too much effort for @dotlambda and me to fix everything after a batch upgrade. Therefore, I indeed proposed on the mailing list a meta attribute that maintainers can use to indicate to not batch-upgrade their package(s).

I asked @ryantm not to use their bot with pythonPackages, and that is because of how tightly Python packages depend on one another and the amount of breakage that occurs between typically major updates. While the bot checks whether the package itself builds, it doesn’t check all its dependents. This is problematic in the case of the libraries. For the applications, on the other hand, it should be fine to use that bot I think.

1 Like

Alright, thanks for the clarification. I simply don’t have the experience with the python ecosystem so I didn’t know the libraries where that much more inter-linked than usual.