Upcoming Python breakage

In hopefully a week or so two large changes will land in master:

  1. Python 3.8 will replace 3.7 as the new default Python 3 interpreter and is currently in staging.
  2. Major package set update which will hopefully land in staging in a couple of days.

Changing default interpreter means some applications will stop functioning and this may mean passing python = python37; will be needed, although preferably the application is made compatible with 3.8.

A large update to the package set has not happened in a long time, which means by now it is quite outdated. It’s important the core set of packages is fairly up to date, especially prior to a NixOS release. After this update I expect a somewhat smaller update again prior to the 20.09 release.

Now, given how tightly the packages are connected it’s typically a lot of work to get everything fixed up. My focus is on the core packages so this means there will be breakage in leaf packages. Also important to consider is that many packages have dropped Python 2 support. In those cases, we can provide an older version for Python 2, but of course someone will need to step up to make that happen. As part of this change I intend to provide Python 2 versions of core packages, but not more than that.

If you like to help out, please have a look at the PR. There should not be any more force pushes to that branch for at least the coming days.

14 Likes

One thing I forgot to mention is that, while by now the far majority of packages are compatible with 3.8, not all are. I think most importantly aiohttp is still stuck at 3.7.

2 Likes

To get even more breakage, this change could also include the long overdue switch to

python = python3;
pypy = pypy3;

in all-packages.nix.

And we should consider dropping python35 for 20.09 because it will be EOL just at the time of release.
As of now there are only two packages left that depend on python35:
pkgs/tools/graphics/luxcorerender/default.nix (fixed in luxcorerender: dependency: python35 -> python36 by wamserma · Pull Request #89755 · NixOS/nixpkgs · GitHub)
pkgs/tools/graphics/yafaray-core/default.nix (fixed in yafaray-core: relax python dependency by wamserma · Pull Request #89738 · NixOS/nixpkgs · GitHub)

Once these are fixed, python35 can simply be dropped.

3 Likes

The first won’t happen, but dropping Python 3.5 will happen, but indeed not now but just before release.

2 Likes

Another package that breaks is gst-python: `python38.pkgs.gst-python` does not link. · Issue #82974 · NixOS/nixpkgs · GitHub

That might potentially block GNOME.

Unfortunately, I cannot debug it at the moment.

1 Like

I put up a deprecation announcement for 3.5: python35: add EOL announcement by wamserma · Pull Request #89761 · NixOS/nixpkgs · GitHub

Those are both my fault. The Yafaray one was merged and I just approved the LuxCore one.

Tensorflow doesn’t seem to support 3.8: https://github.com/NixOS/nixpkgs/blob/fb526fd46914977b611a2e761e9c70141eb42bd8/pkgs/development/python-modules/tensorflow/2/default.nix#L350

It does upstream, tensorflow · PyPI , just needs to be updated from when it didn’t support it

1 Like

To what should that link refer? It doesn’t seem to work.

That would be commit b7ff7465401257e9b0814bb68937a494c58de538 which is currently in staging-next Staging next by vcunat · Pull Request #90058 · NixOS/nixpkgs · GitHub
Note the package set updates are not part of it.

Deep link for convenience of other readers:

commit in context of PR:
https://github.com/NixOS/nixpkgs/pull/90058/commits/b7ff7465401257e9b0814bb68937a494c58de538

commit:
https://github.com/NixOS/nixpkgs/commit/b7ff7465401257e9b0814bb68937a494c58de538

python38 is now the default python3 interpreter on master. Note there will be regressions, certain applications will fail to build with Python 3.8. They either need to be made compatible with 3.8 (preferred solution) or use 3.7 instead.

Note some are fixed on current staging-next. Thus it is best to wait with fixing these applications on master or opening PR’s towards staging-next.

6 Likes

Are there any plans on deprecating/phasing out Python 2? Python 2 is now unsupported and has an slowly increasing backlog of security issues which will possibly not receive upstream fixes.

See also Vulnerability roundup 84: python-2.7.18: 4 advisories · Issue #88384 · NixOS/nixpkgs · GitHub

2 Likes

I think originally it was going to be a soft deprecation, as many applications (such as sage or calibre) still used python2 at the time. But if there’s active CVE’s that are out, then I think that warrants at least marking the package as insecure, as mentioned in the thread. However, I’ll defer to @FRidh on any actionable items.

What would deprecating / phasing out Python 2 mean? In principle we try to avoid Python 2. Indeed, most of Nixpkgs can be built without, leaving only relatively rarely used packages that depend on Python 2.

The plan is to keep the Python 2 interpreter around, and at least a set of core packages. Exactly how many or which entirely depends on the community; if they need packages they need to maintain them. The Python 2 package set is also increasingly diverging from the Python 3 set in terms of versions.

Will there be security issues with them? Yes, likely, but that’s up to the users to accept or not.

Note I won’t bother with patching Python 2 or any of its packages, that will be up to those that want to use it.

3 Likes

at the very least, we could mark python2 as insecure. At least then, the user would have to opt-in to using a known insecure package.

  permittedInsecurePackages = [
    "python-2.7.18"
  ];
4 Likes

The package set updates have now landed in master. If you rely on Python 2 applications then likely you will notice some regressions.

1 Like

A new set of major upgrades was merged into staging and is now in staging-next.

This now included Python 3.9 packages.

4 Likes