Why do I need to compile Python (and packgages) every time there is an update?

Help
1

Are there no precompiled packages on cache.nixos.org for Python?
Does precompilation not work for python, because every user uses different Python package selections?
Or am I doing something wrong in my configuration.nix?

{ config, pkgs, ... }:
let 
my-python-packages = python-packages: with python-packages; [
	# data sciency stuff
	scipy numpy matplotlib pandas ipython
	# other
	distro pygobject3 pyppeteer requests termcolor flake8
]; 
in
{
	environment.systemPackages = with pkgs; [
		(python3Full.withPackages my-python-packages)
	];
}

Like the compilation of Python always takes at least an hour, compared to for example Hyprland (a wayland compositor) is only taking approx. a maximum of 15 min.

Or is it not every time, but it happens so regular for me, that I think python updates this infrequent?

Edit: Oh yes, sorry, I’m on unstable.

2

You shouldn’t have to compile Python or Hyprland. These are both cached by hydra. Generally when the cache fails like that, it either means you’re following the wrong nixpkgs branch (e.g. master, which is often heavily uncached), or you’ve added an overlay to your setup that is low enough level to cause a ton of things to be rebuilt from scratch.

2 Likes
3

Hyprland is expected because I compile Hyprland from master with their nixos module.

And I think I’m on the right channel:

~ $ sudo nix-channel --list
home-manager https://github.com/nix-community/home-manager/archive/master.tar.gz
nixos https://nixos.org/channels/nixos-unstable
4

Do you have any overlays? Either in your NixOS config under nixpkgs.overlays = ..., or in ~/.config/nixpkgs/ (I believe under root’s home dir)?

1 Like
5

i have overlays for some stuff, but none of that involves python

only some bash scripts and stuff like rofi, waybar, polybar and a custom gtk theme

6

And you’re sure your overlays aren’t changing anything that might be a buildInput of python (or a buildInput of one of its buildInputs, etc.)?

7

nope, i just double-checked

8

unless python depends on polybar, rofi or waybar or asciiquarium

9

Are you sure it’s actually Python being compiled? Although Python itself is cached, I don’t think many Python packages are cached. I just tried building your python3Full.withPackages expression and uh… there’s a lot of python packages to build, and a bunch of them are compiling code and running long tests. I think that’s likely what is consuming so much time.

10

Maybe it is not python itself true, but shouldn’t Scipy, Numpy, Matplotlib and these heavy data science packages be cached?

11

is there a way to see which packages are cached?

12

It took my machine 25m18s to build that expression, and I have a 16 core / 32 thread CPU so the compilation parts were compiled with a lot of parallelism (when it was done I had a 15min loadavg of 20).

I think it might be because you’re using python3Full? I just tried the same expression (with a clean store) with python3 instead and it was all cached. Changing back to python3Full, tons of stuff like numpy and scipy were not cached. I dunno why python3Full packages wouldn’t be cached.

1 Like
13

Thats interesting I’ll try that thanks
i need to go to sleep now its way to late

14

and yeah i only have a 4 core intel i5 7th gen laptop so no wonder it takes long for my computer if it takes 25 minutes on your beast

15

Only a single python package set is prebuilt, which is python3Packages. The *Full based sets are generally not prebuilt. unless something within nixpkgs depends on it, and then only that much gets built.

I’m not even sure what pythonFull gives you, what python alone doesn’t, except for longer waiting times :smiley:

4 Likes
16

We currently recurse into python310.pkgs and python39.pkgs, so those package sets are getting built and cached.

  python27Packages = python27.pkgs;
  python37Packages = python37.pkgs;
  python38Packages = python38.pkgs;
  python39Packages = recurseIntoAttrs python39.pkgs;
  python310Packages = recurseIntoAttrs python310.pkgs;
  python311Packages = python311.pkgs;

The python3Full package is a variant of python3 that also links against bluez for AF_BLUETOOTH socket support and X11 libraries.

  python3Full = python3.override {
    self = python3Full;
    pythonAttr = "python3Full";
    bluezSupport = true;
    x11Support = true;
  };

I’m not entirely sure why we don’t provide them by default, initially it was only for X11 support, but here we are.

@FRidh will likely know best.

1 Like
17

Most users/libraries do not need these additional libraries, which are fairly heavy as well if I recall correctly. So it’s a trade-off.

Back in the days there was also an expression for a more modular CPython but it was difficult to maintain and thus dropped. There were also issues with including those separately built modules again.

1 Like