Python3 user package being "masked" by systemd-boot

Help
1

Hello.

For some time now I noticed that I was having trouble configuring my Python3 package via home-manager. I was able to “resolve” the issue temporarily but now I’ve looked into fixing the underlying problem and I’m stuck.

The situation is that I am interested in adding modules to python3 (pandas and numpy, for example). To do so, and as I could find in the Wiki, I added the following to my home.packages:

    (python3.withPackages (pkgs: with pkgs; [
        pandas
        numpy
    ]))

After making the switch, however, the packages are not available when I interactively run python3 in the terminal. It was strange because adding an equivalent expression to my configuration.nix to apply the switch globally, it worked perfectly when using sudo (I only did it to test. I wanted to know if the expression worked fine).

Just yesterday I managed to find this related post " Python3 not importing modules", where the creator has a similar problem to mine and it seems that in his case terminator was interfering in which python3 to use. However in my case it is not terminator but systemd-boot who is masking python3.

Taking advantage of the discussion in the post and the commands with which they helped him diagnose the case, I got the following:

  1. nix-env is NOT masking my Python3:
❯ nix-env --query

home-manager-path
  1. Python3 and numpy are not on the same path. (It was striking to me that the package that produces the conflict in the case of the other user, and me, is exactly the same when comparing hashes, but I don’t know if it’s relevant)
❯ readlink -f $(which python3)
/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6/bin/python3.10
❯ readlink -f $(which f2py)
/nix/store/ghqqfq4f9n3fjb4l9cdd1pqhm4j91swd-python3-3.10.6-env/bin/f2py
  1. Python3 is being a dependency of systemd-boot (…if I understand correctly).
❯ nix why-depends /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6

/nix/store/mjb1w8qri7giahh42hfh4iqrn5xd7xf5-nixos-system-wiseh-22.11.20220922.ae1dc13
└───/nix/store/7nc054hg8d6srbp6b3q36gs3msawaq45-systemd-boot
    └───/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6

After discovering that I thought about, for example, changing systemd-boot to grub and see if the problem was solved. Honestly it was an odyssey trying to configure grub to work with UEFI as it always ended up in a shell and did not run my graphical environment so I could not test my hypothesis; but, besides, I would rather understand and correct the problem with systemd-boot as I want to continue using it (and it seems better than running away from the mishap).

I really appreciate if someone can help me with what my next step should be, and if more information is needed I’ll stay tuned.

2

Try using nix why-depends --all /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6. Multiple things may depend on the same binary, by default why-depends just shows you the first package it finds.

1 Like
3

I’d do a more careful analysis of PATH as well, there are several ways your python could be shadowed. Just as an example, you could have switched home-manager (as a nixos module) to use home-manager.useUserPackages = true;, and not removed the now-defunct home-manager-path from your nix-env profile, which would take precedence over the new location in /etc/profiles/per-user.

3 Likes
4

Well, first of all really thank you both for your time. Now,

about this… That’s right, I didn’t know about the --all option. But my output is over 350 lines so I don’t know if it’s appropriate to share it here. However some packages that also have it as a dependency are: system-path, which in turn uses it as a dependency in flatpak, ranger, fish, git, qtile, kitty, blueman, and others; and, separately, etc, which uses it in unit-display-manager and desktops. It has a total of 56 appearances in the command output, so yes, it appears in many other parts. I suspect that’s bad news?

And as for this, I forgot to mention that tidbit, but I don’t use home-manager as a module. I know it was an example but… I don’t recall ever messing with my PATH in my configurations. Any other ideas on things that can alter the PATH frequently, or where I can search on the subject?

In case it would be useful, this is what my PATH currently looks like

/nix/store/r54z05l2yw8x9iphn2qy1vwaai5ribsp-kitty-0.25.2/bin
/nix/store/44cav0p2yyyfq2d99bb7yi7bm67fsqj1-imagemagick-7.1.0-48/bin
/nix/store/axr1pjba64p3afs2frkhb1ym2p20pid1-ncurses-6.3-p20220507-dev/bin
/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6/bin
/nix/store/apq69l8z45z4k3qvcd3b4snrqn1pfr1w-python3.10-qtile-0.21.0/bin
/nix/store/mj26zd9y4spqlpdn9c6axmi6xbzxddv8-cairo-1.16.0-dev/bin
/nix/store/pzv32b0wylnaq6f05ssdycwx5j259pw1-freetype-2.12.1-dev/bin
/nix/store/shkayl138d07l29bph4f92x3hwbqlk0j-bzip2-1.0.8-bin/bin
/nix/store/j8wsgcjsk523cdj1hd8xjcawqdibhgp4-libpng-apng-1.6.37-dev/bin
/nix/store/128n2b0i0nfk9nl615scgsaly4g318z6-fontconfig-2.14.0-bin/bin
/nix/store/jgvxzgnkp17ljhw2h9g42ccmx3a62w7w-expat-2.4.8-dev/bin
/nix/store/rwr0ly4girpnrq3sqvp3v2k40wb4hh3s-glib-2.72.3-dev/bin
/nix/store/xh696lkivvygg8bkpvj95vv3qspp2awx-gettext-0.21/bin
/nix/store/xa57q9w1wsvmk61nq8krd0fc9jg8j6sc-glib-2.72.3-bin/bin
/nix/store/sn9lcjv4plxmnhsw0a8cv7jzbr3w51ps-python3.10-pywayland-0.4.14/bin
/run/wrappers/bin
/home/wiseh/.local/share/flatpak/exports/bin
/var/lib/flatpak/exports/bin
/home/wiseh/.nix-profile/bin
/etc/profiles/per-user/wiseh/bin
/nix/var/nix/profiles/default/bin
/run/current-system/sw/bin
/nix/store/pywdfpw04di4nb3aas5ya9dfxjg7byxl-python3.10-mypy-0.971/bin
5

No, not at all. I suspect you just have a python added to your system-path somewhere. Lots of packages will use python, and that’s just the python package. Everything that uses python will be using it, which is what I wanted you to see.

The terminator thing is a very special case and I imagine you’re on a wild goose chase. I’ve forgotten how the python resolution works in detail, but check your PYTHONPATH, and double check you don’t actually add python to your systempath.

Tells me that you do just have it set in environment.systemPackages somewhere.

1 Like
6

This confuses me a lot. You were right, I had forgotten to comment on the testing I mentioned earlier in my configuration.nix. But what confuses me is that I already commented out those lines and carefully checked both files (configuration.nix and home.nix), and now there is no uncommented line that is referring to python3 but the problem persists. (Well, only the following configuration block in Neovim, but the situation is before I added this part of the configuration).

  programs.neovim = {
    enable = true;
    viAlias = true;
    vimAlias = true;
    vimdiffAlias = true;
    withPython3 = true;
    extraConfig = (builtins.readFile ./nvim/nvim.vim);
    extraPython3Packages = (ps: with ps; [
        ueberzug
        pynvim
    ]);

Also the result of nix why-depends --all /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6 is basically the same. It dropped from 56 occurrences to 52, but it still appears, in addition to as a systemd-boot dependency, as a nested system-path dependency. (I’ll leave just a snippet of the first few lines to make sure the information I’m giving is being interpreted correctly).

/nix/store/bpmp0np3byfkn5bxgazam3svln97bwj4-nixos-system-Eon-22.11.20220922.ae1dc13
├───/nix/store/7nc054hg8d6srbp6b3q36gs3msawaq45-systemd-boote[0m
│   └───/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6e[0m
├───/nix/store/w6a1fwrf9a75z828r2y7l2ckwmw13nr1-system-pathe[0m
│   ├───/nix/store/3yf7qy2x9g8c2vvfqhs9l7nlmq0nqp11-flatpak-1.12.7e[0m
│   │   └───/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6e[0m
│   ├───/nix/store/4pgd207ivxpfx0zfah0h9hd2mmsmkfxg-ranger-1.9.3e[0m
│   │   ├───/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6e[0m
...

On the other hand, when I check the PYTHONPATH variable I get nothing :confused:.

7

How is this python getting into your PATH? It’s being prepended, but it’s happening before the kitty wrapper script prepends its 3 lines. How are you starting kitty? Did you start it from a terminal with a nix-shell in it, perhaps?

1 Like
8

Ehhh… I really don’t know.

No sir, I run kitty via qtile shortcuts, and kitty is being installed directly from my home.nix. Also when I run it from Alacritty the same PATH appears without the first 3 lines you mention (as expected, I imagine).

Let me ask you a question. After the python that is giving me problems (the one on the fourth line), I see that my PATH has two more python3 (one related to qtile and one to pywayland). Since my home-manager python3 will only run when it gets to /home/wiseh/.nix-profile/bin, does that mean that even if we solve the problem for sz0j8k8ljh7...-python3, there will still be two other pythons left to solve that will want to shadow my user python?

Out of the question, I have been checking over and over again my configurations to see if at any point I declare something that could affect my PATH, but I don’t see anything. I will continue to investigate, but any other ideas I continue to keep an eye out for! Thanks!

9

Well, I don’t know how qtile is packaged in nixpkgs, but it does use python afaik, so it may be leaving its python in PATH when it starts things.

For reference, on my system with i3 + urxvt, my PATH is:

$ printenv PATH | tr ':' $'\n'
/run/wrappers/bin
/home/tejing/.nix-profile/bin
/etc/profiles/per-user/tejing/bin
/nix/var/nix/profiles/default/bin
/run/current-system/sw/bin

Which is the normal PATH on nixos (modulo username). Anything more than that is coming from somewhere more unusual, and shouldn’t be there if you didn’t mean to be in a special environment (like a nix shell).

In my view, kitty is improperly packaged because it adds to the PATH seen by the shell run inside it, and probably the same for qtile. It’s an unfortunately common packaging issue in nixpkgs, and I know at least one similar masking issue I’ve seen in the past was ultimately the result of the a terminal multiplexer it was being run under.

If those bin directories contain a python, yes.

2 Likes
10

We did it! I was able to find the problematic package: qtile.

Just in case it could be of help to someone, and because I feel that it is a really complex bug to debug as a newbie in NixOs, I will leave the process I followed taking into account all the previous help and references:

  1. I started by switching to a TTY to test if after logging in, not having any graphical environment, the problem persisted. The answer was no. Having switched to the TTY the python3 shading disappeared and I ended up with a path identical to the one mentioned by @tejing, i.e. my python3 installation with home-manager worked correctly.

  2. This led me to think that the error could be coming from any graphical program, so I started reproducing generations by restricting batches of packages with the intention of noticing if removing any of these packages would reset the PATH. It did not, no changes to my home.nix packages removed the shadowing.

  3. Remembering @tejing’s mention of qtile and his reference setup, I decided to try switching from WM and installed i3. That was it. The PATH was in perfect shape, and although kitty kept adding 3 lines, the python3 running was correct.

(Actually it is more appropriate to reverse step 2 and 3. I spent more than three hours looking for culprits in my home.nix packages, and it was easier to spend 10 by changing the WM).

Thank you very much for the help and kindness @tejing and @TLATER, your guidance was very helpful.

2 Likes
11

Please do file a nixpkgs bug report about qtile’s packaging if there isn’t one already.

12

Of course! I’ll get right on it.

I will also do it in kitty, maybe the 3 lines added to the path can generate similar problems with other packages.