Python3 not importing modules

dav.dap · September 28, 2022, 10:33am

Hi everyone,

My python3 installation cannot import modules despite apparently installing them:

$ python3
Python 3.10.6 (main, Aug  1 2022, 20:38:21) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

(same as e.g.: pandas).

I followed this wiki and integrated it into my configuration, as follows:

configuration.nix:

{ config, pkgs, ... }:

let
  # Unstable packages
  unstableTarball = fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz;
in
{
  # Unstable packages
  nixpkgs.config = {
    packageOverrides = pkgs: {
      unstable = import unstableTarball {
        config = config.nixpkgs.config;
      };
    };
  };

  imports =
    [ # ...
      ../imports/systemPackages_standard.nix
    ];
  # ...
}

systemPackages_standard.nix:

{ config, pkgs, ... }:

let
  # Custom Python package with all the (Python) imports I need
  my-python-packages = python-packages: with python-packages; [ # ...
    numpy
    pandas
  ]; 
  python-with-my-packages = pkgs.unstable.python3.withPackages my-python-packages;
in
{ # ...
  environment.systemPackages = with pkgs.unstable; [ # ...
    python-with-my-packages
 ];
}

Thank you
Davide

TLATER · September 28, 2022, 11:49am

Interesting! My gut feel is that it’s being masked by e.g. a python installed through nix-env. What does

nix-env --query

give you? If there is anything in there, besides uninstalling it you should also read through this website.

If you use flakes, nix profile list might also show you something that shadows it.

If it’s not being shadowed, what does:

ls -l $(find "$(dirname $(which python))/.."  -name site-packages)

give?

dav.dap · September 28, 2022, 12:48pm

nix-env --query returns a blank page.

Never used flakes:

$ nix profile list
error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override

The last command gives:

$ ls -l $(find "$(dirname $(which python))/.."  -name site-packages)
totale 12
dr-xr-xr-x 2 root root 4096 Jan  1  1970 __pycache__
-r--r--r-- 1 root root  119 Jan  1  1970 README.txt
-r--r--r-- 1 root root 1659 Jan  1  1970 sitecustomize.py
lrwxrwxrwx 1 root root   44 Jan  1  1970 _sysconfigdata__linux_x86_64-linux-gnu.py -> ../_sysconfigdata__linux_x86_64-linux-gnu.py

I don’t know if somehow having set the experimental packages declaratively on configuration.nix instead of through using nix-env may have resulted into two distict versions of pyton? But I do not know how to veryfy that.

TLATER · September 28, 2022, 1:06pm

If you were using the correct python env, there would be directories for your packages in there.

So you’re definitely not using the python from your config. The question is, what python are you using? At this point I’m not sure how to check that, unfortunately… Maybe check if which python points at a non-system profile?

Using nix-env -q, to check if you installed a second instance of python with nix-env. Your configuration looks correct to me otherwise, but sharing your full config could help spot something unusual.

dav.dap · September 28, 2022, 1:35pm

My full configuration

FRidh · September 28, 2022, 2:04pm

You could try and compare

readlink -f $(which python3)

with

readlink -f $(which f2py)

Both should point to the same store path. f2py is a program provided by numpy.

dav.dap · September 28, 2022, 2:13pm

Indeed they are NOT in the same store path:

readlink -f $(which python3)
$ /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6/bin/python3.10
$ readlink -f $(which f2py)
/nix/store/7n5rrvskzg50k0jj111dsyc150d0dk5i-python3-3.10.6-env/bin/f2py

but I honestly do not know why

FRidh · September 28, 2022, 2:39pm

The first is the interpreter by itself.

With the following you can see what refers to your Python

$ nix-store -q --tree /run/current-system

Maybe that helps you locating the source. Alternatively, with the new nix command you can use

$ nix why-depends /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6

If that says it does not depend on /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6 then you got in on your PATH from something else but your system profile.

dav.dap · September 28, 2022, 3:05pm

On python3:

$ nix --extra-experimental-features nix-command why-depends /run/current-system /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6
/nix/store/9qv1x6db19lmha2vk4sj5rk63lqlw9vd-nixos-system-nixos-22.05.3167.e64df9c5c8f
└───/nix/store/x7vabwvxr5nwk72fvcgb5fg36bbydxd8-system-path
    └───/nix/store/2j0i3q3cjwlwadxkn1956zmr7k83717h-vim_configurable-9.0.0244
        └───/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6

python3 depends on vim_configurable ??

On python3-env which contains numpy:

$ nix --extra-experimental-features nix-command why-depends /run/current-system /nix/store/7n5rrvskzg50k0jj111dsyc150d0dk5i-python3-3.10.6-env
/nix/store/9qv1x6db19lmha2vk4sj5rk63lqlw9vd-nixos-system-nixos-22.05.3167.e64df9c5c8f
└───/nix/store/x7vabwvxr5nwk72fvcgb5fg36bbydxd8-system-path
    └───/nix/store/7n5rrvskzg50k0jj111dsyc150d0dk5i-python3-3.10.6-env

I used to have vim_configurable for a while before removing it and deciding to configure vim via home-manager. Why has it remained in the system? Would it be useful (provided that it is possible) to purge the store completely and rebuild the configuration without formatting and reinstalling?

FRidh · September 28, 2022, 3:56pm

Then vim_configurable is propagating using Python to the user environment using ‘propagatedUserEnvPkgs’, something which I think should not be used because it can lead to issues likes these.

Nevermind, it is not propagating, it just depends on it. That is OK. But you still have it somehow in your PATH.

dav.dap · September 28, 2022, 4:03pm

But I do not use propagatedUserEnvPkgs anywhere in my configuration

FRidh · September 28, 2022, 7:45pm

No, but a module might. Anyway, if that reference you showed is the only one, then it would seem some home-manager configuration or something else manipulating your PATH introduced it.

dav.dap · September 28, 2022, 8:30pm

is there any way to verify that?
I linked the gitlab page to my configuration (including home-manager) in a post above

dav.dap · September 29, 2022, 2:10pm

I made some test removing and adding packages to my configuration and I noticed that python lmodules do work in my configuration - if python3 is execuded from from the gnome builtin console. They do NOT work if is executed on terminator, which is the console I routinely use.
Has anyone got any idea about why terminator display this behaviour?

Thank you
Davide

tejing · September 29, 2022, 2:19pm

Because terminator is written in python, so it gets run with (its version of) python in the environment.

TLATER · September 29, 2022, 2:27pm

Ugh. Shouldn’t the loaded shell override the PATH? Guess not since you should inherit the system settings.

Still, that’s painful. You can probably override the python imstance it’s using: nixpkgs/default.nix at 13cbe534ebe63a0bc2619c57661a2150569d0443 · NixOS/nixpkgs · GitHub

FRidh · September 29, 2022, 2:42pm

This could probably be fixed by changing the packaging. Instead of using the default wrapping, one would create a python3.withPackages and patch the shebang of terminator. That way the program is started with it’s own Python environment, but it won’t leak.

tejing · September 29, 2022, 2:46pm

It can definitely be fixed by the packaging. Worst case scenario, you can patch it to run an “unwrapping” script around whatever shell it runs, to clean the environment back up.

dav.dap · September 30, 2022, 8:56pm

Thank you for clarifying this point.
Eventually I opted for switching to another terminal (ddterm, a gnome extension), which now works fine.