Adding python packages to the python interpreter *inside* the neovim derivation

Help
1

I am trying to use a plugin with neovim that needs a special python library (unidecode). I added that python package to my python interpreter only to find out that the neovim derivation ships it’s own python interpreter.

I first tested this

$ nix-shell --pure -p neovim 'python3.withPackages(p: [p.unidecode])'
$ python -c "import unidecode"   # exit code 0
$ nvim -u NONE --headless -c "py3 import unidecode" -cq
Error detected while processing function provider#python3#Call:
line   18:
Error invoking 'python_execute' on channel 3 (python3-script-host):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'unidecode'

and also checked the respective python search paths (same nix-shell)

$ python -c 'import sys; print(sys.path)'
['',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python38.zip',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/lib-dynload',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/site-packages',
'/nix/store/ivsjq570lkr4g8gpgpbqzdw26ia6r76v-python3-3.8.5-env/lib/python3.8/site-packages']
$ nvim -u NONE --headless -c "py3 import sys;print(sys.path)" -cq
['/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python38.zip',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/lib-dynload',
'/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/site-packages',
'/nix/store/qrbjgnc7kbh7c25ly8i400gbh31frjk0-python3-3.8.5-env/lib/python3.8/site-packages',
'_vim_path_']

As you might have guessed python3.withPackages(p: [p.unidecode]) did put
unidecode in /nix/store/ivsjq570lkr4g8gpgpbqzdw26ia6r76v-python3-3.8.5-env/lib/python3.8/site-packages.
For the record, the python interpreter is also different in neovim and outside (same nix-shell):

$ python -c 'import sys; print(sys.executable)'
/nix/store/ivsjq570lkr4g8gpgpbqzdw26ia6r76v-python3-3.8.5-env/bin/python3.8
$ nvim -u NONE --headless -c 'py3 import sys;print(sys.executable)' -cq
/nix/store/qrbjgnc7kbh7c25ly8i400gbh31frjk0-python3-3.8.5-env/bin/python3.8

I tested if I can set the vim variable python3_host_prog to the outside python interpreter but that obviously does not work as this python version does not have the pynvim library installed. I could add that library to its searchpath but than I would have to find the correct path to the outside python interpreter on each launch of neovim.

Is there a way that I can add a python package to the search path of the python interpreter used by neovim, from the nix side of things? The neovim derivation does not have a .withPackages for python stuff but maybe I can access the one of its python dependency?

Or is there some other simple solution to get a specific python package to load inside neovim’s python interpreter?

2

Would you mind trying out something like this:

with import <nixpkgs> {};
neovim.override {
  withPython3 = true;
  extraPython3Packages = p: with p; [ unicode ... ];
}
1 Like
3

It’s also possible to use home-manager, like this:

    neovim = {
      enable = true;
      withPython = true;
      withPython3 = true;

      extraPythonPackages = (ps: with ps; [
        unidecode
        pynvim
      ]);

      extraPython3Packages = (ps: with ps; [
        pynvim
        unidecode
        black
        isort
      ]);
    };
4

@brogos I am not (yet?) using home-manager so I did not try this. It looks very similar to Ma27’s solution.

@Ma27 thank you, that worked.

Now about the “learning” part (this is posted in the “Learn” category ;). Can you describe a “path of thought” or a “search path through the docs and sources” how I could have figured that out myself?
If I am looking at nix edit nixpkgs.neovim the only pname I see is “neovim-unwrapped” but even if I ignore that I do not see any reference to extraPython3Packages (nor withPython3). So I have trouble figuring out where this hooks in. My original assumption was that there must be a reference to a python derivation in there and that I could somehow call its withPackages but the working solution seems to hook in somewhere else.

5

@brogos AFAICS this is just a wrapper on top of the package, right?

@lucc So the reason why I knew about this problem specifically was that I had the same issue the day before :smiley:

But to be more serious, you may have already noticed that this ecosystem isn’t known for its good documentation. Generally I’d suggest to proceed like this when stumbling upon problems like this:

  • Check if something is documented in the NixOS manual (to get the manual for your system precisely, you can run nixos-help and it will be opened in your $BROWSER). A good alternative is also nixos.wiki or the nixpkgs manual (yes, we have two manuals, the latter one is mostly for packages and not NixOS-related things). You can access that at Nixpkgs 23.11 manual | Nix & NixOS
  • If that didn’t work, try to search in the issue tracker of nixpkgs (and probably this Discourse forum as well) for keywords about your problem. In my personal experience you’ll find similar issues with (at least) some workarounds fairly often.
  • If none of that worked, you’ll have to grep through your local nixpkgs checkout (Hint: if you look for a definition of e.g. a library function you want to search for function_name =). While this can be indeed fairly frustrating it’ll help in my experience to get more familir with Nix code and commonly used patterns for e.g. packages :slight_smile:
6

Thanks @Ma27, some of these I knew already :slight_smile:

7

@Ma27 It seems to be a wrapper