A simple question for python development

Hello.

Why if I run a nix-shell like this:

with import <nixpkgs> {};
mkShell {
 name = "python-devel";
 venvDir = "venv";

 buildInputs = with python312Packages; [
  numpy
  jupyter
  ipykernel
  notebook
  pandas
 ];
postShellHook = ''code .''; 
}

Then everything work? I can even start vscode within the shell and I have fully functional jupyter notebooks kernel on vscode. I just need to run commands trough this shell and I’m able to code exactly like on any other ditro.

But why if I install these packages trough my configuration.nix file then nothing works??? I would love to avoid entering on a shell everytime… Just launching vscode from my app launcher and go from there

If you want to globally install python with packages, you should build a python which has all those things in its environment:

{ pkgs, ... }: {
  environment.systemPackages = [
    (pkgs.python3.withPackages (ppkgs: [
      numpy
      jupyter
      ipykernel
      notebook
      pandas
    ])
  ];
}

Then python will literally launch a python with those packages available, so you can import them.

Just installing numpy/jupyter/etc. in environment.systemPackages doesn’t install things into a global python env, because NixOS does not install libraries placed in environment.systemPackages. The packages you’re using are just the binary versions that you’re supposed to use directly from the command line interface, they’re not really intended to be used as development packages.

Using a nix-shell however installs any packages’ development targets as well, so for those packages it will register them in the python packaging env vars, making them available for python imports.

Using python3.withPackages is a bit of a workaround.


The “proper” way to do things on the other hand is to indeed use a shell.nix like you’re using, however instead of starting code with a shellHook, I’d suggest using direnv and one of the direnv vscode plugins that automatically switch into the shell when you open a project with one in it. That way you automatically get the correct environment for the project you’re working on, without having to rely on global library installations, which helps make your projects more reproducible.

That will also work perfectly for other languages :slight_smile:

3 Likes

Thanks. For my need the first option would be perfect.

This is what I did:

  environment.systemPackages = with pkgs; [
    # PYTHON
    (python312.withPackages (ps: with ps; [
        numpy
        jupyter
        ipykernel
        notebook
    ]))
  ];

When I import packages from python everythg works perfectly. Only one problem:

  • I create a venv and a ipynb file
  • I open vscode and open the ipynb file
  • If I select as kernel the .venv then vscode tells me to install jupyter notebook packages, then it simply does not work. Altough I can select as kernel my global python and then it would work.

But, if I do the same inside a nix-shell (using the shell file at post 1) then the venv that is generated works out of the box with vscode, no popup to install anything…

Any hint on that? If I get to fix this then my nixos would be a complete paradise

Sorry, not off the top of my head. Most likely the environment variable withPackages uses is the same as the one the venv modifies. You’d need to check the implementation of both withPackages and your venv implementation.

Realistically, you should be picking either venv or nix to manage your python packages.

1 Like

Thanks… But I don’t understand why, I mean the setup is exactly the same on configutation.nix and shell.nix… Why they do not work the same?

Maybe is it because the whole python installation is placed in the “venv” folder and vscode is picking up that one instead of “.venv” created by me?

That way when vscode uses the venv folder as kernel it would be the same situation of when it uses my system one

Because python3.withPackages works differently from the code that inserts python modules into the environment in such a way that python will find them.

I’ve not looked into the implementation, but my guess is that withPackages symlinks the packages into the directory that python loads modules from, while the nix-shell library installation code puts things into an environment variable.

Ah, hah, yes, that appears to be true: nixpkgs/doc/languages-frameworks/python.section.md at 9016c6f39ec462a112837b0a9819e703113a1377 · NixOS/nixpkgs · GitHub

I did not know that hook exists.

1 Like

Perfect. Thanks so much for giving me some more understanding. For now using the “global” way suits me best as I don’t want to bother creating a shell.nix file for every project of mine.

Have a great day and thanks again

Sorry, just one last question.

This means that everything gets installed on “venv” directory on folder where I run nix-shell. This means that everything will be available offline and will also survive garbage collections?

Bypassing your particular question but trying to answer the main one, if you are going to work on python projects which require diverse set of up-to-date dependencies I would stick with python-native package manager (pip, poetry, uv, etc).

You can setup those through nix-shell just fine, binding them to the python version from nixpkgs.

When you will encounter problems with libraries like numpy requiring runtime dependencies, you can solve them by providing LD_LIBRARY_PATH in your mkShell. This can mess with your other packages, but in practice I found it to be easy to fix by just overriding this env var for those again. Alternatively, you can look at the source code of devenv for how to override it only for python.

I’ve burnt dozens and dozens of hours on wrestling with python dependencies in nix in production environments, trying a lot of nix-native solutions, but sooner or later they all broke (especially when CUDA was involved). This turned out to be the most simple and robust one.

1 Like

Yes I’m aware of that library “trick”, for future reference here is a proper guide:

Anyway, thus far I’m just using something like that on my configuration.nix:

  environment.systemPackages = with pkgs; [
    # ---------------------------------- PYTHON ---------------------------------- #

    (python312.withPackages (ps: with ps; [
        pip
        uv

        numpy
        pandas
        
        streamlit
        
        jupyter
        ipykernel
        notebook

        statsmodels
        scipy
        tkinter
    ]))
   ];

Thus far no problem, even using jupyter notebooks on vscode.