NixOS + IntelliJ + Python

Help
1

I’ve just started working on a Python project in IntelliJ Idea. It uses some Python packages so I added them to /etc/nixos/configuration.nix:

python39Packages.setuptools
python39Packages.psycopg2
python39Packages.flask_sqlalchemy
python39Packages.sqlalchemy
python39Packages.flask

and ran nixos-rebuild switch and nix-collect-garbage.

IntelliJ shows the packages os, flask, flask_sqlalchemy and psycopg2 as missing. It redlines them in the import declarations.

So in IntelliJ I go to platform settings / SDKs and add a system SDK for Python 3.9. Where is it?


$ find /nix/ -name python -type f -ls
  5880022     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/zmv5aycaa7jqsym7m8r5jf6ij85h92y1-python3-3.9.13-env/bin/python
  5881530     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/gp38glll1sfbjk3a0sgf3gnqjn6blls9-python3-3.9.13-env/bin/python
  6600857     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/4h9b1dl9ivzbkqxs1ix9hsqbp8gq8l82-python-2.7.18-env/bin/python
  6867049      4 -r--r--r--   1 root     root         1767 Jan  1  1970 /nix/store/rwv2yf25yrwfqp9fia6hjhw9825iyfcq-bash-completion-2.11/share/bash-completion/completions/python
  6867241      4 -r--r--r--   1 root     root          331 Jan  1  1970 /nix/store/rwv2yf25yrwfqp9fia6hjhw9825iyfcq-bash-completion-2.11/share/bash-completion/helpers/python
  7128467     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/af2yihxp23l5ial6p6j5jp54h9m6xwll-python3-3.9.13-env/bin/python
  7130887     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/54xnhsbk25a9w87z4ana80kyr4gkzg47-python3-3.9.13-env/bin/python
  7282398     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/jxa2j7x0gxw4kjf39ipn3s4yp6m4a0d3-python3-3.9.13-env/bin/python
  7373098     40 -r-xr-xr-x   1 root     root        37912 Jan  1  1970 /nix/store/i6f7qkpa8z8c80swfwhigf4dqcrbzscr-python3-3.9.13-env/bin/python

I tried all six candidates in the Python SDK home path but IntelliJ still redlines the packages it says are missing.

What do I do to fix this? I tried to set up a venv but the build of psycopg2 fails with a compilation error.

2

Python packages don’t work like that, which is unfortunately a bit confusing. The manual explains it: Nixpkgs 23.11 manual | Nix & NixOS

But in a nutshell, what you’ve done is you’ve installed the python packages, but you’re using a python that is unaware of them because they’re not in the sitepackages of that python installation.

If they were, there would be no way to have different python environments for different packages, and you’d end up in the weird world you have on other distros where either you use virtualenv and sidestep the distro package manager (and indiscriminately download random python binaries from the internet), or you pollute all your applications that use python with random installed libraries, which is a nightmare if two applications want two different package versions.

Instead, on NixOS, you essentially craft a python “instance” that has a specific set of python packages installed. You do that using python.withPackages, like so:

python39.withPackages(ps: with ps; [
    setuptools
    psycopg2
    flask_sqlalchemy
    sqlalchemy
    flask
])

That will create a python 3.9 with those packages installed, so replace any existing python, or python packages, with it. If you don’t want to use this python for everything, use specific shell.nix-es with each of your projects instead.

Note that you will need to put brackets around that whole expression if you put it in the environment.systemPackages list, otherwise nix will treat the argument as additonal list entries and complain.

It’s not very intuitive, but as I’ve explained, there are good reasons for this, and I don’t think there’s a better way of doing this. It’s arguably also largely python’s fault for having a rather short-sighted packaging story, but I don’t think any modern language does this better…

There may be some ways to make it more obvious to new users that python packages work a little differently here, like having a special search UI, not showing libraries in search results at all, or having an option for python packages, but those choices would have trade-offs of their own.

I think the recent suggestion of “usage notes” for package metadata might ultimately be the place best suited for gotchas like this one.

2 Likes
3

That’s brilliant; thank you. It all works now. Explanation noted. I must admit that with a lot on my plate currently, I don’t have time to delve deeply into Nix and NixOS and remain very much a consumer-grade user.

4

Yep, nix/os certainly still needs some work to be comfortably usable by a user who doesn’t put deliberate effort into learning. Hope you get enough help here for the time being :slight_smile: