Can nix replace brew/pyenv for managing python's installation?

Help
1

Pyenv is a massive pain in its own right for various other reasons, which is why I’m looking to nix to maybe alleviate that sort of issue. In large part nix+home-manager has rid me of brew, however pyenv basically requires brew on macOS so I can’t quite get rid of it.

As a contributor to projects where no one else is using nix, it’s very much outside my intention to be writing a lot of custom per-project nix-specific stuff that I’d rather not leak into the actual project source. So I’d really like to be using nix expressly for the purposes of installing multiple python versions. If it can also replace venv, great! (but based on articles I’ve already followed, I feel like that’s not gonna work out.

Unfortunately, in all my attempts I’ve been plagued by issues like

ImportError: dlopen(project/.venv/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-darwin.so, 2): Library not loaded:
 /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib
  Reason: image not found

or interactions with existing tooling, where bash scripts or things which spawn suprocesses fail to work.

tl;dr is there some way to use nix as though it’s pyenv or whatever without swapping my entire workflow and build chain to it?

2

This seems mostly to be a missing package problem, mostly when I’m contributing to projects I’m not the author of and who do not use Nix I just import a very simple shell.nix file, like this:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "<pkgname>";
  buildInputs = with pkgs; [
    pkgconfig
    openssl
    [...]
  ];
}

And for most projects it’s enough. Occasionally you’ll need some libraries for image conversions and so on, but for the most part you look at the error message, locate the missing package, do a nix search <pkg> and add it to the shell.nix file. In my case I use direnv to automatically install and launch a nix shell but you can do so manually with a simple nix-shell.

As for managing multiple versions of Python, sorry, I mostly just use whatever is the latest release of Python 3 in nixpkgs.

3

To complete @sondr3 answer: You may want to use the mkShell instead of mkDerivation since your Nix expression is meant to be used in a Nix shell only. If you need Python packages for your project, you may also define a Python environment:

with import <nixpkgs> {};
let
  python3Env = pkgs.python3.withPackages (ps: with ps; [
    numpy
    pylint
  ]);
in
pkgs.mkShell rec {
  buildInputs = [
    python3Env
];
}
2 Likes
4

I definitely started at a place similar to the above shell.nix, but kept running into issues related to psycopg2 and openssl. But copying your above and adding “python36Full” and “postgresql”, it maybe appears to be working now.

Is there no chance that my reinstalling brew and installing openssl there could be affecting things somehow? I could have sworn I tried every combination i could find of various examples i found online and nothing would fix all the problems.

One issue with your suggestion about python packages @smaret is that in order for package installations to work properly (e.g. ptpython to pick up venv-installed libraries) I can’t be using virtualenvs in a naive way any more as far as I can tell.

I had previously followed the advice of blog posts like this one: Using Nix to Create Python Virtual Environments – Thomas Churchman, but they inevitably always failed for the aforementioned openssl-type issues; which is why I’m suspicious of this currently working.

5

Is there no chance that my reinstalling brew and installing openssl there could be affecting things somehow?

It don’t think so.

One issue with your suggestion about python packages @smaret is that in order for package installations to work properly (e.g. ptpython to pick up venv-installed libraries) I can’t be using virtualenvs in a naive way any more as far as I can tell.

Why do you need venv? The nix-shell above should contain a Python interpreter with all the packages you need (similar to a venv).

but they inevitably always failed for the aforementioned openssl-type issues; which is why I’m suspicious of this currently working.

Could you please post the shell.nix you’ve been using so we can try to reproduce your problem?

6

Could you please post the shell.nix you’ve been using so we can try to reproduce your problem?

Unfortunately, I deleted it. Oh well, it doesn’t matter, if these new ones are working.

My current (apparently working) shell.nix is

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "example";
  buildInputs = with pkgs; [
    pkgconfig
    openssl
    postgresql
    python36Full
    python36Packages.pip
  ];
}

Adapting it to the style you suggested (while still using venv) appears to have no visible effect to me.

Why do you need venv? The nix-shell above should contain a Python interpreter with all the packages you need (similar to a venv).

I need to work within the existing tooling of the projects that I work on. These assume build tools like poetry (which calls out to pip), and pip directly. So I can go and add python36Packages.pip to the build inputs, but then see ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/nix/store/.../site-packages/...., which makes sense given what I know about how nix works.

Then from the above linked article, I can set PIP_PREFIX and PYTHONPATH in order to have pip/python install to and read from a writeable location to fix that.

While this initially seems to work, if I try to include any python packages (like ptpython) in the shell.nix, trying to install the requirements.txt eventually fails with

  Found existing installation: six 1.12.0
    Uninstalling six-1.12.0:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'RECORD'

because presumably ptpython or some upstream dep of it requires a different version of six, pip wants to swap the version, but that version is in a read-only location.

Removing that, proceeds past dependency installation, but fails at editably installing the project’s package with

    Checking .pth file support in .../_build/pip_packages/lib/python3.7/site-packages
    /nix/store/jn8dic1klsd1d550wb682ssnq9wsffg5-python3-3.7.5/bin/python3.7 -E -c pass
    TEST FAILED: .../_build/pip_packages/lib/python3.7/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH

Which I have no idea how to solve.

I wish I had my previous shell.nix, because with that, I had had something which could install everything (seemingly correctly, despite the openssl errors at runtime) but there was some issue with executing bash scripts which called out to python.

So far for me, it seems as though unless I’m willing to go through a lot of preparation before working on any project, or go all-in on using nix as the tooling (which i cant), these sorts of issues have cropped up in various places.

Unless there’s some silver bullet that’s guaranteed to solve all these issues, I think, for now, I’m happy just using it to obtain a working python interpreter specific to the project (which allows me to drop pyenv/brew), and then manage everything after that point the same way is typical for python development.

7

I’ve combined pipenv and nix-shell in a project where I needed to collaborate with a friend of mine who didn’t use Nix(OS). I ended up using the following default.nix file. It only includes Python 3 and virtualenv and then install pipenv via pip and all the dependencies in the Pipfile during the shellHook. Might be of some use.

8

For local python development, i generally leverage virtualenv, my shell.nix (this was written when I first started using nix, it’s messy):

let
  pkgs = import <nixpkgs> {};
  python = pkgs.python37;
  pythonPackages = python.pkgs;
in

mkShell {
  name = "pip-env";
  buildInputs = with pythonPackages; [
    # System requirements.
    readline
    git

    # Python requirements (enough to get a virtualenv going).
    h5py
    jupyter
    requests
    tensorflow
    typed-ast
    pandas
    virtualenvwrapper

    libffi
    openssl
  ];

  shellHook = ''
    # Allow the use of wheels.
    SOURCE_DATE_EPOCH=$(date +%s)

    # Augment the dynamic linker path or other packages you care about
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${pkgs.ncurses}/lib

    VENV=master37env
    if test ! -d $VENV; then
      virtualenv $VENV
    fi
    source ./$VENV/bin/activate

    # allow for the environment to pick up packages installed with virtualenv
    export PYTHONPATH=`pwd`/$VENV/${python.sitePackages}/:$PYTHONPATH
  '';
}
1 Like
9

Just a word of caution with using nixpkgs to manage python dependencies: you can’t easily control the versions of your packages.

If your project uses jsonschema>=2,<3, but your nixpkgs channel has jsonschema==3.1.1, then you’re going to get a version mismatch. In general, i use nixpkgs to bring in heavy-weight native dependencies (tensorflow, pandas, jupyter), and then just use virtualenv+pip to install the “light-weight” pure-python libraries.

10

On the flip side, if you’re a package maintainer, and you want to make sure you’re compatible with the latest versions of your dependencies, then nixpkgs is great as you get “caching” of your dependencies between environments.