Python development with Poetry, with nix supplying Python

I’d like to have NixOS supply Python, but let Poetry manage my dependencies entirely. Currently I’m doing this:

default.nix:

with import <nixpkgs> {};
with pkgs.python38Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    python39Full
    python39Packages.virtualenv
    python39Packages.pip
    postgresql
  ];
  src = null;
}

.envrc:

use_nix
source venv/bin/activate
unset PS1

I do this to setup my environment:

  1. python -m virtualenv venv
  2. . ./venv/bin/activate
  3. pip install poetry
  4. poetry install

What I want to happen from now on is to work in the virtual environment, and have all the installed Python packages be inside of that virtual environment. It doesn’t look like that happening, though. See the location of these two packages:

$ poetry run pip show boto3
Name: boto3
Version: 1.18.29
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /home/stian/.cache/pypoetry/virtualenvs/some_virtualenv-py3.8/lib/python3.8/site-packages
Requires: s3transfer, jmespath, botocore
Required-by: moto

$ poetry run pip show pluggy
Name: pluggy
Version: 0.13.1
Summary: plugin and hook calling mechanisms for python
Home-page: https://github.com/pytest-dev/pluggy
Author: Holger Krekel
Author-email: holger@merlinux.eu
License: MIT license
Location: /nix/store/l2i2x3g16g6dbwsw4iii614dh3xqgg9m-python3.8-pluggy-0.13.1/lib/python3.8/site-packages
Requires:
Required-by: tox, pytest

Why is pluggy from the nix store being used? Why is it not installed into my virtual environment?

Any pointers towards a solution, or tips for improvement of my workflow, would be very much appreciated. Thanks!

Probably because poetry either did not install it on its own because it is not a dependency of your project, or because the version of pluggy in the store is the version of pluggy as in the lockfile.

Thanks for answering! Those are good theories. Some more info:

  • poetry show --tree shows me that pluggy is a dependency of pytest, which I have a dev dependency on.
  • pluggy is present in poetry.lock.
  • The version of pluggy in poetry.lock matches a version of pluggy that I have in my nix store.
  • I see that when I do echo $PYTHONPATH | grep pluggy in my project folder, I do see that specific pluggy being available on the python path. This leads me to think that pluggy is a dependency of one of the things stated in my default.nix.

I’ve tried googling variants of “force poetry to not use local packages”/“poetry don’t use local packages”/“poetry only use pypi”, but no luck yet.

Poetry will always reuse what it sees in the system, to save time, space and data.

Poetry does also not “purify” the environment. Whatever you have installed in a “parent” environment will be visible to your python application you develop.

Thanks for confirming that, @NobbZ . Do you know if there’s a way to disable that behaviour?

No, I am not aware of a way to do so, sadly… Caused already trouble in the company I am working for, due to the global polution of the pythonpath on the Ubuntu machines of my coworkers…

The next logical step seems to use a more minimal Python setup by replacing python39Full by python39

I suggest you use pkgs.poetry and don’t touch anything from python*Packages in your development environment, as you’ve already experienced this works by setting $PYTHONPATH which leaks into your project.
pkgs.poetry takes care of properly vendoring the Poetry dependencies and will not cause them to leak into your project.

A correct shell.nix/default.nix would look something like:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  nativeBuildInputs = [
    pkgs.poetry
    pkgs.python3
  ];
}
4 Likes

side note:

Why do you call this “logical” (with or without accounting for a poetry setup)?

  • doesn’t it depend on the project/requirements? Do you know what Python3.xFull means?

I’ve tried now with this:

default.nix:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    python39
    python39Packages.poetry
  ];
  src = null;
}

But there’s a bug in poetry version 1.1.6 (which the stable channel gives me): I get the error 'EmptyConstraint' object has no attribute 'allows' when running poetry install. I see here that the current version of poetry in nixpkgs is 1.1.7, but the error I get is only fixed in version 1.1.8 (source). When I get some time, I’ll try to update the poetry derivation to 1.1.8.

Thanks for the help so far!

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    python39
    python39Packages.poetry
  ];
  src = null;
}

Is still incorrect (leaks into $PYTHONPATH) and is less up-to-date than the standalone Poetry derivation:

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    python39
    poetry
  ];
  src = null;
}

If you are interested in building Poetry with a specific Python version you can override it like:

stdenv.mkDerivation {
  name = "impurePythonEnv1d";
  buildInputs = [
    python39
    (poetry.override { python = python39; })
  ];
  src = null;
}

I would also recommend using a nicer abstraction for development shells:

mkShell {
  packages = [
    python39
    (poetry.override { python = python39; })
  ];
}

In my mind the Poetry derivation in the python*Packages sets is not to be used for project development (ie interactive use) but should instead only be used as a PEP517 build-system when manually packaging for nixpkgs.

3 Likes

Logical, just in the meaning to construct environment incrementally from a minimal core

to my knowledge, poetry is only tested with python (means pythonX.XFull in nix)

  • poetry is not build to work with pythonX.X (or other slim/manipulated variant of python) ?

If you would like to go minimal, it should not happen

  • before/while constructing an environment incrementally but
    after everything is developed and all tests are in place (and preferable as parameter/config argument)

(*in a lot of use cases there is no need for the trade-off to save 3MB vs being not maintainable/“tested”)

I agree with you. But it’s difficult to identity what is covered by pythonX.XFull (not a simple Nix expression to explore). Consequently my proposal was more a debug approach to minimize the elements to consider.

Thank you for all the tips! I got things working here with this:

{ pkgs ? import (fetchTarball https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) {} }:

pkgs.mkShell {
  nativeBuildInputs = [
    pkgs.python39
    (pkgs.poetry.override { python = pkgs.python39; })
  ];
}

Using unstable got me version 1.1.8 of poetry instead of 1.1.5 on stable.

Following up on this now, since I’m back working on python projects using nixos.

This stackoverflow answer says to use the flag --ignore-installed along with the pip install to force installation of packages into the virtual environment. This worked for me, and made pycharm happy. When I didn’t include that flag, then pycharm wouldn’t see fex the six package or the requests package. Since they were dependencies of something in my shell.nix, they were available in the nix store, visible to pip, and thus pip wouldn’t install them into the virtual environment. Adding that flag solved my problem.