Issues trying to create a dev environment with Flakes (`ModuleNotFoundError: No module named 'poetry'`)

1

I have been working on a Python project using Flakes, and I had no issues so far running it until today.

I have been trying to run nix develop, but I’m getting the error on the title. Here are the logs:

Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/aw1mh276dfy8j852zpq9p89nckpq9qxm-python-trovo-0.1.3.tar.gz
source root is python-trovo-0.1.3
setting SOURCE_DATE_EPOCH to timestamp 1643126500 of file python-trovo-0.1.3/PKG-INFO
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
Removing path dependencies
Finished removing path dependencies
Removing git dependencies
Finished removing git dependencies
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing pipBuildPhase
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the perm
issions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /build/pip-ephem-wheel-cache-t1ig8s0k
Created temporary directory: /build/pip-req-tracker-i0np8bvq
Initialized build tracking at /build/pip-req-tracker-i0np8bvq
Created build tracker: /build/pip-req-tracker-i0np8bvq
Created temporary directory: /build/pip-wheel-dhug02us
Processing /build/python-trovo-0.1.3
  Created temporary directory: /build/pip-req-build-_czw8a2z
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Added file:///build/python-trovo-0.1.3 to build tracker '/build/pip-req-tracker-i0np8bvq'
    Created temporary directory: /build/pip-modern-metadata-08hvcn9q
    Running command /nix/store/19j43hyh37jqnn9l7rg6shk5barbn209-python3-3.9.6/bin/python3.9 /nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmpmnnpi6ba
    Preparing wheel metadata ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 142, in run
    requirement_set = resolver.resolve(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 429, in make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 200, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 306, in __init__
    super().__init__(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 317, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
    File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 570, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 60, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 36, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 549, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 534, in _generate_metadata
    return generate_metadata(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 31, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 184, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 275, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/nix/store/4csz58gg7m76a6zi08jqjyx06rg2zkhj-python3.9-pip-21.1.3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/nix/store/19j43hyh37jqnn9l7rg6shk5barbn209-python3-3.9.6/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'

Removed file:///build/python-trovo-0.1.3 from build tracker '/build/pip-req-tracker-i0np8bvq'
Removed build tracker: '/build/pip-req-tracker-i0np8bvq'

Not sure if this is an error from Flakes or if this is an issue from the library.

Here is my flake.nix:

{
  description = "Alerts when a streamer is live";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/release-21.11";
    poetry2nix.url = "github:nix-community/poetry2nix";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, poetry2nix, utils }: utils.lib.eachDefaultSystem (system:
    let
      pkgs = import nixpkgs {
        inherit system;
        overlays = [
          poetry2nix.overlay
        ];
      };

      customOverrides = self: super: {
        python-twitter = super.python-twitter.overrideAttrs(old: {
          buildInputs = old.buildInputs ++ [ self.pytest-runner ];
        });
      };

      # General poetry settings
      python = pkgs.python39;
      projectDir = ./.;
      overrides = pkgs.poetry2nix.overrides.withDefaults (customOverrides);

      env = pkgs.poetry2nix.mkPoetryEnv {
        inherit overrides python projectDir;
      };

      # Other project settings
      extraPkgs = with pkgs; [ gnumake poetry ];

    in {
      devShell = pkgs.mkShell {
        buildInputs = [ env ] ++ extraPkgs;
      };

      defaultPackage = pkgs.poetry2nix.mkPoetryApplication {
        inherit overrides projectDir;
      };
    });
}
2

If I try to use a shell.nix file and import the python-trovo library, it works with no issues.

nix/pkgs.nix

import (builtins.fetchTarball {
    name = "nixos-21.11";
    url = "https://github.com/NixOS/nixpkgs/archive/386234e2a61e1e8acf94dfa3a3d3ca19a6776efb.tar.gz";
    sha256 = "sha256:1qhfham6vhy67xjdvsmhb3prvsg854wfw4l4avxnvclrcm3k2yg8";
})

nix/requirements.nix

with import ./pkgs.nix {};

rec {
    python = python39;

    pythonPackages = python39Packages;

    base = with pythonPackages; [
      python-twitter
      pytwitchapi
      pyyaml
      requests
    ];

    tests = [ pythonPackages.pytest ];

    dev = [ pythonPackages.pycodestyle ];
}

shell.nix

with import ./nix/pkgs.nix { };
let
  requirements = import ./nix/requirements.nix;
  pythonEnvironment = requirements.python.buildEnv.override {
    extraLibs = requirements.base ++ requirements.tests ++ requirements.dev;
  };
  
  python-trovo = requirements.pythonPackages.buildPythonPackage rec {
    pname = "python-trovo";
    version = "0.1.3";

    src = requirements.pythonPackages.fetchPypi {
      inherit pname version;
      sha256 = "sha256-iKG6UGc3pUjN00kxkBRL0ZbzB8OCbWOi1kb+YpLzOfE=";
    };

    propagatedBuildInputs = with requirements.pythonPackages; [ requests ];
  };
  extraPkgs = [ gnumake poetry ];
in
mkShell {
    buildInputs = [ pythonEnvironment python-trovo ] ++ extraPkgs;
}
3

I found a way to create the environment again (not a fix):

  • Cloned the project on another place
  • Ran nix develop on its current state to see if there are any issues. None so far.
  • Ran poetry update to update the libraries (including the one that is giving issues so far, python-trovo)
  • Committed the new poetry.lock
  • Ran nix develop again to see if this is an issue regarding a version update. No issues so far, my environment is available.
4

I hit the same issue when I added the package sh to my pyproject.toml
so weird

Did you ever root cause what was causing it?

5

Okay I needed:

  poetryOverrides = self: super: {
    sh = super.sh.overridePythonAttrs (old: {
      buildInputs = (old.buildInputs or [ ]) ++ [ super.poetry ];
    });

You can read about why here: https://github.com/nix-community/poetry2nix/blob/5b3a5151cf212021ff8d424f215fb030e4ff2837/docs/edgecases.md

1 Like
6

I can’t recall what I did at that time, but I do remember I had to use another override besides the one I did with python-twitter. Maybe it was that.