Poetry2nix no module name pdm

Help
1

I’m trying to migrate a project (https://github.com/Nebucatnetzer/network_inventory/tree/poetry) to poetry and use it together with poetry2nix.
It contains this flake.nix:

{
  description = "A Python API for various tools I use at work.";
  inputs = {
    nixpkgs.url = github:NixOS/nixpkgs/nixos-22.05;
    flake-utils.url = github:numtide/flake-utils;
  };
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShell = pkgs.mkShell {
          buildInputs = [
            pkgs.gnumake
            (pkgs.poetry2nix.mkPoetryEnv {
              projectDir = ./.;
            })
            pkgs.python39Packages.poetry
          ];
        };
        shellHook = ''
          export DJANGO_SETTINGS_MODULE=network_inventory.settings.local
        '';
      });
}

However when I enter the shell I get this error and don’t know how to solve it.

[0/167 built, 4/9/77 copied (9.1/78.5 MiB), 8.1/35.8 MiB DL] fetching django-filter-22.1.tar.gz from https://cache.nixos.orgdirenv: ([/nix/store/a5kmn3sij0dlkxy5xxv7drs08k0qp267-direnv-2.31.0/bin/direnv export bash]) is taking a while to execute. Use CTRL-C to give up.
error: builder for '/nix/store/5axpdf5qb9fq2gx8c0iy2i8ajidszb9q-python3.9-findpython-0.2.2.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'pdm'
       > 
       > 
       For full logs, run 'nix log /nix/store/5axpdf5qb9fq2gx8c0iy2i8ajidszb9q-python3.9-findpython-0.2.2.drv'.
error: 1 dependencies of derivation '/nix/store/m7x3rrlj34m7dpjf78c87w0jlryw5519-python3-3.9.13-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kq0qc849zp7ml1s9qww2bvifam9m2w2w-nix-shell-env.drv' failed to build
direnv: nix-direnv: renewed cache
direnv: export ~XDG_DATA_DIRS

I tried to add it as a dependency with poetry and I tried to add it as a dependency through the flake.nix. But both attempts didn’t really work.

In addition I tried to just provide Python and Poetry to the shell and then create the virtual environment with Poetry but every time I exit the shell the Poetry venv seems to break and I have to reinstall it, in order to work.

2

I have the exact same problem with a different package that’s the dependency of a dev project (GitHub - frostming/marko: A markdown parser with high extensibility.).

3

I actually got the solution today thanks to the Matrix chat for poetry2nix.
You need to add an override for the build dependency of that package.
Something like the following, where findpython is the package that is failing to build and pdm is the missing build dependency. You can add as many overrides as you need.
The best is that you add the overrides to the upstream poetry2nix package when you fixed them (https://github.com/nix-community/poetry2nix/blob/0c5bae34bee822876a2dbb1c62d89cd1b29b648d/overrides/build-systems.json)

(pkgs.poetry2nix.mkPoetryEnv {
  projectDir = ./.;
  overrides = pkgs.poetry2nix.defaultPoetryOverrides.extend
    (self: super: {
      findpython = super.findpython.overridePythonAttrs (
        old: {
          buildInputs = (old.buildInputs or [ ]) ++ [ super.pdm ];
        }
      );
1 Like
4

Weirdly I’m still getting an error when overriding a couple of packages with a very similar pattern to you. marko builds, but findpython now doesn’t…

  outputs = { self, nixpkgs, flake-utils, flake-compat }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        # https://discourse.nixos.org/t/poetry2nix-no-module-name-pdm/23310/3?u=numinit
        overrides = pkgs.poetry2nix.defaultPoetryOverrides.extend
          (self: super: {
            findpython = super.findpython.overridePythonAttrs (
              old: {
                buildInputs = (old.buildInputs or []) ++ [ super.pdm ];
              }
            );
            marko = super.marko.overridePythonAttrs (
              old: {
                buildInputs = (old.buildInputs or []) ++ [ super.pdm ];
              }
            );
          });
      in {
        packages.mypackage = pkgs.poetry2nix.mkPoetryApplication {
          projectDir = ./.;
          inherit overrides;
        };
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            python3Full
            python3Packages.poetry
            (poetry2nix.mkPoetryEnv {
              projectDir = ./.;
              inherit overrides;
            })
          ];
        };
      });
}


$ nix develop
error: infinite recursion encountered

       at /nix/store/p6xfpp6i1a2h4ia3v81d1r414v6a603i-source/pkgs/stdenv/generic/make-derivation.nix:314:7:

          313|       depsHostHost                = lib.elemAt (lib.elemAt dependencies 1) 0;
          314|       buildInputs                 = lib.elemAt (lib.elemAt dependencies 1) 1;
             |       ^
          315|       depsTargetTarget            = lib.elemAt (lib.elemAt dependencies 2) 0;
(use '--show-trace' to show detailed location information)
5

Do you have a link to your project?

6

Not open source (yet) sadly. :frowning:

In any case, appreciate the help and quick response, think I’ll contribute the changes upstream once I get this working.

7

Ah well :slight_smile:

It took me quite some time to get it fully working today.
I had to override the packages here:
https://github.com/nix-community/poetry2nix/pull/819

I extend the documentation in poetry2nix a bit, my version can be found under the following link until the changes get merged upstream:
https://github.com/Nebucatnetzer/poetry2nix/blob/212b031b39e50f9518782edb35f335d40ecaf8e8/docs/edgecases.md

1 Like
8

Alright, I got it. That was annoying, and this may help those in a similar situation:

TL;DR: use pdm-pep517 instead of pdm if you get infinite recursion.

        overrides = pkgs.poetry2nix.defaultPoetryOverrides.extend                                                                                                                                                           
          (self: super: {                                                                                                                                                                                                   
            marko = super.marko.overridePythonAttrs (                                                                                                                                                                       
              old: {                                                                                                                                                                                                        
                buildInputs = (old.buildInputs or []) ++ [ super.pdm-pep517 ];                                                                                                                                              
              }                                                                                                                                                                                                             
            );                                                                                                                                                                                                              
            unearth = super.unearth.overridePythonAttrs (                                                                                                                                                                   
              old: {                                                                                                                                                                                                        
                buildInputs = (old.buildInputs or []) ++ [ super.pdm-pep517 ];                                                                                                                                              
              }                                                                                                                                                                                                             
            );                                                                                                                                                                                                              
            findpython = super.findpython.overridePythonAttrs (                                                                                                                                                             
              old: {                                                                                                                                                                                                        
                buildInputs = (old.buildInputs or []) ++ [ super.pdm-pep517 ];                                                                                                                                              
              }                                                                                                                                                                                                             
            );                                                                                                                                                                                                              
            mypackage = super.mypackage.overridePythonAttrs (                                                                                                                                                                       
              old: {                                                                                                                                                                                                        
                buildInputs = (old.buildInputs or []) ++ [ super.poetry ];                                                                                                                                                  
              }                                                                                                                                                                                                             
            );                                                                                                                                                                                                              
        });
9

Right I had the same problem with pytoolconfig.
For anyone struggling like me I found the solution here:

So search for the build dependencies in the upstream package e.g. in setup.py, pyproject.toml.

10

Updated nixpkgs and this is happening again, even with the super.findpython.overridePythonAttrs change now…

          findpython = super.findpython.overridePythonAttrs (
            old: {
              buildInputs = (old.buildInputs or []) ++ [ super.pdm-pep517 ];
            }
          );