How to create a poetry2nix environment with a flake?

[jademackay]

I have created a flake in an existing poetry project with

nix flake init --template github:nix-community/poetry2nix

Which yields

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    {
      # Nixpkgs overlay providing the application
      overlay = nixpkgs.lib.composeManyExtensions [
        poetry2nix.overlay
        (final: prev: {
          # The application
          myapp = prev.poetry2nix.mkPoetryApplication {
            projectDir = ./.;
          };
        })
      ];
    } // (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlay ];
        };
      in
      {
        apps = {
          myapp = pkgs.myapp;
        };

        defaultApp = pkgs.myapp;

        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            (python310.withPackages (ps: with ps; [ poetry ]))
          ];
        };
      }));
}

However, my attempts to modify this to produce a devShell with the contents of pyproject.tom/poetry.lock available in the environment have all been failures, typically of this kind:

error: infinite recursion encountered

       at /nix/store/n0f21zpy9w90sa9v75iwx6fmk897z3nw-source/pkgs/stdenv/generic/make-derivation.nix:311:7:

          310|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          311|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          312|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;
(use '--show-trace' to show detailed location information)

Any guidance on this is appreciated!

[Nebucatnetzer]

I’m still figuring out how to build a container image but the dev shell, tests and package should be working. FYI it’s a Django application.

GitHub - Nebucatnetzer/network_inventory

[sepiabrown]

How are pyproject.tom/poetry.lock written?

It’s quite hard to reproduce the error if you don’t provide any details.

[jademackay]

Thanks @Nebucatnetzer , I cloned your project and was able to instantiate the environment. Your flake is the same as my first variations on the template in that I too added a mkPoetryEnv attribute to the overlay and then included that in the buildInputs of a devShell. This indicates there is something that upsets poetry2nix in the pyproject/lock. This was my hunch from the beginning, I was hoping the infinite recursion would be strongly diagnostic of something

[jademackay]

Hi @sepiabrown , Thanks for your reply. I will share the poetry toml/lock as soon as I can.

[jademackay]

Hi @sepiabrown, here is a more concrete description. Thanks for taking an interest!

Minor changes to the poetry2nix template:

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    {
      # Nixpkgs overlay providing the application
      overlay = nixpkgs.lib.composeManyExtensions [
        poetry2nix.overlay
        (final: prev: {
          # The application
          myapp = prev.poetry2nix.mkPoetryApplication {
            projectDir = ./.;
          };
          # The env
          myenv = prev.poetry2nix.mkPoetryEnv {
            projectDir = ./.;
          };
        })
      ];
    } // (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlay ];
        };
      in
      {

        packages.default = pkgs.myapp;
        packages.myapp = pkgs.myapp;
        packages.myenv = pkgs.myenv;
        
        devShells.dev = pkgs.mkShell {
          buildInputs = with pkgs; [
            #(python310.withPackages (ps: with ps; [ poetry ]))
            pkgs.myenv
          ];
        };
        
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            (python310.withPackages (ps: with ps; [ poetry ]))
          ];
        };
      }));
}

Re-create the poetry project:

nix develop
poetry init -n
poetry add setuptools@^61.2.0 geopandas@^0.11.1 pandas@^1.4.3 shapely@^1.8.2 folium@^0.12.1 matplotlib@3.5.2 numpy@^1.23.1 openmatrix@^0.3.5.0 pyyaml@^6.0 dask@2022.9.2 distributed@2022.9.2 tqdm@^4.64.0 scipy@^1.9.1 ipfn@^1.4.4 pyarrow@^9.0.0 jupyter@^1.0.0 notebook@^6.4.12 ipykernel@^6.15.3 keplergl@^0.3.2 sklearn@^0.0 protobuf@3.20.0 matsim-tools@^0.0.14 pygeos@^0.13 dvc@^2.34.2 dvc-s3@^2.21.0 

Now let’s materialize our poetry2nix environment

nix develop .#dev
error: infinite recursion encountered

       at /nix/store/n0f21zpy9w90sa9v75iwx6fmk897z3nw-source/pkgs/development/interpreters/python/python-packages-base.nix:52:41:

           51|     modules = lib.filter hasPythonModule drvs;
           52|   in lib.unique ([python] ++ modules ++ lib.concatLists (lib.catAttrs "requiredPythonModules" modules));
             |                                         ^
           53|
(use '--show-trace' to show detailed location information)

… pondering how to see what is leading to the infinite recursion.

[sepiabrown]

@jademackay

In my experience, this kind of infinite recursion in poetry2nix happens when there is setuptools, setuptools-scm or pip is in poetry.lock.

After doing poetry add setuptools@^61.2.0 geopandas@^0.11.1 pandas@^1.4.3 shapely@^1.8.2 folium@^0.12.1 matplotlib@3.5.2 numpy@^1.23.1 openmatrix@^0.3.5.0 pyyaml@^6.0 dask@2022.9.2 distributed@2022.9.2 tqdm@^4.64.0 scipy@^1.9.1 ipfn@^1.4.4 pyarrow@^9.0.0 jupyter@^1.0.0 notebook@^6.4.12 ipykernel@^6.15.3 keplergl@^0.3.2 sklearn@^0.0 protobuf@3.20.0 matsim-tools@^0.0.14 pygeos@^0.13 dvc@^2.34.2 dvc-s3@^2.21.0, there is matplotlib that has dependency as setuptools-scm in poetry.lock.

One way to be sure that it is causing the infinite recursion is to remove all the setuptools-scm related line in the poetry.lock and check whether infinite recursion occurs. If you erase setuptools-scm in dependency of matplotlib and setuptools-scm itself with file/hash info in poetry.lock, infinite recursion would probably disappear.

Since it is nearly impossible to erase them every time you poetry add a package, try the following flake.nix. Currently this is the best solution I’ve found so far. Any other solution that’s not as dirty as mine would be appreciated!

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    {
      # Nixpkgs overlay providing the application
      overlay = nixpkgs.lib.composeManyExtensions [
        poetry2nix.overlay
        (final: prev: {
          # The application
          myapp = prev.poetry2nix.mkPoetryApplication {
            projectDir = ./.;
          };
          # The env
          myenv = prev.poetry2nix.mkPoetryEnv {
            projectDir = ./.;
          };

          poetry2nix = prev.poetry2nix.overrideScope' (p2nixfinal: p2nixprev: {
            # pyfinal & pyprev refers to python packages
            defaultPoetryOverrides = (p2nixprev.defaultPoetryOverrides.extend (pyfinal: pyprev:
              {
                ### dodge infinite recursion ###
                setuptools = prev.python39Packages.setuptools.override {
                  inherit (pyfinal)
                    bootstrapped-pip
                    pipInstallHook
                    setuptoolsBuildHook
                  ;
                  python = pyfinal.python_selected;
                };

                setuptools-scm = prev.python39Packages.setuptools-scm.override {
                  inherit (pyfinal)
                    packaging
                    typing-extensions
                    tomli
                    setuptools;
                };

                pip = prev.python39Packages.pip.override {
                  inherit (pyfinal)
                    bootstrapped-pip
                    mock
                    scripttest
                    virtualenv
                    pretend
                    pytest
                    pip-tools
                  ;
                };
              }
            ));
          });
        })
      ];
    } // (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlay ];
        };
      in
      {

        packages.default = pkgs.myapp;
        packages.myapp = pkgs.myapp;
        packages.myenv = pkgs.myenv;
        
        devShells.dev = pkgs.mkShell {
          buildInputs = with pkgs; [
            #(python310.withPackages (ps: with ps; [ poetry ]))
            pkgs.myenv
          ];
        };
        
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            (python310.withPackages (ps: with ps; [ poetry ]))
          ];
        };
      }));
}

[jademackay]

@sepiabrown , this looks very promising, and I expect to find many uses for it in future, thank you.

When I run it I encounter

error: attribute 'python_selected' missing

I have performed a few variations on overriding the setuptools python with the effect of un-dodging the infinite recursion error . Have you any thoughts?

[sepiabrown]

@jademackay Oops, try this one.

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    {
      # Nixpkgs overlay providing the application
      overlay = nixpkgs.lib.composeManyExtensions [
        poetry2nix.overlay
        (final: prev: {
          # The application
          myapp = prev.poetry2nix.mkPoetryApplication {
            projectDir = ./.;
          };
          # The env
          myenv = prev.poetry2nix.mkPoetryEnv {
            projectDir = ./.;
          };

          poetry2nix = prev.poetry2nix.overrideScope' (p2nixfinal: p2nixprev: {
            # pyfinal & pyprev refers to python packages
            defaultPoetryOverrides = (p2nixprev.defaultPoetryOverrides.extend (pyfinal: pyprev:
              {
                ### dodge infinite recursion ###
                setuptools = prev.python310Packages.setuptools.override {
                  inherit (pyfinal)
                    bootstrapped-pip
                    pipInstallHook
                    setuptoolsBuildHook
                  ;
                };

                setuptools-scm = prev.python310Packages.setuptools-scm.override {
                  inherit (pyfinal)
                    packaging
                    typing-extensions
                    tomli
                    setuptools;
                };

                pip = prev.python310Packages.pip.override {
                  inherit (pyfinal)
                    bootstrapped-pip
                    mock
                    scripttest
                    virtualenv
                    pretend
                    pytest
                    pip-tools
                  ;
                };
              }
            ));
          });
        })
      ];
    } // (flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ self.overlay ];
        };
      in
      {

        packages.default = pkgs.myapp;
        packages.myapp = pkgs.myapp;
        packages.myenv = pkgs.myenv;
        
        devShells.dev = pkgs.mkShell {
          buildInputs = with pkgs; [
            #(python310.withPackages (ps: with ps; [ poetry ]))
            pkgs.myenv
          ];
        };
        
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            (python310.withPackages (ps: with ps; [ poetry ]))
          ];
        };
      }));
}

[jademackay]

Ah yes, this is one off the variations I tried: remove the python attribute from setuptools and update python39 to python310 throughout. For me this yields error: infinite recursion encountered again. Might we now be in the situation where another package (not setuptools) is responsible for the infinite recursion? Are we looking for the situation where pyproject.toml specifies a package i and a package j, and package j itself also depends on i?

[sepiabrown]

Could you copy & paste poetry.lock and pyproject.toml here?

[jademackay]

I certainly could!

pyproject.toml

poetry.lock

I really appreciate your perseverance on this.

[reckenrode]

This is a flake I use to create a poetry2nix dev shell for beancount, but it doesn’t work with your poetry.lock. I suspect the issue might be related to Infinite recursion · Issue #810 · nix-community/poetry2nix · GitHub.

[sepiabrown]

I found the source of the infinite recursion.

If you experiment with poetry.lock, you can find that dvc depends on dvc-http and dvc-http depends on dvc!

poetry.lock was reproducible on my machine, so poetry.lock is not damaged.

I don’t know how to resolve this kind of issue. Maybe make an issue to the upstream and change the below code which includes dvc under install_requires?

iterative/dvc-http/blob/0b468f5e0b72e745095c3136d45b149d0609a4df/setup.cfg

In the meantime, you would have to manually delete dvc = "*" in [package.dependencies] of dvc-http.

Without dvc in pyproject.toml, the project is free of infinite recursion. (But I haven’t check whether it builds to the end without error.)

[jademackay]

Thanks @sepiabrown , the circular dvc/dvc-http dependency helps explain the infinite recursion and removing dvc-http's dvc = * line from the poetry.lock dodges the infinite recursion. Thanks also for your suggestion to resolve upstream. The demonstration of overridescope' was also very enlightening. I consider the original query resolved. I think the actual poetry project is still a ways from being materialised using poetry2nix but is closer.

[sepiabrown]

Yeah, my build error tells me that isal(python-isal) needed by xopen needed by matsim-tools fails to build because libisal.a cannot be found.

In nixpkgs there is isa-l already packaged but it seems it does’t offer libisal.a (I haven’t checked thoroughly so I may be wrong). If that’s true, then libisal needs to be packaged in nixpkgs!

[Nebucatnetzer]

Containers are now working, however be advised that I hacked it really badly together.
There aren’t that many projects out there which combine, poetry2nix, django and containers.

[jademackay]

Nice, thanks for posting.