Python: Torch + Poetry doesn't work, but Torch does

Hi,

If I use this slightly obtuse command line, I get a Python with Torch:

nix develop --impure --expr "with import <nixpkgs> {}; pkgs.mkShell { packages = [python3 python3Packages.torch ];}"

If I wanted to package this with Python in a proper way that should integrate nicely with Nix, I think I should use Poetry. So I make a directory, do nix shell nixpkgs/23.05#poetry and then do poetry init. It interviews me, but the only thing I do of interest is add torch==2.0.1 when it asks me for project dependencies. Then I run poetry lock to pin the dependency. Then I ran nix flake init -t poetry2nix#app and modify the generated flake to get this:

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/23.05";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        inherit (poetry2nix.legacyPackages.${system}) mkPoetryApplication;
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        packages = {
          myapp = mkPoetryApplication { projectDir = self; };
          default = self.packages.${system}.myapp;
        };

        devShells.default = pkgs.mkShell {
          packages = [ poetry2nix.packages.${system}.poetry ];
        };
      });
}

Seems innocuous. However, nix shell fails with the following error:

python3.10-torch-test>     File "/nix/store/r6vfjn6isavbqvnnmfvwdfy9anv8cgd2-python3.10-poetry-core-1.4.0/lib/python3.10/site-packages/poetry/core/masonry/utils/package_include.py", line 66, in check_elements
python3.10-torch-test>       raise ValueError(
python3.10-torch-test>   ValueError: /private/tmp/nix-build-python3.10-torch-test-0.1.0.drv-0/source/torch_test does not contain any element
python3.10-torch-test>   error: subprocess-exited-with-error
python3.10-torch-test>   × Preparing metadata (pyproject.toml) did not run successfully.
python3.10-torch-test>   │ exit code: 1
python3.10-torch-test>   ╰─> See above for output.
python3.10-torch-test>   note: This error originates from a subprocess, and is likely not a problem with pip.
python3.10-torch-test>   full command: /nix/store/xmxc0xc03fqq8b19sxfxwyd8k71kaz5m-python3-3.10.11/bin/python3.10 /nix/store/9zqasc9xrs5nlgv6wqfvkbah7gdfcvlg-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /private/tmp/nix-build-python3.10-torch-test-0.1.0.drv-0/tmp74sy6atb
python3.10-torch-test>   cwd: /private/tmp/nix-build-python3.10-torch-test-0.1.0.drv-0/source
python3.10-torch-test>   Preparing metadata (pyproject.toml) ... error
python3.10-torch-test> error: metadata-generation-failed
python3.10-torch-test> × Encountered error while generating package metadata.
python3.10-torch-test> ╰─> See above for output.
python3.10-torch-test> note: This is an issue with the package mentioned above, not pip.
python3.10-torch-test> hint: See above for details.
error: builder for '/nix/store/003ysq6zxk7jmvvgs6id7lrhhly63x5m-python3.10-torch-test-0.1.0.drv' failed with exit code 1;

My sense is that if I could tell Nix to use the NixPkgs version of Torch, all would be well. But I am confused as to why I need to. What is the right way to proceed here, assuming I want a Python development environment and a Poetry package, and I’d rather the two not differ meaningfully?

Thanks!

It was suggested to me on the Element chat that I try using overrides, so I wound up modifying my configuration like so:

        packages = {
          myapp = mkPoetryApplication {
            projectDir = self;
            overrides = pkgs.poetry2nix.overrides.withDefaults (final: prev: {
              torch = pkgs.python3Packages.torch;
            });
          };
          default = self.packages.${system}.myapp;
        };

This fails because it cannot find pyyaml:

python3.10-torch> Building wheel torch-2.0.1
python3.10-torch> -- Building version 2.0.1
python3.10-torch> Traceback (most recent call last):
python3.10-torch>   File "/private/tmp/nix-build-python3.10-torch-2.0.1.drv-1/source/setup.py", line 451, in check_pydep
python3.10-torch>     importlib.import_module(importname)
python3.10-torch>   File "/nix/store/xmxc0xc03fqq8b19sxfxwyd8k71kaz5m-python3-3.10.11/lib/python3.10/importlib/__init__.py", line 126, in import_module
python3.10-torch>     return _bootstrap._gcd_import(name[level:], package, level)
python3.10-torch>   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
python3.10-torch>   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
python3.10-torch>   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
python3.10-torch> ModuleNotFoundError: No module named 'yaml'
python3.10-torch> The above exception was the direct cause of the following exception:
python3.10-torch> Traceback (most recent call last):
python3.10-torch>   File "/private/tmp/nix-build-python3.10-torch-2.0.1.drv-1/source/setup.py", line 1276, in <module>
python3.10-torch>     main()
python3.10-torch>   File "/private/tmp/nix-build-python3.10-torch-2.0.1.drv-1/source/setup.py", line 1047, in main
python3.10-torch>     build_deps()
python3.10-torch>   File "/private/tmp/nix-build-python3.10-torch-2.0.1.drv-1/source/setup.py", line 402, in build_deps
python3.10-torch>     check_pydep('yaml', 'pyyaml')
python3.10-torch>   File "/private/tmp/nix-build-python3.10-torch-2.0.1.drv-1/source/setup.py", line 453, in check_pydep
python3.10-torch>     raise RuntimeError(missing_pydep.format(importname=importname, module=module)) from e
python3.10-torch> RuntimeError: Missing build dependency: Unable to `import yaml`.
python3.10-torch> Please install it via `conda install pyyaml` or `pip install pyyaml`
python3.10-torch> /nix/store/crv0aygayvds8xwyqln2p1zrdsmfw54s-stdenv-darwin/setup: line 145: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/sii1z2ld30a44cnsj4wq7zqzimcc9kbc-python3.10-torch-2.0.1.drv' failed with exit code 1;

This seems really strange because it’s listed in the propagatedBuildInputs of the original. I added it manually to the buildInputs and that seemed to help it proceed, but then it failed on another dependency, so I eventually changed the line to this:

        packages = {
          myapp = mkPoetryApplication {
            projectDir = self;
            overrides = pkgs.poetry2nix.overrides.withDefaults (final: prev: {
              torch = pkgs.python3Packages.torch.overridePythonAttrs (old: {
                buildInputs = (old.buildInputs or []) ++ old.propagatedBuildInputs;
              });
            });
          };
          default = self.packages.${system}.myapp;
        };

This failed in an interesting way, with this:

python3.10-torch> pythonCatchConflictsPhase
python3.10-torch> Found duplicated packages in closure for dependency 'numpy':
python3.10-torch>   numpy 1.24.2 (/nix/store/4pydxvq4ck7bg21akgp9kq7p7rdlhrlr-python3.10-numpy-1.24.2/lib/python3.10/site-packages)
python3.10-torch>   numpy 1.24.2 (/nix/store/iclkprvy7lwjkclsdhpzjw4pq4svkkm0-python3.10-numpy-1.24.2/lib/python3.10/site-packages)
python3.10-torch> Found duplicated packages in closure for dependency 'typing-extensions':
python3.10-torch>   typing-extensions 4.5.0 (/nix/store/sxs91g02wl32c28j07bznmz8mqsy4bkz-python3.10-typing-extensions-4.5.0/lib/python3.10/site-packages)
python3.10-torch>   typing-extensions 4.7.1 (/nix/store/qi834gz9z34skh2afz1byw6saizhmppf-python3.10-typing-extensions-4.7.1/lib/python3.10/site-packages)
python3.10-torch> Found duplicated packages in closure for dependency 'future':
python3.10-torch>   future 0.18.3 (/nix/store/cdcs4nvh56g8h6spy5q3z7szax8kx02s-python3.10-future-0.18.3/lib/python3.10/site-packages)
python3.10-torch>   future 0.18.3 (/nix/store/3m1hky8fnq6giqr5gfiqf2g9di1b94qf-python3.10-future-0.18.3/lib/python3.10/site-packages)
python3.10-torch> Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
error: builder for '/nix/store/rrjjcm2g8lyqci9z1zh8cbmvj9812vf8-python3.10-torch-2.0.1.drv' failed with exit code 1;

Frankly, that seems accurate to me, it is probably getting multiple on the path. I’m not really sure why overriding the poetry-detected Torch with the NixPkgs one isn’t building.

I’m starting to wonder if the problem here has to do with Torch itself.

I have solved the issue, and it amounted to the fact that the Poetry package was misconfigured (missing README.md and init.py). Sorry for the noise!

Was it still required to override the torch dependency with the version from nixpkgs?

No, actually it wasn’t! User error entirely.