Cant Install Python package from Pypi

mike1 · July 18, 2023, 11:34am

I am new to Nix. I have followed the Wiki under ‘Package unavailable in Nixpkgs’ from here. With the example (in my shell.nix):

 buildPythonPackage rec {
      pname = "deserialize";
      version = "1.8.3";
      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256-0aozmQ4Eb5zL4rtNHSFjEynfObUkYlid1PgMDVmRkwY=";
      };

Which works fine. But when I substitute:

    buildPythonPackage rec {
      pname = "sourcery";
      version = "1.6.0";
      src = fetchPypi {
        inherit pname version;
        sha256 = "sha256-c1c50553de6c5bda05ce49e2b8156a47c75eb2700a02a92fb25ca6cda30a451f";
      };

I get:
nix-shell --show-trace
error:
… while calling the ‘derivationStrict’ builtin

     at //builtin/derivation.nix:9:12: (source not available)

   … while evaluating derivation 'interactive-python3-3.11.4-environment'
     whose name attribute is located at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'nativeBuildInputs' of derivation 'interactive-python3-3.11.4-environment'

     at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:347:7:

      346|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
      347|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
         |       ^
      348|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

   … while evaluating derivation 'python3-3.11.4-env'
     whose name attribute is located at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'passAsFile' of derivation 'python3-3.11.4-env'

     at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/build-support/trivial-builders/default.nix:88:7:

       87|       inherit buildCommand name;
       88|       passAsFile = [ "buildCommand" ]
         |       ^
       89|         ++ (derivationArgs.passAsFile or []);

   … from call site

     at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/development/interpreters/python/wrapper.nix:20:13:

       19|   env = let
       20|     paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
         |             ^
       21|     pythonPath = "${placeholder "out"}/${python.sitePackages}";

   … while calling 'requiredPythonModules'

     at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/development/interpreters/python/python-packages-base.nix:53:27:

       52|   # Get list of required Python modules given a list of derivations.
       53|   requiredPythonModules = drvs: let
         |                           ^
       54|     modules = lib.filter hasPythonModule drvs;

   … while calling anonymous lambda

     at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/lib/lists.nix:658:25:

      657|    */
      658|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
         |                         ^
      659|

   … while evaluating derivation 'python3.11-sourcery-1.6.0'
     whose name attribute is located at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'src' of derivation 'python3.11-sourcery-1.6.0'

     at /home/me/dev/shell.nix:9:7:

        8|       version = "1.6.0";
        9|       src = fetchPypi {
         |       ^
       10|         inherit pname version;

   … while evaluating derivation 'sourcery-1.6.0.tar.gz'
     whose name attribute is located at /nix/store/99669jgbl29z1s6cr3i93m29ab0k55jc-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   error: invalid SRI hash 'c1c50553de6c5bda05ce49e2b8156a47c75eb2700a02a92fb25ca6cda30a451f'

The files downloaded from Pypi are whl files and the hash obviously doesn’t work.
So how do I get the hash for the zip file? Or just get this to work?

zeec123 · July 19, 2023, 9:44am

Probably the hash is wrong. Set it to an empty string and then copy the correct hash after nix complains.

mike1 · July 19, 2023, 10:19am

I tried that and got:

error:
… while calling the ‘derivationStrict’ builtin

     at //builtin/derivation.nix:9:12: (source not available)

   … while evaluating derivation 'interactive-python3-3.11.4-environment'
     whose name attribute is located at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'nativeBuildInputs' of derivation 'interactive-python3-3.11.4-environment'

     at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:347:7:

      346|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
      347|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
         |       ^
      348|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

   … while evaluating derivation 'python3-3.11.4-env'
     whose name attribute is located at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'passAsFile' of derivation 'python3-3.11.4-env'

     at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/build-support/trivial-builders/default.nix:88:7:

       87|       inherit buildCommand name;
       88|       passAsFile = [ "buildCommand" ]
         |       ^
       89|         ++ (derivationArgs.passAsFile or []);

   … from call site

     at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/development/interpreters/python/wrapper.nix:20:13:

       19|   env = let
       20|     paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
         |             ^
       21|     pythonPath = "${placeholder "out"}/${python.sitePackages}";

   … while calling 'requiredPythonModules'

     at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/development/interpreters/python/python-packages-base.nix:53:27:

       52|   # Get list of required Python modules given a list of derivations.
       53|   requiredPythonModules = drvs: let
         |                           ^
       54|     modules = lib.filter hasPythonModule drvs;

   … while calling anonymous lambda

     at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/lib/lists.nix:658:25:

      657|    */
      658|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
         |                         ^
      659|

   … while evaluating derivation 'python3.11-sourcery-1.6.0'
     whose name attribute is located at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   … while evaluating attribute 'src' of derivation 'python3.11-sourcery-1.6.0'

     at /home/me/pathto/shell.nix:9:7:

        8|       version = "1.6.0";
        9|       src = fetchPypi {
         |       ^
       10|         inherit pname version;

   … while evaluating derivation 'sourcery-1.6.0.tar.gz'
     whose name attribute is located at /nix/store/s4x0fspf7s2q06vpv0fhqcmgd1a9kcp9-nixos-23.05/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7

   error: invalid SRI hash '0000000000000000000000000000000000000000000000000000000000000000'

But I dont see the hash.

NobbZ · July 19, 2023, 11:37am

Where did you get this hash from?

That looks as if it was base16 (hex) encoded, though the SRI hashes use some variant of base64 IIRC.

Sadly I do not know, if fetchPypi is one of the fetch-helpers that already got adjustet properly to the “fetcher without hash specified” behaviour or when this happened, nor what commits you base your experiments on.

Therefore my suggestion is that you trie hash = lib.fakeHash; or sha256 = lib.fakeHash respectively.

mike1 · July 19, 2023, 2:34pm

The hash was taken from Pypi for the package whl file.

For the first (hash=lib.fakeHash) I got:
error: multiple hashes passed to fetchurl

For the second I got:
error: attribute ‘sha256’ already defined…

As I said I am new to Nix hence not sure exactly where to put these lines.
I inserted each one just before the line starting sha256 …

NobbZ · July 19, 2023, 2:49pm

The way nix determines this hashes is different from how the pypi.org website does. Whenever you need to tell nix any hashes, you have to use nix-prefetch tooling or acquire it the “TOFU” way:
1. Use lib.fakeHash (or if supported the empty string or leaving of any hash related attributes)
2. build something related
3. copy the correct hash from the error message into the hash attribute.
4. repeat for any missing hash you still have.
Replace the current sha256 = … with one of the lines I gave you, at a time!

edit:

I missed to explain: “TOFU” means “Trust On First Use”, it is not related to soy

mike1 · July 19, 2023, 4:23pm

Using hash = lib.fakeHash;
… some similar lines before this
copying path ‘/nix/store/v2d4a626bb0qybpy9i2jyr01a4kx9hmf-setuptools-setup-hook’ from ‘https://cache.nixos.org’…

trying https://files.pythonhosted.org/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
copying path ‘/nix/store/cqmy36icxz7rvvgnn8bbpgpb4q3vw3vq-pip-install-hook’ from ‘https://cache.nixos.org’…
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
curl: (22) The requested URL returned error: 404

trying https://pypi.io/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 100 122 0 0 1156 0 --:–:-- --:–:-- --:–:-- 1150
100 276 100 276 0 0 793 0 --:–:-- --:–:-- --:–:-- 793
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
curl: (22) The requested URL returned error: 404
error: cannot download sourcery-1.6.0.tar.gz from any mirror
error: builder for ‘/nix/store/jsylzh1ii1d50vhvwki89sfkq2v1anrc-sourcery-1.6.0.tar.gz.drv’ failed with exit code 1;
last 10 log lines:
> curl: (22) The requested URL returned error: 404
>
> trying https://pypi.io/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 122 100 122 0 0 1156 0 --:–:-- --:–:-- --:–:-- 1150
> 100 276 100 276 0 0 793 0 --:–:-- --:–:-- --:–:-- 793
> 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
> curl: (22) The requested URL returned error: 404
> error: cannot download sourcery-1.6.0.tar.gz from any mirror
For full logs, run ‘nix log /nix/store/jsylzh1ii1d50vhvwki89sfkq2v1anrc-sourcery-1.6.0.tar.gz.drv’.
error: 1 dependencies of derivation ‘/nix/store/ln5qa37k7xydglqx5pz2n6h5bdj9r440-python3.11-sourcery-1.6.0.drv’ failed to build
error: 1 dependencies of derivation ‘/nix/store/9d2q17v7lx81nfz2h5rxvjlhp0rqr11w-python3-3.11.4-env.drv’ failed to build

Using sha256 = lib.fakeHash;
these 3 derivations will be built:
/nix/store/c00s95f9zv5bzkl32l57n3m30bl7yn4a-sourcery-1.6.0.tar.gz.drv
/nix/store/ajic54if62m45545l5xhdz25b41g99xl-python3.11-sourcery-1.6.0.drv
/nix/store/k4j8j3v23acqxrx0ai2vl4f9vldnxzm5-python3-3.11.4-env.drv
building ‘/nix/store/c00s95f9zv5bzkl32l57n3m30bl7yn4a-sourcery-1.6.0.tar.gz.drv’…

trying https://files.pythonhosted.org/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
curl: (22) The requested URL returned error: 404

trying https://pypi.io/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 100 122 0 0 630 0 --:–:-- --:–:-- --:–:-- 628
100 276 100 276 0 0 854 0 --:–:-- --:–:-- --:–:-- 854
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
curl: (22) The requested URL returned error: 404
error: cannot download sourcery-1.6.0.tar.gz from any mirror
error: builder for ‘/nix/store/c00s95f9zv5bzkl32l57n3m30bl7yn4a-sourcery-1.6.0.tar.gz.drv’ failed with exit code 1;
last 10 log lines:
> curl: (22) The requested URL returned error: 404
>
> trying https://pypi.io/packages/source/s/sourcery/sourcery-1.6.0.tar.gz
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 122 100 122 0 0 630 0 --:–:-- --:–:-- --:–:-- 628
> 100 276 100 276 0 0 854 0 --:–:-- --:–:-- --:–:-- 854
> 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
> curl: (22) The requested URL returned error: 404
> error: cannot download sourcery-1.6.0.tar.gz from any mirror
For full logs, run ‘nix log /nix/store/c00s95f9zv5bzkl32l57n3m30bl7yn4a-sourcery-1.6.0.tar.gz.drv’.
error: 1 dependencies of derivation ‘/nix/store/ajic54if62m45545l5xhdz25b41g99xl-python3.11-sourcery-1.6.0.drv’ failed to build
error: 1 dependencies of derivation ‘/nix/store/k4j8j3v23acqxrx0ai2vl4f9vldnxzm5-python3-3.11.4-env.drv’ failed to build

The package is available from Releases · sourcery-ai/sourcery · GitHub

zeec123 · July 20, 2023, 5:12am

The python experience on nixos is not very pleasant, but one can get most things running. With the following, you can get the wheel directly:

  buildPythonPackage rec {
    pname = "sourcery";
    version = "1.6.0";
    format = "wheel";

    src = fetchPypi {
      inherit pname version format;
      sha256 = "sha256-wcUFU95sW9oFzkniuBVqR8desnAKAqkvslymzaMKRR8=";
      platform = "manylinux1_x86_64";
    };

    # No tests in archive
    doCheck = false;
  };

nat-418 · July 20, 2023, 8:16am

I am working on the following Python dependencies:

{ pkgs ? import <nixpkgs> {} }:

let
  pyPi = pname: version: sha256: propagatedBuildInputs:
    pkgs.python3Packages.buildPythonPackage {
      pname = pname;
      version = version;
      src = pkgs.python3Packages.fetchPypi {
        inherit pname version;
        sha256 = sha256;
      };
      doCheck = false;
      propagatedBuildInputs = propagatedBuildInputs;
    };
in
{
  generalDependencies = with pkgs; [
    pre-commit
    ruff
    nodePackages.conventional-changelog-cli
  ];

  pythonDependencies = with pkgs.python3Packages; [
    black
    distlib
    distutils_extra
    elasticsearch
    mypy
    pdoc
    ptpython
    types-requests
    types-urllib3
    requests
    uuid
    (pyPi "xdoctest" "1.1.1" "sha256-LqyBMb3N8ngbTlpi1t6H8ES3MMyNuK8UKlG7KcJF53k=" [
      six
    ])
    (pyPi "code2flow" "2.5.1" "sha256-rZhxeSDaZZjaw4oZUfOLhIkfSqS9LzbBGbDbrHKfR4U=" [
      pkgs.graphviz
      pkgs.python3Packages.graphviz
    ])
    (pyPi "awscli" "1.29.5" "sha256-GFQxA6ImhNZ/cT/7LrNjDZLvRalPKKuD52ByGd5RUVQ=" [
      (pyPi "botocore" "1.31.5" "sha256-s1EU2unEUYlaEf7xPXaIHiu1Qo5d6KcCzIWJoo+zTHo=" [
        (pyPi "jmespath" "0.7.1" "sha256-zVoS7j36RwKDoCCjXmnoOwcA1E/kEwFP01rVWExfX9E=" [])
        (pyPi "urllib3" "1.25.4" "sha256-MZzvcjEeUR2Uvhu0eNIC/eSZk10DR6no8NIy3DvOR8Y=" [])
        (pyPi "python-dateutil" "2.1" "sha256-TETsP5/wV7jHtMeL7KX92HEGAOqaHfQvMb/Lri8Fne4=" [
          six
        ])
      ])
      (pyPi "colorama" "0.2.5" "sha256-BiV/876WqRy8bnEh2zWTUpfB7ERFNrNhvJ+k8sooo5Y=" [])
      (pyPi "docutils" "0.16" "sha256-wt46YOnn0Hvia38rAMoDCcIH4GwQD5zCqUkx/HWkePw=" [])
      (pyPi "rsa" "3.1.2" "sha256-ZuuHUqHem5LXZ56g4VVs8uShVRYdACTpfgaZkEHjX1g=" [
        (pyPi "pyasn1" "0.1.3" "sha256-k3/XUcTCXb03gQ0VE4EUZAdzx8J2A5yfbXJ2/ayzVbc=" [])
      ])
      (pyPi "s3transfer" "0.6.0" "sha256-LtB9OGb1I8xWG/SgD8VTWCeYGxF914dvA2sMGspCyUc=" [
        (pyPi "botocore" "1.31.5" "sha256-s1EU2unEUYlaEf7xPXaIHiu1Qo5d6KcCzIWJoo+zTHo=" [
          (pyPi "jmespath" "0.7.1" "sha256-zVoS7j36RwKDoCCjXmnoOwcA1E/kEwFP01rVWExfX9E=" [])
          (pyPi "urllib3" "1.25.4" "sha256-MZzvcjEeUR2Uvhu0eNIC/eSZk10DR6no8NIy3DvOR8Y=" [])
          (pyPi "python-dateutil" "2.1" "sha256-TETsP5/wV7jHtMeL7KX92HEGAOqaHfQvMb/Lri8Fne4=" [
            six
          ])
        ])
      ])
      (pyPi "PyYAML" "6.0" "sha256-aPtRnBQwb+yXIKKltFvJ8MjRuccq30XDe67fzZScNaI=" [])
    ])
  ];
}

But it’s with awscli that I have an error I don’t understand:

this derivation will be built:
  /nix/store/fry91zs4yldgnfr252cpqrma7yw50gkl-python3-3.10.11-env.drv
building '/nix/store/fry91zs4yldgnfr252cpqrma7yw50gkl-python3-3.10.11-env.drv'...
error: collision between `/nix/store/5x83rccvlyw2kck8kyav8ggi824fyp7q-python3.10-docutils-0.16/lib/python3.10/site-packages/docutils/examples.py' and `/nix/store/fhraysjy7q3jgb549b0w9j3c3sg91hh2-python3.10-docutils-0.19/lib/python3.10/site-packages/docutils/examples.py'
error: builder for '/nix/store/fry91zs4yldgnfr252cpqrma7yw50gkl-python3-3.10.11-env.drv' failed with exit code 25;
       last 1 log lines:
       > error: collision between `/nix/store/5x83rccvlyw2kck8kyav8ggi824fyp7q-python3.10-docutils-0.16/lib/python3.10/site-packages/docutils/examples.py' and `/nix/store/fhraysjy7q3jgb549b0w9j3c3sg91hh2-python3.10-docutils-0.19/lib/python3.10/site-packages/docutils/examples.py'
       For full logs, run 'nix-store -l /nix/store/fry91zs4yldgnfr252cpqrma7yw50gkl-python3-3.10.11-env.drv'.

I don’t understand why this collision occurs or how to fix it.

mike1 · July 20, 2023, 10:21am

Hi zeec123,
That at least compiles with:
… lots of line before this
building ‘/nix/store/41r2hh7yg1vrpimd5vzwqq84mrydqw3b-python3-3.11.4-env.drv’…
created 242 symlinks in user environment

Then when I run:
> sourcery review lot.py
Traceback (most recent call last): File "/nix/store/7w925ijpr87k2wl02ahm9bwvf317m6sz-python3.11-sourcery-1.6.0/bin/.sourcery-wrapped", line 9, in <module> sys.exit(main()) ^^^^^^ File "/nix/store/46fw4w2vbiad1n0c07jjb63jmnb1p6ab-python3-3.11.4-env/lib/python3.11/site-packages/sourcery/wrapper.py", line 13, in main return subprocess.call([str(command), *sys.argv[1:]], bufsize=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/d3q49jp0h57dzn4p9rydma2ybgmgazs3-python3-3.11.4/lib/python3.11/subprocess.py", line 389, in call with Popen(*popenargs, **kwargs) as p: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/d3q49jp0h57dzn4p9rydma2ybgmgazs3-python3-3.11.4/lib/python3.11/subprocess.py", line 1026, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/nix/store/d3q49jp0h57dzn4p9rydma2ybgmgazs3-python3-3.11.4/lib/python3.11/subprocess.py", line 1950, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/nix/store/46fw4w2vbiad1n0c07jjb63jmnb1p6ab-python3-3.11.4-env/lib/python3.11/site-packages/sourcery/sourcery'

But if I:
ls -al /nix/store/46fw4w2vbiad1n0c07jjb63jmnb1p6ab-python3-3.11.4-env/lib/python3.11/site-packages/sourcery/sourcery
-r-xr-xr-x 2 root root 131300952 Jan 1 1970 /nix/store/46fw4w2vbiad1n0c07jjb63jmnb1p6ab-python3-3.11.4-env/lib/python3.11/site-packages/sourcery/sourcery

I see that its actually there! So something went wrong in the build I guess.

zeec123 · July 20, 2023, 11:14am

With my snippet, there should be no build at all. I also tested it, and it installed on my machine. (I am on nixpkgs-unstable, but that should make difference.)

mike1 · July 21, 2023, 9:46am

Hi zeec, so whats different to what you did and myself?
I created a shell.nix as per your suggestion.
Then I entered nix-shell at the command line then I got a build output.
Then I ran sourcery review lot.py
Which gave me the error shown above.
Why did you not get a build response? Second time around I got no build response.

Even if I take out the doCheck I get the same error.

Also, changed shell.nix for sourcery_cli and amended sha256, but same result.