fetchPypi cannot find shap on Pypi

Development
1

I am new to creating derivations for Python packages. I am trying to create a derivation for shap (GitHub - slundberg/shap: A game theoretic approach to explain the output of any machine learning model., shap · PyPI) so my derivation begins with

{ stdenv, buildPythonPackage, fetchPypi, pandas, numpy, scipy } :                                                                                                                                                   

buildPythonPackage rec {                                                                                                                                      
    pname = "shap";                                                                                                                                           
    version = "0.29.1";                                                                                                                                       
                                                                                                                                                             
    src = fetchPypi {                                                                                      
      inherit pname version; 
      sha256 = "...";                                                
    }; 
...

The issue is whenever I try to build it I get

building '/nix/store/vqk1nn20lwh2i1jc7shxb0qxmvhbjff1-shap-0.29.1.tar.gz.drv'...

trying https://files.pythonhosted.org/packages/source/s/shap/shap-0.29.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found

trying https://pypi.io/packages/source/s/shap/shap-0.29.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0    622      0 --:--:-- --:--:-- --:--:--   622
100   269  100   269    0     0    634      0 --:--:-- --:--:-- --:--:--   634
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
error: cannot download shap-0.29.1.tar.gz from any mirror

I tried using format = "wheel" but that also failed. I see that the Download files section on Pypi only has windows wheels for shap, does that mean that fetchPypi is not compatible with this package? shap is a popular package used in machine learning. Not having it in nixpkgs is the only reason I still have conda…

2

We have a couple of packages in nixpkgs which are not on pypi, in that case normally we just use

fetchFromGitHub {
  owner = "slundberg";
  repo = "shap" ;
  rev = "git-hash-from-master";
  sha256 = "0ssgvjm0z399l62wkgjk8c75mvhgn5z7g1dkb78r8vrih9428bb8";
};
3

It’s simply because the sources packages hasn’t been uploaded by the author… there are only binary wheels for Windows, see shap · PyPI . You will better download the sources from the github repo by using fetchTarball or fetchGithub.

1 Like
4

Yes, that’s what I ended up doing - fetchFromGitHub worked with no issues. I wonder if the Python section of the Nixpkgs manual should mention the limitations of fetchPypi.

5

What limitation? fetchPypi cannot download a package which isn’t on the package index.

6

@azazel75 The package my question was about - shap - is in pypi index (shap · PyPI, does it not count as being in the index?) but fetchPypi was unable to download it. It is installable with pip. As I mentioned I am new to python packaging and Pypi infrastructure. I assumed that if a package can be installed with pip (i.e. it is listed on Pypi) then fetchPypi should be able to download it. I was clearly wrong. Am I missing anything?

7

Yes, the fact that until yesterday, such package listed packages for installing it only on Windows. If you had tried installing it on macOS or other linux distros with the standard pip the installation would have failed as well. pip usually tries to install the o.s. specific wheel if it exists and falls back to the source distribution, but that also wasn’t there until yesterday. If you look at the available packages now you will see that the author has uploaded a binary package for macOS and the source tarball. If you try now with fetchPypi it will succeed, because now there’s actually a package to download, the source tarball.

1 Like
8

That’s good news, thanks! I guess I am confused as far as the best practices for creating nix derivations for python packages go. If the presence of actual downloads for pypi packages is at the whims of the authors, shouldn’t we always prefer github since most of the packages are there? Can they remove the pypi downloads for older versions thus breaking pinned nixpkg somebody uses for reproducibility?

My prior experience is mostly with R and if a package is available on CRAN it is supposed to be installable on all platforms (provided that non-R dependencies are present and that’s what Nix solves so well). As far as older versions go Microsoft maintains daily snapshots of CRAN called MRAN so they will never disappear. I know nixpkgs uses MRAN for a particular date as a source of R packages.

9

How you understood, it all drills down to how one platform/language deals with the management of the packages that have a binary (and thus O.S. specific) part. The Python history in this regard is that Window’s users mostly aren’t able to compile this packages by themselves (because their platform is one that tries to create dependency to “the company” instead of giving them the tools to be self capable) and there’s no distribution concept they need those packages to be pre-compiled (also the same standardization for the Linux O.S. took much more time to complete because of the more diverse ecosystem). So usually you will find compiled wheel packages for Windows and source tarballs for all the other platforms.

I don’t know how R deals with them.

1 Like