Updating python project's nix-shell requirements.nix

Guides
1

Edit: This could probably use input from someone with more perspective on how the Python packaging update has and may continue to shake out. The changes in my first post got it to build, but ran into Python pkg_resources import errors while running tests this morning. Getting it running makes me suspect this is a moving target, and that there’s a good chance my changes are incomplete or miss the forest for the trees.

Today I updated my unstable channel for the first time in a bit to get a newer chromedriver and found that a requirements.nix expression I use (generated by pypi2nix 1.8.1, and significantly hand-edited) was no longer working.

It took me a bit to track down changes and fiddle/debug until this worked, so I figured I’d share in case it gets indexed and saves someone else the same work…

I got an error like:

error: attribute ‘patchPhase’ missing, at /Users/abathur/…/requirements.nix :25:24

Which I fixed by following what pypi2nix did in Do not patch bootstrapped-pip in generated requirements.nix by seppeljordan · Pull Request #323 · nix-community/pypi2nix · GitHub, which amounts to something like:

--- oldrequirements.nix	2019-10-23 22:19:11.837634932 -0500
+++ requirements.nix	2019-10-23 22:18:17.201316313 -0500
@@ -18,15 +18,6 @@
     inherit pkgs;
     inherit (pkgs) stdenv;
     python = pkgs.python36;
-    # patching pip so it does not try to remove files when running nix-shell
-    overrides =
-      self: super: {
-        bootstrapped-pip = super.bootstrapped-pip.overrideDerivation (old: {
-          patchPhase = old.patchPhase + ''
-            sed -i               -e "s|paths_to_remove.remove(auto_confirm)|#paths_to_remove.remove(auto_confirm)|"                -e "s|self.uninstalled = paths_to_remove|#self.uninstalled = paths_to_remove|"                  $out/${pkgs.python36.sitePackages}/pip/req/req_install.py
-          '';
-        });
-      };
   };

and subsequently found I needed to track the changes in Autodetect setup requirements for PEP 517 packages by seppeljordan · Pull Request #273 · nix-community/pypi2nix · GitHub, like:

     in {
       __old = pythonPackages;
       inherit interpreter;
-      mkDerivation = pythonPackages.buildPythonPackage;
+      mkDerivation = args: pythonPackages.buildPythonPackage (args // {
+        nativeBuildInputs = (args.nativeBuildInputs or []) ++ args.buildInputs;
+      });
       packages = pkgs;
       overrideDerivation = drv: f:
         pythonPackages.buildPythonPackage (drv.drvAttrs // f drv.drvAttrs //                                            { meta = drv.meta; });
@@ -217,7 +210,7 @@

By this point I assumed I’d need to track most upstream pypi2nix changes to its own requirements.nix file, but adopting the changes in Update dependencies by seppeljordan · Pull Request #346 · nix-community/pypi2nix · GitHub caused trouble with multiple active setuptools versions which manifested like:

pythonCatchConflictsPhase
Processing ./setuptools-41.4.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/nix/store/pqz2jb2y8sfz7m0xb4kgw705im1in86a-python3.6-setuptools-41.2.0/lib/python3.6/site-packages/setuptools-41.2.0-py3.6.egg'
Consider using the `--user` option or check the permissions.

I won’t show a diff for these changes, since they depend a lot on what packages are in your requirements.nix. I wonder if this just comes down to the lag between nixpkgs master and unstable, but in any case my short-run fix (which is working for now…) is to just comment out the derivation for setuptools 41.4.0, and all of the self."setuptools" buildInputs references to it.

I also had to make some smaller changes for specific dependencies:

-      installFlags = ["--no-deps"];
+      pipInstallFlags = ["--no-deps"];

HTH

1 Like
2

Hi,

maintainer of pypi2nix here. Though pypi2nix is under active development, the nixpkgs package of it is quite old (2 years). This will probably change very soon since I am planning to release pypi2nix to nixpkgs. In those two years some things in the python infrastructure on nixpkgs changed, like how additional flags for pip are handled or which version of pip is used. Also pips capabilities changed during this time.

This is why I would recommend you try out the current master version of pypi2nix nix. The master branch is tested with every commit and has been quite stable over the last weeks, also pypi2nix has more features now, like automatic detection of build-only dependencies in most cases.

In case you try to generate a new version of your requirements.nix file I would like to see how it went. And if you run into problems, I would appreciate if you opened an issue on github. I could assist you in case of problems and also it would give me the opportunity to improve the UX or fix some bugs based on your feedback.

Cheers,
Sebastian

1 Like