Python envrionment with pkgs.python.withPackages + mach.Python

Help
1

Why does this not work? Why can’t combine packages?

pyEnv = (pkgs.${python}.withPackages
          (ps: with ps; [
            rasterio
          ]));

        pyDevEnv = mach.mkPython {
          requirements = ''
            xarray
            dask
            s3fs
            numpy
            pandas
            typer
          '';
        };
      in
      {
        devShells.default = pkgs.mkShellNoCC {
          packages = with pkgs; [
            pyDevEnv
            pyEnv
          ];
        };
      }

if I enter the shell pyDevEnv packages are found but not pyEnv if change the order it’s vice versa - pyEnv packages are found but not pyDevEnv.

2

That’s because those are two independent python “instances”. Try it out, swap them around and use command -v python on both. You’ll see that they are in different directories.

When you put a package in the packages of a devshell, nix will put their “bin” directory in your $PATH when you run nix develop. So depending on the order, one python will override the other - you can’t have two different pythons available with the same command.

You’ll probably need to put all the packages in the same python instance. Why do you want to use one from nixpkgs and all the others from mach-nix?

Alternatively you could put python3Packages.rasterio in the devshell’s packages, I think that will add it to $PYTHONPATH and should then just work with your other python env, but I’m not 100% sure.

1 Like
3

Thanks for the clarification! I was not aware that they’re independent python instances.

The reason is that I had issues installing rasterio and other dependencies with `mach-nix:

ERROR: Could not find a version that satisfies the requirement pytest>=2.6.0 (from pytest-env) (from versions: none)
       > ERROR: No matching distribution found for pytest>=2.6.0

However I found the package on nixpkgs and was able to build it. Your suggestion worked, thanks!

4

Actually, I take it back. Seems this:

        devShells.default = pkgs.mkShellNoCC {
          packages = with pkgs; [
            gdal
            python3Packages.rasterio
            pyDevEnv
          ];
        };

creates also 2 independent python instances.

$ command -v python
/nix/store/w4agbx5b17d57jij2i4klv67r8qnl6g3-python3-3.10.7/bin/python
5

Fair enough, I thought so, but the documentation suggested otherwise.

Try this:

This will make mach-nix use the nixpkgs version for rasterio, see the docs: GitHub - DavHau/mach-nix: Create highly reproducible python environments

Though I thought this was the default behavior. If that still doesn’t work, make sure you set inputs.mach-nix.inputs.nixpkgs.follows = "nixpkgs", so that mach-nix uses the same version of nixpkgs as you are.

6

Thanks! I was not aware I could specify this! In the end I have ditched mach-mix and just used native python virtual env.

7

Heh, fair enough, maybe for a future attempt :wink:

1 Like