### Author:  [@FRidh](https://discourse.nixos.org/u/FRidh)
#### Post date: [April 25, 2020, 9:13am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/6 "2020-04-25T09:13:06Z")
Since PEP 517/518 there is now support for backends other than `setuptools` for _building_ a wheel. This means there are now tools for building wheels from projects that do not have a `setup.py`. Several backends already exist and are in use. Often these backends declare their dependencies somehow, but they don’t have to; it’s really up to the back-end. Thus, the only _reliable_ way for extracting dependencies is from wheels. Clearly one does not always want to build a wheel, so indeed in case of `setuptools` monkey-patching is a popular choice and should work fine.
In the past I did something similar to your [GitHub - DavHau/nix-pypi-fetcher: Pypi Fetcher for Nix with simplified interface. (contains hashes for all packages)](https://github.com/DavHau/nix-pypi-fetcher), essentially downloading the PyPI database into a git repo ([GitHub - FRidh/make-pypi-dump: Extract all the JSON from PyPI](https://github.com/FRidh/make-pypi-dump)) with the idea that could then be used for a second stage that would extract all dependencies. That information would be very useful for other projects as well, if well-described. The crucial part here is to consider dependencies per platform and Python version (or in general the markers), which I see you have done.
> [@doronbehar](#):
>
> It’s interesting to think that other languages’ dependecies websites have learned the lesson I guess, and made it easy to discover the dependency graph. But Python which is super popular, hasn’t.
Graphs with dependency information constructed from source packages won’t ever happen with Python, but forming graphs from wheels is possible.
---
### Author:  [@das-g](https://discourse.nixos.org/u/das-g)
#### Post date: [April 25, 2020, 9:23am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/7 "2020-04-25T09:23:16Z")
> [@FRidh](#):
>
> Since PEP 517/518 there is now support for backends other than `setuptools` for _building_ a wheel.
The mentioned PEPs:
> **[PEP 517 – A build-system independent format for source trees | peps.python.org](https://peps.python.org/pep-0517/)**
>
> While distutils / setuptools have taken us a long way, they suffer from three serious problems: (a) they’re missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like...
> **[PEP 518 – Specifying Minimum Build System Requirements for Python Projects |...](https://peps.python.org/pep-0518/)**
>
> This PEP specifies how Python software packages should specify what build dependencies they have in order to execute their chosen build system. As part of this specification, a new configuration file is introduced for software packages to use to...
---
### Author:  [@doronbehar](https://discourse.nixos.org/u/doronbehar)
#### Post date: [April 25, 2020, 9:48am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/8 "2020-04-25T09:48:11Z")
BTW there’s also:
> **[Libraries - The Open Source Discovery Service](https://libraries.io/api)**
>
> Discover open source libraries, modules and frameworks you can use in your code
---
### Author:  [@das-g](https://discourse.nixos.org/u/das-g)
#### Post date: [April 25, 2020, 11:07am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/9 "2020-04-25T11:07:48Z")
Nice! Instead of
```nix
let pkgs = import { };
in pkgs.mkShell {
buildInputs = let
python-with-pkgs = pkgs.python3.withPackages (ps:
with ps;
[
# Figure out what to put here based on requirements.txt
]);
in [python-with-pkgs];
}
```
I can now put this `shell.nix` alongside `requirements.txt`:
```nix
let
mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix/";
ref = "master";
rev = "fa2bb2d33fb9b9dc3113046e4fcc16088f56981a";
});
in mach-nix.mkPythonShell {
requirements = builtins.readFile ./requirements.txt;
}
```
And it works in conjunction with [lorri](https://discourse.nixos.org/t/introducing-lorri-your-projects-nix-env/2560), too.
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 27, 2020, 11:12am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/10 "2020-04-27T11:12:49Z")
> [@doronbehar](#):
>
> BTW have you used eventually any of these solutions internally:
>
> [stackoverflow.com](https://stackoverflow.com/questions/29751572/how-to-find-a-python-packages-dependencies)
>
> [ ](https://stackoverflow.com/users/247542/cerin)
>
> #### [How to find a Python package’s dependencies](https://stackoverflow.com/questions/29751572/how-to-find-a-python-packages-dependencies)
>
> **python, pip**
>
> asked by [Cerin](https://stackoverflow.com/users/247542/cerin) on [03:05PM - 20 Apr 15 UTC](https://stackoverflow.com/questions/29751572/how-to-find-a-python-packages-dependencies)
>
> ?
I just checked that thread and none of these solutions are similar to what i use. All solutions proposed there require you to do a full installation of the package which is exactly what i wanted to avoid. Installing costs far more computational resources than my current solution which is implemented and explained with more detail in the [pypi-crawlers](https://github.com/DavHau/pypi-crawlers) project.
> [@doronbehar](#):
>
> BTW there’s also:
>
> [Libraries.io](https://libraries.io/api)
>
> ### [Libraries - The Open Source Discovery Service](https://libraries.io/api)
>
> Discover open source libraries, modules and frameworks you can use in your code
Thanks, but i could not find any information about dependencies there. Can you?
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 27, 2020, 11:27am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/11 "2020-04-27T11:27:08Z")
I saw that the actual dump produced by your make-pypi-dump project actually got taken down by github with the notice **Repository unavailable due to DMCA takedown.**
Do you think i need to worry about such complications regarding the data I’m publishing? I guess your data contained wrong information about some projects license and that could have been the problem.
The only thing I’m publishing are project names, their dependency relations, and download URLs.
Could this lead to any trouble of that kind? It would be sad if the project would break because of that. I’d like to make sure to undertake any measures to prevent that.
---
### Author:  [@doronbehar](https://discourse.nixos.org/u/doronbehar)
#### Post date: [April 27, 2020, 11:32am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/12 "2020-04-27T11:32:35Z")
> [@DavHau](#):
>
> > [@doronbehar](#):
> >
> > Discover open source libraries, modules and frameworks you can use in your code
>
> Thanks, but i could not find any information about dependencies there. Can you?
Yes but it seems you still need to crawl them: [API Documentation - Libraries.io - security & maintenance data for open source software](https://libraries.io/api#project-dependencies) . I also wonder whether their [search](https://libraries.io/api#project-search) is capable of giving you _all_ pypi projects there are…
---
### Author:  [@FRidh](https://discourse.nixos.org/u/FRidh)
#### Post date: [April 27, 2020, 11:54am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/13 "2020-04-27T11:54:36Z")
It got taken down because a certain company had uploaded an apparently internal package. Even though they removed it, the package description was still in the PyPI database and thus in my dump.
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 29, 2020, 3:37am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/14 "2020-04-29T03:37:52Z")
Cool, I now played around with their API a little bit. Their general collection of python packages seems quite complete. But concerning the requirements, their data seems to be less complete than mine. First of all, they are not differentiating between **install\_requires** , **setup\_requires** , **tests\_require** , **extras\_require**. Information about markers or differentiation between python versions is also missing. Checking the dependencies for `requests` i noticed that they are missing the requirements `idna` and `urllib3`. Not sure how they mine their data. I also checked for `scipy` for which my crawler failed to extract the requirements. And there they also don’t have the data at all.
---
### Author:  [@FRidh](https://discourse.nixos.org/u/FRidh)
#### Post date: [April 29, 2020, 5:18am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/15 "2020-04-29T05:18:31Z")
As far as I know they collect their data from the wheels. Because wheels are already produced artifacts, it won’t contain `setup_requires` because that’s `setuptools` specific for _building_ the wheel. The same goes for `tests_require`, which upstream still wants to remove, but that issue has stalled.
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 29, 2020, 10:41am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/16 "2020-04-29T10:41:23Z")
I’m currently trying to understand which benefit it will bring to support wheels. For example scipy and tensorflow are currently unsupported by mach-nix. But I’m not sure if wheel support will help.
They both release manylinux{x} wheels. I’ve seen that manylinux wheels are now supported since nixos 20.03. I tested using their wheels with buildPythonPackage. They build, but then fail during import because they link against libstdc++.so.6. This proposal [here](https://github.com/NixOS/nixpkgs/issues/71935) seems to target these linking issues, but it has been closed. Not sure why. If this could be accomplished, it would be a really nice thing for mach-nix.
In general i assume, with the current state of nixpkgs, wheels which include pre-compiled binaries will most likely not work and therefore supporting wheels won’t help for those python packages.
Apart from that, I’m aware that `wheel` is the current distribution standard. Therefore there are libraries out there which only release wheels and no sdist even they don’t contain any binaries. For these libraries wheel support in mach-nix would make them available. It would be interesting to know how high the number of these libraries actually is. I assume it’s low. In nixpkgs19.09 the number of libraries using wheels as an installation method is less than 10. But of course nixpkgs might not be representing the general situation well. Maybe it’s time for another pypi crawling session 😉
All in all, supporting wheels doesn’t sound anymore like that big of an advantage as i originally thought.
Maybe it might be more beneficial to use nixpkgs itself as a provider for the resolver and take packages directly from there. Currently only sdist releases on pypi are considered. That makes tensorflow drop out which would actually be available in nixpkgs and could easily be included. Of course then only that specific version of tensorflow which is in nixpkgs would be available and as soon as you specify anything else in your requirements.txt, it will fail again.
---
### Author:  [@FRidh](https://discourse.nixos.org/u/FRidh)
#### Post date: [April 29, 2020, 1:49pm UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/17 "2020-04-29T13:49:50Z")
> [@DavHau](#):
>
> This proposal [here](https://github.com/NixOS/nixpkgs/issues/71935) seems to target these linking issues, but it has been closed. Not sure why. If this could be accomplished, it would be a really nice thing for mach-nix.
See the [manylinux packages for Python by FRidh · Pull Request #73866 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/pull/73866). Including a manylinux package or list of packages and `autoPatchelfHook` should fix that issue.
> [@DavHau](#):
>
> In general i assume, with the current state of nixpkgs, wheels which include pre-compiled binaries will most likely not work and therefore supporting wheels won’t help for those python packages.
Correct, they need to be patched with this method and additional libraries that are used need to be included as well.
> [@DavHau](#):
>
> Apart from that, I’m aware that `wheel` is the current distribution standard. Therefore there are libraries out there which only release wheels and no sdist even they don’t contain any binaries. For these libraries wheel support in mach-nix would make them available. It would be interesting to know how high the number of these libraries actually is.
Currently that count is very low, but given the new backends that number will go up. New packages I create (though they’re private/work) seldom use `setuptools`. Also, there are already widely used packages that have such packages as dependencies. An example of such a dependency is the `entrypoints` package, which includes a generated `setup.py` for compatibility reasons and is used in packages such as `flake8` and `nbconvert`.
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 29, 2020, 4:41pm UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/18 "2020-04-29T16:41:55Z")
> [@FRidh](#):
>
> See the [https://github.com/NixOS/nixpkgs/pull/73866](https://github.com/NixOS/nixpkgs/pull/73866). Including a manylinux package or list of packages and `autoPatchelfHook` should fix that issue.
Thanks so much! That’s amazing! I now manged to build tensorflow from wheel. And it’s for sure much much faster than building from source. One of the problems of mach-nix currently is that build times can be very long. Should i consider using wheels by default wherever possible? Or are there any troubles ahead I’m not seeing right now?
---
### Author:  [@FRidh](https://discourse.nixos.org/u/FRidh)
#### Post date: [April 29, 2020, 5:11pm UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/19 "2020-04-29T17:11:09Z")
> [@DavHau](#):
>
> Should i consider using wheels by default wherever possible? Or are there any troubles ahead I’m not seeing right now?
As you’ve seen already, wheels are build artifacts, thus they do not list build-time dependencies. That’s no problem if your users are fine using those pre-built wheels. In Nixpkgs we prefer source builds.
---
### Author:  [@DavHau](https://discourse.nixos.org/u/DavHau)
#### Post date: [April 30, 2020, 3:22am UTC](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858/20 "2020-04-30T03:22:41Z")
> [@FRidh](#):
>
> That’s no problem if your users are fine using those pre-built wheels. In Nixpkgs we prefer source builds
Then i should probably let the user decide if using sdist releases should be preferred/enforced.
Of course wheel support might blow up the dataset for the dependency graph. To reduce the size, can i make the following 2 assumptions ?:
1. Since my current dependency extraction uses setup.py and fails on anything else, i know that the build backend for all packages in my current database must be setuptools.
2. If a package’s sdist release uses setuptools as build backend, the requirements specified via `install_requires` are exactly the requirements of the wheel release for this package on pypi.
If this is true, I don’t need to store any dependency information for the wheel release if i already know the dependencies of the sdist.
Also, I would only need to download and analyze wheel’s for packages which either don’t have ‘sdist’ or dependency extraction failed on their ‘sdist’.
[Next page](https://discourse.nixos.org/t/mach-nix-create-python-environments-quick-and-easy/6858.md?page=2)