Mach-nix: Create python environments quick and easy

[DavHau]

Mach-nix makes the process of creating a python environment from a requirements.txt file a no-brainer.

Motivation: When I started using NixOS, I was amazed! (Thanks BTW!) But what I was missing was a quick way to set up a python environment from a requirements.txt file. There are amazing tools like pypi2nix which simplify the task a lot, but the whole experience is still far from perfect. Non-pythonic build inputs are not handled, which often makes it a long manual process of rebuilding stuff over and over again and fixing errors by adding inputs. Compared to a simple pip install -r requirements.txt, this was still quite unpleasant. To gain back the comfort, I started working on my own tool a few months ago. The whole problem turned out to be a bit more complex, but I kept working on it until it did what I wanted. I optimized the UI to be super simple. I’m confident to assume that this could be the perfect tool for nix beginners who quickly need to load a python environment, and also for anyone else because it is just super simple to use.

The main differences to similar projects like pypi2nix are:

• Speed: mach-nix resolves python dependencies via an internal graph and does not need to download and try-build packages first.
• Reproducibility: If you run the same version of mach-nix against the same requirements.txt, it will always produce the same result.
• No tool required: mach-nix can be called directly via nix expression (see below)
• Extra build inputs are handled: Many python packages require non-pythonic build inputs and are difficult to build from scratch. Mach-nix reuses the definitions from nixpkgs to build those packages smoothly

Example usage to create a python derivation:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  });
in
mach-nix.mkPython {
  requirements = ''
    pillow>=7.1.1
    numpy>=1.17
    requests<2.22
  '';
}

Be patient on first use. It needs to fetch the dependency graph which is around ~ 200 MB.

Alternatively, mach-nix can be installed and used as a cmdline tool. Its interface is similar to python’s venv. It is supposed to be newcomer friendly and does not require any knowledge about nix.

For more information please visit the github page: GitHub - DavHau/mach-nix: Create highly reproducible python environments
Always feel free to collaborate!

A big thanks goes to the guys from genesiscloud.com who supported me with the necessary computation resources to generate that massive pypi dependency graph. Check out their service! They offer cheap GPU based cloud computing. Currently 50% off due to beta.

There is still a lot to do to make this tool perfect. Suggestions / ideas / bugs / issues or any other feedback is very welcome!

EDIT: version 2.0.0 is out. changelog . More details in announcement

[doronbehar]

That’s amazing.

I was thinking just this morning on a related concept: Why do we need to write an almost identical buildPythonPackage and buildPythonApplication derivation every time we want a new Python package in Nixpkgs ? Almost always this doesn’t consist of anything specific besides putting the requirements as specified by upstream. Since pypi makes that link by itself anyway, what’s the point in regenerating that data on our end?

While I was merely thinking about it, you’ve done it - you automated the process of linking the dependencies data from pypi and made it available to Nix. Plus, you took one step further - the step of wrapping it nicely in cleaner fetchpypi function. That’s just one benefit this brings.

Perhaps your project, is only an implementation of this idea for Python. Rust has crates.io, Golang has it’s modules system where most of them are documented and (I think) indexed etc.

---

I have some comments / questions:

1. You wrote:

• Extra build inputs are handled: Many python packages require non-pythonic build inputs and are difficult to build from scratch. Mach-nix reuses the definitions from nixpkgs to build those packages smoothly

Just to make sure, if there’s a Python package available on Pypi that has an external dependency and it’s not pacakged in Nixpkgs, Mach-nix isn’t capable of knowing how that external dependency is named in Nixpkgs right?

2. Do you use Pypi’s shas, or do you generate it yourself? It should be available according to this.
3. GitHub - DavHau/pypi-crawlers: Collection of crawlers to mine pypi packages metadata and their dependencies which you link to in the README is 404

[DavHau]

@doronbehar

https://github.com/DavHau/pypi-crawlers which you link to in the README is 404

I forgot to make the pypi-crawlers project public. Thanks for noticing. It should now be available.

@doronbehar

Almost always this doesn’t consist of anything specific besides putting the requirements as specified by upstream. Since pypi makes that link by itself anyway, what’s the point in regenerating that data on our end?

Pypi actually does not provide any information about dependencies. Please correct me if I’m wrong. Because this would make my project a lot simpler. As far as i figured it out, getting information about python dependencies is non-trivial. It was in fact one of the major points to tackle while developing mach-nix.

To sum it up: Dependencies of a python package are only revealed during its installation itself. Regexing the setup.py for definitions like “install_requires =”. or using the abstract syntax tree module failed utterly, since there are too many variants to be considered on how these variables can be defined. Sometimes requrements are loaded from txt file etc. Also, over the years, python introduced additional methods to specify dependencies which are unrelated to setup.py. It’s baically a jungle.

It appeared to me that there is no simple way of mining dependency information without executing the actual installation.

The current strategy of the crawler maintaining the dependency graph is, to run all packages through a nix builder which fake-installs them through a patched python version. That means, it executes each projects setup.py which is untrusted code. Can i trust the nix sandbox? Should i add additional encapsulation layers?

I need to add that I’m only handling sdist python distributions so far and ignore wheels. Using wheels could make this process easier since they might contain some useful extra metadata. I started out handling only sdist because not all python packages on pypi have wheels and also it seemed python packages in nixpkgs are usually built using sdist. I guess that’s for a reason. Only a few of them use wheels. I need to learn more about wheels, so any information regarding this is welcome. I would like to support wheels in the future since some projects like tensorflow only release wheels.

@doronbehar

Just to make sure, if there’s a Python package available on Pypi that has an external dependency and it’s not pacakged in Nixpkgs, Mach-nix isn’t capable of knowing how that external dependency is named in Nixpkgs right?

That is correct. A package which requires external dependencies and is itself not specified in nixpkgs will fail. During this project I already started working on a tool with the goal to build a mapping from ‘build error messages’ to nix package attributes to build a database for missing inputs. But it apppeared that really nearly all of these difficult-to-build python packages are already packaged in nixpkgs. Therefore I trashed that project again and decided to better just rely on nixpkgs as a general base. In case my filter bubble view is wrong and external dependencies will still be a problem for many people, we could start working on a solution for that. But currently I have the feeling that other things would gain more benefits, like adding support for wheels for example.

@doronbehar

Do you use Pypi’s shas, or do you generate it yourself? It should be available according to this

Indirectly I’m using the hashes coming from pypi. But as mach-nix itself is running as a nix builder i cannot and should not do arbitrary api request’s since we cannot trust the integrety of the data coming from pypi. Therefore I built my own pypi fetcher tool nix-pypi-fetcher. It includes a mapping from (pkg_name, pkg_version) to URL and sha256. The mapping is updated twice a day (by querying pypi). Mach-nix pins one specific version of that fetcher together with one specific version of the dependency graph.

[doronbehar]

@DavHau

Also, over the years, python introduced additional methods to specify dependencies which are unrelated to setup.py . It’s baically a jungle.

It’s interesting to think that other languages’ dependecies websites have learned the lesson I guess, and made it easy to discover the dependency graph. But Python which is super popular, hasn’t.

Reading your explanation, I realise how much work you’ve done. Well done .

[doronbehar]

BTW have you used eventually any of these solutions internally:

How to find a Python package's dependencies

?

[FRidh]

Since PEP 517/518 there is now support for backends other than setuptools for building a wheel. This means there are now tools for building wheels from projects that do not have a setup.py. Several backends already exist and are in use. Often these backends declare their dependencies somehow, but they don’t have to; it’s really up to the back-end. Thus, the only reliable way for extracting dependencies is from wheels. Clearly one does not always want to build a wheel, so indeed in case of setuptools monkey-patching is a popular choice and should work fine.

In the past I did something similar to your GitHub - DavHau/nix-pypi-fetcher: Pypi Fetcher for Nix with simplified interface. (contains hashes for all packages), essentially downloading the PyPI database into a git repo (GitHub - FRidh/make-pypi-dump: Extract all the JSON from PyPI) with the idea that could then be used for a second stage that would extract all dependencies. That information would be very useful for other projects as well, if well-described. The crucial part here is to consider dependencies per platform and Python version (or in general the markers), which I see you have done.

@doronbehar

It’s interesting to think that other languages’ dependecies websites have learned the lesson I guess, and made it easy to discover the dependency graph. But Python which is super popular, hasn’t.

Graphs with dependency information constructed from source packages won’t ever happen with Python, but forming graphs from wheels is possible.

[das-g]

@FRidh

Since PEP 517/518 there is now support for backends other than setuptools for building a wheel.

The mentioned PEPs:

PEP 517 – A build-system independent format for source trees | peps.python.org

PEP 518 – Specifying Minimum Build System Requirements for Python Projects |...

[doronbehar]

BTW there’s also:

Libraries - The Open Source Discovery Service

[das-g]

Nice! Instead of

let pkgs = import <nixpkgs> { };
in pkgs.mkShell {
  buildInputs = let
    python-with-pkgs = pkgs.python3.withPackages (ps:
      with ps;
      [
        # Figure out what to put here based on requirements.txt
      ]);
  in [ python-with-pkgs ];
}

I can now put this shell.nix alongside requirements.txt:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "master";
    rev = "fa2bb2d33fb9b9dc3113046e4fcc16088f56981a";
  });
in mach-nix.mkPythonShell {
  requirements = builtins.readFile ./requirements.txt;
}

And it works in conjunction with lorri, too.

[DavHau]

@doronbehar

BTW have you used eventually any of these solutions internally:

stackoverflow.com

How to find a Python package’s dependencies

python, pip

asked by Cerin on 03:05PM - 20 Apr 15 UTC

?

I just checked that thread and none of these solutions are similar to what i use. All solutions proposed there require you to do a full installation of the package which is exactly what i wanted to avoid. Installing costs far more computational resources than my current solution which is implemented and explained with more detail in the pypi-crawlers project.

@doronbehar

BTW there’s also:

Libraries.io

Libraries - The Open Source Discovery Service

Discover open source libraries, modules and frameworks you can use in your code

Thanks, but i could not find any information about dependencies there. Can you?

[DavHau]

I saw that the actual dump produced by your make-pypi-dump project actually got taken down by github with the notice Repository unavailable due to DMCA takedown.

Do you think i need to worry about such complications regarding the data I’m publishing? I guess your data contained wrong information about some projects license and that could have been the problem.

The only thing I’m publishing are project names, their dependency relations, and download URLs.
Could this lead to any trouble of that kind? It would be sad if the project would break because of that. I’d like to make sure to undertake any measures to prevent that.

[doronbehar]

@DavHau

@doronbehar

Discover open source libraries, modules and frameworks you can use in your code

Thanks, but i could not find any information about dependencies there. Can you?

Yes but it seems you still need to crawl them: API Documentation - Libraries.io . I also wonder whether their search is capable of giving you all pypi projects there are…

[FRidh]

It got taken down because a certain company had uploaded an apparently internal package. Even though they removed it, the package description was still in the PyPI database and thus in my dump.

[DavHau]

Cool, I now played around with their API a little bit. Their general collection of python packages seems quite complete. But concerning the requirements, their data seems to be less complete than mine. First of all, they are not differentiating between install_requires, setup_requires, tests_require, extras_require. Information about markers or differentiation between python versions is also missing. Checking the dependencies for requests i noticed that they are missing the requirements idna and urllib3. Not sure how they mine their data. I also checked for scipy for which my crawler failed to extract the requirements. And there they also don’t have the data at all.

[FRidh]

As far as I know they collect their data from the wheels. Because wheels are already produced artifacts, it won’t contain setup_requires because that’s setuptools specific for building the wheel. The same goes for tests_require, which upstream still wants to remove, but that issue has stalled.

[DavHau]

I’m currently trying to understand which benefit it will bring to support wheels. For example scipy and tensorflow are currently unsupported by mach-nix. But I’m not sure if wheel support will help.
They both release manylinux{x} wheels. I’ve seen that manylinux wheels are now supported since nixos 20.03. I tested using their wheels with buildPythonPackage. They build, but then fail during import because they link against libstdc++.so.6. This proposal here seems to target these linking issues, but it has been closed. Not sure why. If this could be accomplished, it would be a really nice thing for mach-nix.

In general i assume, with the current state of nixpkgs, wheels which include pre-compiled binaries will most likely not work and therefore supporting wheels won’t help for those python packages.

Apart from that, I’m aware that wheel is the current distribution standard. Therefore there are libraries out there which only release wheels and no sdist even they don’t contain any binaries. For these libraries wheel support in mach-nix would make them available. It would be interesting to know how high the number of these libraries actually is. I assume it’s low. In nixpkgs19.09 the number of libraries using wheels as an installation method is less than 10. But of course nixpkgs might not be representing the general situation well. Maybe it’s time for another pypi crawling session

All in all, supporting wheels doesn’t sound anymore like that big of an advantage as i originally thought.
Maybe it might be more beneficial to use nixpkgs itself as a provider for the resolver and take packages directly from there. Currently only sdist releases on pypi are considered. That makes tensorflow drop out which would actually be available in nixpkgs and could easily be included. Of course then only that specific version of tensorflow which is in nixpkgs would be available and as soon as you specify anything else in your requirements.txt, it will fail again.

[FRidh]

@DavHau

This proposal here seems to target these linking issues, but it has been closed. Not sure why. If this could be accomplished, it would be a really nice thing for mach-nix.

See the manylinux packages for Python by FRidh · Pull Request #73866 · NixOS/nixpkgs · GitHub. Including a manylinux package or list of packages and autoPatchelfHook should fix that issue.

@DavHau

In general i assume, with the current state of nixpkgs, wheels which include pre-compiled binaries will most likely not work and therefore supporting wheels won’t help for those python packages.

Correct, they need to be patched with this method and additional libraries that are used need to be included as well.

@DavHau

Apart from that, I’m aware that wheel is the current distribution standard. Therefore there are libraries out there which only release wheels and no sdist even they don’t contain any binaries. For these libraries wheel support in mach-nix would make them available. It would be interesting to know how high the number of these libraries actually is.

Currently that count is very low, but given the new backends that number will go up. New packages I create (though they’re private/work) seldom use setuptools. Also, there are already widely used packages that have such packages as dependencies. An example of such a dependency is the entrypoints package, which includes a generated setup.py for compatibility reasons and is used in packages such as flake8 and nbconvert.

[DavHau]

@FRidh

See the https://github.com/NixOS/nixpkgs/pull/73866 . Including a manylinux package or list of packages and autoPatchelfHook should fix that issue.

Thanks so much! That’s amazing! I now manged to build tensorflow from wheel. And it’s for sure much much faster than building from source. One of the problems of mach-nix currently is that build times can be very long. Should i consider using wheels by default wherever possible? Or are there any troubles ahead I’m not seeing right now?

[FRidh]

@DavHau

Should i consider using wheels by default wherever possible? Or are there any troubles ahead I’m not seeing right now?

As you’ve seen already, wheels are build artifacts, thus they do not list build-time dependencies. That’s no problem if your users are fine using those pre-built wheels. In Nixpkgs we prefer source builds.

[DavHau]

@FRidh

That’s no problem if your users are fine using those pre-built wheels. In Nixpkgs we prefer source builds

Then i should probably let the user decide if using sdist releases should be preferred/enforced.

Of course wheel support might blow up the dataset for the dependency graph. To reduce the size, can i make the following 2 assumptions ?:

1. Since my current dependency extraction uses setup.py and fails on anything else, i know that the build backend for all packages in my current database must be setuptools.
2. If a package’s sdist release uses setuptools as build backend, the requirements specified via install_requires are exactly the requirements of the wheel release for this package on pypi.

If this is true, I don’t need to store any dependency information for the wheel release if i already know the dependencies of the sdist.
Also, I would only need to download and analyze wheel’s for packages which either don’t have ‘sdist’ or dependency extraction failed on their ‘sdist’.