Build python application with newer dependencies than in python3Packages

Help
1

I’m trying to create a python derivation of a GitHub project that uses a setup.py.

The issue is that it uses dependencies that are a lot newer than the one’s nix offers. What is the correct way to install them? Is there a best practice?

The only option I can think of is to create derivations for each dependency and sub dependency until I got everything the application requires.

Function that fails:

pkgs.python3Packages.buildPythonApplication rec {
    pname = "tv_time_export";
    version = "1.0.2";

    src = pkgs.fetchFromGitHub {
      owner = "rhoriguchi";
      repo = pname;
      rev = version;
      sha256 = "0k6yk630pilpibpi9jlcq911vyy15wp4gszz934zzarj3rlhannn";
    };
  }

Requirements:

beautifulsoup4==4.9.0
pyyaml==5.3.1
requests==2.23.0
2

The main options for using non-nixpkgs python module versions are Mach-nix: Create python environments quick and easy or poetry2nix, and pypi2nix. You could also make an overlay in your nix-shell and override the python module versions to match the requirements.txt, but I would check out one of the above tools.

1 Like
3

That reminds me that it would be cool if there was a buildPythonApplication equivalent function in mach-nix which automatically extracts requirements from setup.py / setup.cfg and so on. I put it on my list :slight_smile:

4 Likes
4

I usually make overlays using python-package-init for the packages I need. It seems like the fastest way to add the new versions to the default environment.

5

That would be really nice.

This is how I read the values from the setup.py. This only works if you don’t have tests_require.

readRequirementsFromSetupPy = path:
  let
    content = builtins.readFile path;
    lines = lib.splitString "\n" content;
    pattern = ".*'([a-zA-Z0-9]*==[0-9.]*)'.*";
    matches = lib.lists.flatten (builtins.map (builtins.match pattern) lines);
  in 
    builtins.concatStringsSep "\n" (builtins.filter (match: match != null) matches);
6

That’s super cool!
The only problem with regexing the setup.py is that there are too many possibilities which are difficult to cover. Some projects load the requirements from another file or generate them via some strange methods.
I’ve already written some reliable setuptools dependency extractor for the crawlers for mach-nix. It could be reused for buildPythonApplication.

The downside with this extractor: Because it requires some patches for python builtins, it currently needs a rebuild of the whole python interpreter. This would be a bad user experience. Maybe there is a way of patching the python builtin .pyc files without rebuilding the whole thing.

1 Like
7

Since today mach-nix supports automatic requirements extraction. (without rebuilding python).
The example from above can now be built via:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/2.3.0";
  });
in mach-nix.buildPythonPackage "https://github.com/rhoriguchi/tv_time_export/tarball/1.0.2"

will result in: /nix/store/78njdaz5ry8sh21ijw1m4qbms9h304cb-python3.8-tv_time_export-1.0.2

see more examples here: https://github.com/DavHau/mach-nix/blob/master/examples.md#buildpythonpackage–buildpythonapplication

4 Likes
8

Thank you for the info and continue your great work.