Dependencies (hell) among Python packages

Help
1

Hi,
I am trying to create a development shell in order to compile the Python package for LTX-Video. This is compiled from github like so,

# ltx-video.nix
{ pkgs, einops }:

pkgs.python312Packages.buildPythonPackage rec {
  pname = "ltx-video";
  version = "";
  pyproject = true;

  src = pkgs.fetchFromGitHub {
    owner = "Lightricks";
    repo = "LTX-Video";
    rev = "ltx-video-0.9.1";
    hash = "sha256-jHLoyAhZuMJxoY3bqCIsp1QSMo4IF4M6Fe6I5LVQtkA=";
  };
  nativeBuildInputs = [
    pkgs.python312
  ];
  propagatedBuildInputs = [
    pkgs.python312Packages.setuptools
    pkgs.python312Packages.torch
    pkgs.python312Packages.diffusers
    pkgs.python312Packages.transformers
    pkgs.python312Packages.sentencepiece
    pkgs.python312Packages.huggingface-hub
    einops
  ];
  doCheck = false;
  pythonImportsCheck = [ "ltx-video" ];
}

Here, einops is not contained in nixpkgs, and so I have for my development shell

let
  einops           = import ./einops.nix          { inherit pkgs; }; 
  ltx-video        = import ./ltx-video.nix       { inherit pkgs; inherit einops; }; 
in
pkgs.mkShell rec {
  nativeBuildInputs = with pkgs.buildPackages; [
    einops
    python312Packages.numpy
    [...]
    ltx-video
  ];
}

This fails to build because LTX-Video requires a specific version of huggingface-hub (0.25.2), but python312Packages.huggingface-hub provides 0.26.2.

Now 0.25.2 seems nowhere to be found in nixpkgs, but it’s on PyPi, and so I can build it like so

# huggingface-hub.nix
{ pkgs }:

pkgs.python312Packages.buildPythonPackage rec {
  pname = "huggingface_hub";
  version = "0.25.2";
  pyproject = true;

  src = pkgs.python312Packages.fetchPypi {
    inherit pname;
    inherit version;
    hash = "sha256-oQFOoRGl9AzNI/f3uorEbiD6O2WM7R+GoAx1wG7GQjw=";
  };
  nativeBuildInputs = [
    pkgs.python312
    pkgs.python312Packages.setuptools
  ];
  propagatedBuildInputs = [
    pkgs.python312Packages.filelock
    pkgs.python312Packages.fsspec
    pkgs.python312Packages.pyyaml
    pkgs.python312Packages.requests
    pkgs.python312Packages.tqdm
    pkgs.python312Packages.typing-extensions
  ];
  doCheck = false;
  pythonImportsCheck = [ "huggingface_hub" ];
}

and add my own huggingface-hub.nix at the same level as einops.nix in my development shell. The shell correctly provides a python with 0.25.2 of huggingface-hub, but I end up in hell in another place:

The package ltx-video depends on transformers and diffusers and each of them further depends on huggingface-hub, but of course take it from python312Packages and so use 0.26.2. That’s fine. But the buildPythonPackage when trying to build my LTX-Video, seems to see both 0.25.2 (my own) and 0.26.2 (from the other packages) of huggingface-hub, seems to prefer 0.26.2 and then fails because it is not 0.25.2.

I can imagine two solutions, but I don’t know how to do it in nix:

  1. Tell both transformers and diffusers to use my own version of huggingface-hub instead of the version from python312Packages. This might be doable by an overlay. Can someone help?

  2. Tell the build procedure of LTX-Video to prefer the version 0.25.2 of huggingface-hub that is provided explicitly by me and to ignore any other version of it that might come through from the other required packges.

Any help would be much appreciated.

2

You can try pinning a specific version of Nixpkgs that has all versions your need. For example, by checking out History for pkgs/development/python-modules/huggingface-hub/default.nix - NixOS/nixpkgs · GitHub, you might want to start from some commit like 627b07c.
Also this topic would be a better fit in the Help category.

3

Thanks, but this does not help b/c the version 0.25.2 of huggingface-hub that I need for LTX-Video, seems not to be contained in nixpkgs anywhere.

https://lazamar.co.uk/nix-versions/?channel=nixos-24.05&package=python312Packages.huggingface-hub

4

Actually this is why I suggest you look at file history. Check this commit: python312Packages.huggingface-hub: 0.25.1 -> 0.25.2 · NixOS/nixpkgs@180386b · GitHub.

5

… I just understood you after the second remark :joy: - you are right, this is a viable strategy.

I now use two versions of nixpkgs, the usual one and a specific one for ltx-video. I also found the required version of huggingface-hub in nixpkgs (git is your friend). The only downside is that the two relevant versions use different versions of Python itself, and I managed to build LTX-Video only if all its requirements use the same version of Python. This in turn leads to a duplication of torch (compile time 2hrs+), but never mind, it is up and running now.

Can I keep asking? IMO the following stategy would be more efficient:

I use the huggingface-hub.nix included above which compiles my desired version of the package. Then I am trying an override in order to force tokenizers to use this version (rather than the default):

let
  einops = import ./einops.nix { inherit pkgs; }; 
  huggingface-hub2 = import ./huggingface-hub.nix { inherit pkgs; }; # specific version
in
pkgs.mkShell rec {
  nativeBuildInputs = with pkgs.buildPackages; [
    [...]
    einops
    python312Packages.numpy
    python312Packages.tokenizers.override {
      huggingface-hub = huggingface-hub2;
    }
  ];
}

My hope was that the function defined in

contains huggingface-hub as an argument and that I can use an override in order to put in my own version. This gives the error error: Dependency is not of a valid type: element 10 of nativeBuildInputs for nix-shell.

What’s wrong here?

6

Missing parentheses.

[ a b ] is two list elements.
[ (a b) ] is one.

7

Thanks - rookie mistake. Let’s conclude this thread with a piece of correct code. This is the most economical solution - this way I can give selected packages the correct dependency version.

let
  einops = import ./einops.nix { inherit pkgs; }; 
  huggingface-hub2 = import ./huggingface-hub.nix { inherit pkgs; }; # specific version
in
pkgs.mkShell rec {
  nativeBuildInputs = with pkgs.buildPackages; [
    [...]
    einops
    python312Packages.numpy
    (python312Packages.tokenizers.override {
      huggingface-hub = huggingface-hub2;
    })
  ];
}