Broken python path in custom python package

Help
1

I believe there is something fundamental I am misunderstanding about nix python packaging and dependency management. In my nix configs, I have created a custom Python package as follows:

{ lib, python311, fetchFromGitHub }:
python311.pkgs.buildPythonApplication rec {
  pname = "mssql-cli";
  version = "1.0";

  propagatedBuildInputs = with python311.pkgs; [
    applicationinsights
    cli-helpers
    click
    configobj
    future
    humanize
    prompt-toolkit
    sqlparse
    wheel
  ];

  src = fetchFromGitHub {
    owner = "dbcli";
    repo = "mssql-cli";
    rev = "HEAD";
    hash = "sha256-XB+r8FW81oJ5h86LN1gkbOaN2s7QyVIW98YgDQQzH50=";
  };

  doCheck = false;

  pythonImportsCheck = [
    "mssqlcli"
  ];
}

But when I build it, the result has python path issues. When I run python -m mssqlcli I receive a:

ModuleNotFoundError: No module named 'mssqlcli'

If I forcefully add the correct path, then run it again:

export PYTHONPATH="/nix/store/qlq2djhc493vyi7knjs0c8dr5akv1hdp-mssql-cli-1.0/lib/python3.11/site-packages"
python -m mssqlcli

Then I get

ModuleNotFoundError: No module named 'configobj'

One of the build inputs was configobj, so I assumed it would be included? I am sure I could make a custom script that adds each dependency from the nix store to the PYTHONPATH, but I am pretty sure this should be taken care of if I do the derivation right.

Any thoughts on what I am doing wrong? Thank you!

1 Like
2

(thanks to @hexa for help in https://app.element.io/#/room/#python:nixos.org)

So what happens here is that mssql-cli is configured in the setup.py via scripts (this script here). I.e. the scripts are just copy/pasted naively without being wrapped by nix. The cleanest solution would be to setup them as entrypoints in the setup.py calling the mssqlcl.main:main function, instead of scripts, following for instance this example. But since this is not under our control, the simplest is certainly to wrap the binary file with:

  postFixup = ''
    wrapProgram "$out/bin/mssql-cli" \
      --prefix PYTHONPATH : "$PYTHONPATH" \
      --prefix PATH : "${python311}/bin"
  '';

The whole derivation:

{ lib, python311, fetchFromGitHub }:
python311.pkgs.buildPythonPackage rec {
  pname = "mssql-cli";
  version = "1.0";

  nativeBuildInputs = [ python311.pkgs.wrapPython ];
  
  propagatedBuildInputs = with python311.pkgs; [
    applicationinsights
    cli-helpers
    click
    configobj
    future
    humanize
    prompt-toolkit
    sqlparse
    wheel
    pygments
  ];

  patchPhase = ''
    substituteInPlace setup.py \
      --replace "click >= 4.1,<7.1" "click"
  '';

  src = fetchFromGitHub {
    owner = "dbcli";
    repo = "mssql-cli";
    rev = "HEAD";
    hash = "sha256-v2RKJXG2bqIviQeppaa6JVhlgBdbj0axip12bxfTcMs=";
  };

  postFixup = ''
    wrapProgram "$out/bin/mssql-cli" \
      --prefix PYTHONPATH : "$PYTHONPATH" \
      --prefix PATH : "${python311}/bin"
  '';
  
  doCheck = false;

  pythonImportsCheck = [
    "mssqlcli"
  ];
}
3 Likes
3

Thank you! That did work, although I did not need the substituteInPlace modification for click. Should I be worried? (I am on unstable)

Any chance you could point me to the docs for wrapPython? I am still new to nix and its documentation, and am having a hard time finding the authority on that function (or package?). I can tell it adds site-package directories, which is exactly what I needed.

4

Oh, don’t think you need to worry. Oh, actually the wrapPython is actually a test that failed, I think you should rather require makeWrapper instead (this might have worked above because wrapPython certainly requires makeWrapper?). So the documentation for wrapProgram is here, it points to the documentation of makeWrapper here, that points to the source code here with all options in comments.

There is also wrapPythonPrograms, but it seems like it does not work for scripts, and most of the time it is automatically applied if you use buildPythonPackage (just filled an issue here). The manual mentions only it very briefly here, but if you really want to read more you can check the source code of nix (yeah, I hate to say that). But the source is not too hard to read:

  • just install rg (better and faster grep -r),
  • git clone https://github.com/NixOS/nixpkgs somewhere (I do it in ~/ directly, quite practical to have it close)
  • Go to this folder and run: (-F disables any regexp)
$ rg -F "wrapPythonPrograms()"
pkgs/development/interpreters/python/wrap.sh
3:wrapPythonPrograms() {

This points you directly to the source code, that, hopefully, contains enough comments to help you understand it. You can also try rg "yourFunction =" if the above fails (80% of the time this will bring you to the definition), or directly rg yourFunction (but you will need a bit more pruning of the results).

1 Like
5

Your response has helped me understand quite a few elements of the nix ecosystem. Thank you SO much!