How do I package a python app using importlib.metadata

Help
1

I have the following package

{
  lib,
  python3,
  fetchFromGitHub,
}:

python3.pkgs.buildPythonApplication rec {
  pname = "ofxstatement";
  version = "0.9.2";
  pyproject = true;

  src = fetchFromGitHub {
    owner = "kedder";
    repo = "ofxstatement";
    rev = version;
    hash = "sha256-GLOBunyE4BWHTtiNmPw6CPR+xgze8aXn7bC0uYuWWi8=";
  };
  prePatch=''
    cp ${fetchFromGitHub {
    owner = "3v1n0";
    repo = "ofxstatement-n26";
    rev = "0.2";
    hash = "sha256-zGpYMNf1O0tfj0CqUCy1nDxUEKsZg5eEgqL5F0iTgLs=";
  }}/src/ofxstatement/plugins/n26.py src/ofxstatement/plugins
  '';

  build-system = [
    python3.pkgs.setuptools
  ];

  dependencies = with python3.pkgs; [
    importlib-metadata
    platformdirs
    zipp
  ];

  pythonImportsCheck = [
    "ofxstatement"
  ];

  meta = {
    description = "Tool to convert proprietary bank statement to OFX format, suitable for importing to GnuCash or other personal finance applications";
    homepage = "https://github.com/kedder/ofxstatement";
    changelog = "https://github.com/kedder/ofxstatement/blob/${src.rev}/CHANGES.rst";
    license = lib.licenses.gpl3Only;
    maintainers = with lib.maintainers; [ pasqui23 ];
    mainProgram = "ofxstatement";
  };
}

but after I build it I get the following error:

result/bin/ofxstatement list-plugins
No plugins available. Install plugin eggs or create your own.
See https://github.com/kedder/ofxstatement for more info.

This is wrong as it should list the n26 plugin
GitHub - 3v1n0/ofxstatement-n26: N26 parser for ofxstatement only list that n26.py file, so I thought (erroely) that merely copying it would suffice

2

Why not packaging both ofxstatement and the plugin (in two different derivations that is), then install them both?

FWIW I have this flake-parts module (no-need to get distracted if you don’t use flake-parts), that may serve as inspiration:

{
  perSystem = { pkgs, ... }:
    let
      systemPkgs = { buildPlatformPkgs = pkgs; inherit (pkgs) python3Packages; };

      mkOFXStatement = import ../../../third_party/pypi/ofxstatement.nix;
      ofxstatement = mkOFXStatement systemPkgs;

      mkOFXStatementCommon = { python3Packages, ofxstatement }:
        python3Packages.buildPythonPackage {
          pname = "ofxstatement-common";
          src = ./common;
          version = "0.0.1";
          propagatedBuildInputs = [ ofxstatement ];
        };
        ofxstatement-common = mkOFXStatementCommon {
          inherit ofxstatement;
          inherit (pkgs) python3Packages;
        };

      mkOFXStatementUSSchwab = { python3Packages, ofxstatement, ofxstatement-common }:
        python3Packages.buildPythonPackage {
          pname = "ofxstatement-us-schwab";
          src = ./us-schwab;
          version = "0.0.1";
          propagatedBuildInputs = [
            python3Packages.python-dateutil
            ofxstatement
            ofxstatement-common
          ];
          doCheck = false;
        };
        ofxstatement-us-schwab = mkOFXStatementUSSchwab {
          inherit ofxstatement ofxstatement-common;
          inherit (pkgs) python3Packages;
        };
    in {
      devShells.ofxstatement = pkgs.mkShell {
        buildInputs = [ ofxstatement-us-schwab pkgs.python3Packages.ipython ];
      };
    };
}

And that ofxstatement.nix file I wrote a while ago can be found here.

3

Tried that, didn’t work

4

Can you maybe tell us more about that? Did you get any error message?

5

running ofxamstatement list-plugins still returns no plugins

6

Sorry about that, you may have to debug Python’s import machinery from the Nix generated wrappers. I use a fork of ofxstatement from 2017…

A little over a year ago, I had to dig quite deep to figure out some import issues that had to do with namespaces package and resulted in this commit.

FWIW, below are my notes from back then (machine translated from French), that’s all I have:

This piece of code seems to be problematic:

initializer = frozenset(site._init_pathinfo())
print(f"LOUIS: initializer={sorted(initializer)}")


count = 0


def _addsitedir(seen_paths, path):
    global count
    print(f"LOUIS: [{count}] _addsitedir({seen_paths}, {path})")
    count += 1
    return site.addsitedir(path, known_paths=seen_paths)


output = functools.reduce(
    _addsitedir, [
        '/nix/store/nxsgs6snh241sy0bldv538qzn4vw9pjm-python3.10-ofxstatement-0.6.1/lib/python3.10/site-packages',  # noqa
        '/nix/store/jnxjcwa7sy609awh6msbn0vbz2q9fbkf-python3.10-appdirs-1.4.4/lib/python3.10/site-packages',  # noqa
        '/nix/store/q6s6m8wjxyp0552k0y9cg0arisgvjn5g-python3.10-setuptools-65.3.0/lib/python3.10/site-packages'  # noqa
    ], initializer
)
assert output == initializer

When you remove it, ofxstatement list-plugins works. Maybe check why this snippet is being generated.

In the end, since you’re only adding something that already exists, it leans toward being a bug in Python nonetheless. There’s no explanation with nixpkgs@01624e1ac29ee0854cb63d0c1efb6d791c1441d4, but maybe you can trace it further. It’s strange that site._init_pathinfo (by the way, what does this function do?) already returns all the paths.

Relevant PEPs are: 302 & 420 at least. It would be good to understand what these .pth files do.

It seems that, in the end, the problem is related to the -nspkg.pth file generated by setuptools for the ofxstatement package. It might be worth switching everything to implicit namespace packages (PEP 420), as I strongly believe that could make the issue disappear.

Converting everything to PEP-420 and also using the src-layout with setuptools fixed the issue.