Installed custom Python script can't find it's modules

PhilT · January 11, 2024, 9:31pm

I have the following steamvr_utils/default.nix file:

{ lib
, python3Packages
, fetchFromGitHub
, daemon
}:

python3Packages.buildPythonPackage rec {
  pname = "steamvr_utils";
  version = "1.1.2"; # This is currently master but hopefully it'll be tagged at some point
  format = "pyproject";

  src = fetchFromGitHub {
    owner = "DavidRisch";
    repo = "steamvr_utils";
    rev = "master";
    sha256 = "1HQiO8JlY170WeWkD8dUwoh6OaH1SL7Gff07SvrrYew=";
  };

  nativeBuildInputs = with python3Packages; [
    hatchling
    hatch-vcs
  ];

  env.SETUPTOOLS_SCM_PRETEND_VERSION = version;

  propagatedBuildInputs = with python3Packages; [
    bluepy
    psutil
  ];

  preBuild = ''
      cat > pyproject.toml << EOF
      [build-system]
      requires = ["hatchling"]
      build-backend = "hatchling.build"

      [project]
      name = "steamvr_utils"
      version = "${version}"

      dependencies = [
          "bluepy",
          "psutil",
      ]

      [project.scripts]
      steamvr_utils = "scripts.steamvr_utils:main"
      EOF
    '';
}

The main issue is when I run steamvr_utils I get

ModuleNotFoundError: No module named 'audio'

It’s failing on line 6 of scripts/steamvr_utils.py which is import audio. I’m not too familiar with Python but this appears to be a local folder in the scripts folder containing a bunch of .py files. So I don’t think it’s finding them. The project isn’t following standard build conventions as there was no setup.py or pyproject.toml which is why I needed to add one but I assume scripts/audio/__init__.py would be used to import the audio module/folder. I’m not sure how to make it find that folder though.

rjpc · January 11, 2024, 11:18pm

Have a look at this thread Broken python path in custom python package it might hold the answer; though I tried what is suggested without success. You’ll need to tweak it somehow

bowmanjd · January 12, 2024, 12:38am

I am trying to figure out if that package is compatible with hatchling’s expectations as far as what directories the code is in. Any thoughts on that?

I am new to hatchling, but is there a chance you need to explicitly include scripts directory?

include = [
  "scripts/*.py"
]

Or you could use the setuptools backend, specifying where = ["scripts"]

That package is put together in a… let’s say “nontraditional” manner for a Python package.

573 · January 12, 2024, 10:44am

Hi, I’m a also new to this but interested in Python in general, here is a comparison to pyproject.toml (pip).

PhilT · January 12, 2024, 4:35pm

Thanks everyone, I think I’m stuck on the entrypoint bit now. Probably more of a Python related issue.

I have:

     [project.scripts]
      steamvr_utils = "steamvr_utils.steamvr_utils:main"

in my pyproject.toml but I’m getting:

  File "/nix/store/ps2rxp4p2ni45ghf8c3pd7z3gm8sqfa8-python3.11-steamvr_utils-1.1.3c/bin/.steamvr_utils-wrapped", line 6, in <module>
    from steamvr_utils.steamvr_utils import main
ModuleNotFoundError: No module named 'steamvr_utils'

I’ve created a fork with the following structure:

steamvr_utils/
- src/
  - steamvr_utils/ ← I understood this is a package
    - audio/
    - basestation_interface/
    - pactl_interface/
    - steamvr_utils.py ← and this is a module containing def main()

My understanding is that [project.scripts] format is exe_name = "package_name.module_name:main".

Any ideas what I’m doing wrong?

PhilT · January 14, 2024, 11:50am

Wow! That was a tough one! Every time I think I’ve setup the hardest NixOS derivation I get a harder one!

Thanks to the comments above and a lot of searching and trial and error I eventually got to the bottom of the problem.

src

First, I had set up a local src dir to make debugging the issue easier. However, ../steamvr_utils didn’t seem to detect changes I’d made to the project so I had to keep nudging the version number to rebuild the NixOS.
This was fixed by using the following. I’m not exactly sure what this does (other than most likely ignores files in .gitignore as part of rebuild changes) but it meant I could update the source without nudging the version.

src = pkgs.nix-gitignore.gitignoreSource [ ] /data/code/steamvr_utils;

setuptools

I switched to setuptools. Some docs I was following was using setuptools and I didn’t have any investment in Hatchling so I just switched to that.

nativeBuildInputs = with python3Packages; [
    setuptools
    wrapPython
  ];

wrapper

I added a wrapper as per some of the suggestions above (thanks!). Not sure all of this is necessary, I’ll update once I’ve tested a bit more:

  postFixup = ''
    wrapProgram "$out/bin/steamvr_utils" \
      --prefix PYTHONPATH : "$PYTHONPATH" \
      --prefix PYTHONPATH : "$out/${python311.sitePackages}" \
      --prefix PATH : "${python311}/bin"
  '';

bluepy

steamvr_utils uses bluepy which needs setcap run on the bluepy-helper.

  security.wrappers.bluepy-helper = {
    owner = "root";
    group = "root";
    capabilities = "cap_net_raw,cap_net_admin+eip";
    source = "${pkgs.python3Packages.bluepy}/${pkgs.python311.sitePackages}/bluepy/bluepy-helper";
  };

This also meant I’d need to tweak the path that the bluepy library uses to the bluepy-helper so instead of pointing to a relative path it points to the wrapped bluepy-helper in /run/wrappers/. If anyone knows of a nicer way to do this I’m all ears (perhaps through patching?). Relevant change in the bluepy library (I’ll be making a fork of this unless a better solution comes up):

diff --git a/bluepy/btle.py b/bluepy/btle.py
index 3fa3564..dff6c08 100755
--- a/bluepy/btle.py
+++ b/bluepy/btle.py
@@ -21,7 +21,7 @@ def preexec_function():

 Debugging = False
 script_path = os.path.join(os.path.abspath(os.path.dirname(__file__)))
-helperExe = os.path.join(script_path, "bluepy-helper")
+helperExe = os.path.join("/run/wrappers/bin/bluepy-helper")

restructuring of steamvr_utils

What I discovered here (and the main issue I was having) was that the script files weren’t present in the store once it’d been built.

I eventually found that adding packages in the pyproject.toml got them into the output:

[tool.setuptools]
packages = [
  "src",
  "src.config",
  "src.images",
  "src.steamvr_utils",
  "src.steamvr_utils.audio",
  "src.steamvr_utils.basestation_interface",
  "src.steamvr_utils.pactl_interface",
]

Various folders needed __init__.py files or they also didn’t get included.

Some assets needed adding to a MANIFEST.in file in the root of the project alongside pyproject.toml.

include src/config/*.yaml
include src/images/*.png

I can probably pull these out of src/ but I’m going to tidy things up a bit generally now that I have a better idea of what’s going on.

There are a few other bits which I’m still working out (like whether __init__.py actually requires the imports or not. I might also go back and see if I can now configure it with the existing repo (rather than my fork) now I have a better understanding.

[project.scripts] what finally worked

Once I had the actually scripts in the store I finally understood what went into [project.scripts]:

[project.scripts]
steamvr_utils = "src.steamvr_utils.steamvr_utils:main"

I’ll update this post once I’ve cleaned things up a bit.

Thanks again all.