Packaging a Python script without setup.py

Help
1

Hello there,

I am trying to package this tool that only provides a Dockerfile installation, without any specification of what the setup process should be. My assumption is that it involves

  • installing the dependencies
  • copying the full source tree to nix’s $out
  • writing some kind of wrapper script that can serve as a “binary” for nix

I already wrote a preliminary flake for this, available here and reproduced here:

{
  description = "nnenum Nix flake.";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nix-filter.url = "github:numtide/nix-filter";
    nixpkgs.url = "nixpkgs";
  };
  outputs =
    { self
    , flake-utils
    , nixpkgs
    , nix-filter
    , ...
    }:
    flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs = nixpkgs.legacyPackages.${system};
      pythonPkgs = pkgs.python311Packages;
      lib = pkgs.lib;
      swiglpk = (with pkgs; pythonPkgs.buildPythonPackage rec {
        pname = "swiglpk";
        version = "5.0.10";
        src = fetchPypi {
          inherit pname version;
          hash = "sha256-V6w0rTNNqV3RaBFL/bUK4Qoqaj3e8h5JQfRv5DDFp+E=";
        };
        buildInputs = with pkgs; [ glpk ];
        nativeBuildInputs = with pkgs; [ glpk swig ];
      });
    in
    rec {
      packages = rec {
        pname = "nnenum";
        version = "0.1";
        default = self.packages.${system}.nnenum;
        # Since nnenum does not provide a binary, we need to use it as a package
        nnenum = pythonPkgs.buildPythonPackage
          {
            inherit pname version;
            src = pkgs.fetchFromGitHub {
              inherit pname version;
              owner = "stanleybak";
              repo = "nnenum";
              rev = "cf7c0e72c13543011a7ac3fbe0f5c59c3aafa77e";
              hash = "sha256-/EoGMklTYKaK0UqX/MKbUBWXrCtQQSC2c9ZRV/L6QLI=";
            };
            format = "other";
            propagatedBuildInputs = with pythonPkgs; [
              numpy
              scipy
              onnx
              onnxruntime
              skl2onnx
              termcolor
              swiglpk
            ];
            # nnenum does not provide a setup.py build, so we only need the
            # dependencies and add the `src/` folder to the nix path
            buildPhase = ''
              mkdir $out
            '';
            installPhase = ''
              export OPENBLAS_NUM_THREADS=1
              export OMP_NUM_THREADS=1
              cp -r src/ $out/
            '';
            # TODO: add the PYTHONPATH to the src folder, and set the two
            # environment variables in the derivation
            checkPhase = ''
              export PYTHONPATH=src/:$PYTHONPATH
              bash run_tests.sh
            '';
          };
      };
    });
}

Build completes, but I don’t get quite right how to “expose” the nnenum.py script in my flake.

It is the first time I’m trying to package a Python application, so any constructive feedback is appreciated.

2

The tool you are trying to build isn’t a python package. A python package is only one module, commonly with a pyproject.toml file that defines its runtime, test and build requirements.

In this case, the tool is a collection of modules that are not installable as a common library does. These python scripts need some python environment, so instead of building a package you should build the environment (i.e. the requirements) and wrap the call to the python script into a shell application.

nnenum-python-env = nixpkgs.python311.buildEnv.override {
    extraLibs = [ numpy scipy onnx ]; # the requirements
    ignoreCollisions = false;
};
output = nixpkgs.writeShellApplication {
    name = "nnenum";
    runtimeInputs = [ nnenum-python-env ];
    text = ''
      python "${src}/nnenum.py"
    '';
  };

When output is built, you can execute <derivation-output>/bin/nnenum to execute the script using the expected python environment.

1 Like
3

I see, thank for your insight. The issue I see with this approach is that if I package the modules that way, the nnenum.py is not able to find other modules.

With the following modified flake:

{
  description = "nnenum Nix flake.";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nix-filter.url = "github:numtide/nix-filter";
    nixpkgs.url = "nixpkgs";
  };
  outputs =
    {
      self,
      flake-utils,
      nixpkgs,
      ...
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        pythonPkgs = pkgs.python311Packages;
        swiglpk = (
          with pkgs;
          pythonPkgs.buildPythonPackage rec {
            pname = "swiglpk";
            version = "5.0.10";
            src = fetchPypi {
              inherit pname version;
              hash = "sha256-V6w0rTNNqV3RaBFL/bUK4Qoqaj3e8h5JQfRv5DDFp+E=";
            };
            buildInputs = with pkgs; [ glpk ];
            nativeBuildInputs = with pkgs; [
              glpk
              swig
            ];
          }
        );
        nnenum-python-env = pkgs.python311.buildEnv.override {
          extraLibs = with pythonPkgs; [
            numpy
            scipy
            onnx
            onnxruntime
            skl2onnx
            termcolor
            swiglpk
          ];
          ignoreCollisions = false;
        };
      in
      {
        packages = {
          default = self.packages.${system}.nnenum;
          # Since nnenum does not provide a binary,
          # we need to write a shell application that uses
          # a python environment with nnenum dependencies
          nnenum =
            let
              src = pkgs.fetchFromGitHub {
                pname = "nnenum";
                version = "0.1";
                owner = "stanleybak";
                repo = "nnenum";
                rev = "cf7c0e72c13543011a7ac3fbe0f5c59c3aafa77e";
                hash = "sha256-/EoGMklTYKaK0UqX/MKbUBWXrCtQQSC2c9ZRV/L6QLI=";
              };
            in
            pkgs.writeShellApplication {
              name = "nnenum";
              runtimeInputs = [ nnenum-python-env ];
              text = ''
                python ${src}/src/nnenum/nnenum.py
              '';
            };
        };
      }
    );
}

I get the error when running nix run .: ModuleNotFoundError: No module named 'nnenum.enumerate'; 'nnenum' is not a package

My understanding is that I should add to the PYTHONPATH the full filetree of the module, such that calling nnenum.py will be done with the correct env vars. Could there be a clean, nix way of doing that?

4

That’s correct. Using PYTHONPATH is not the best approach to integrate libraries since it can hide collisions, but in this case is ok since it is not a lib and these are “local” scripts that are expected to be integrated this way.

As a normal shell script, you can include more statements in addition to the python script call. i.e.

text = ''
    export PYTHONPATH="${src}/src/nnenum"
    python ${src}/src/nnenum/nnenum.py
'';
1 Like
5

Thank you. I was able to successfully package several Python modules using this wrapper script.

1 Like