Could use some help with Python packaging

I’m trying to package a certain python program from Github to use in my personal overlay, but I have very limited experience doing this and in particular have never attempted to package a Python program, so I would really appreciate any help.

For reference, I’ve looked at expressions for packaging similar programs that are in Nixpkgs, but there are still some issues. Here is the code I have at the moment:

{ lib
, buildPythonPackage
#, python311Packages.exif - how do I abstract it out from a specific package set? "exif" package in nixpkgs is not the same program
, fetchFromGitHub
, pyqt6
#, pyqt6-qt6 - doesn't exist?
#,rectangle-packer - this program is not in nixpkgs!
}:

buildPythonPackage rec {
  pname = "beeref";
  version = "0.3.0-dev";
  format = "setuptools"; # is this correct?

  src = fetchFromGitHub {
    owner = "rbreu";
    repo = "beeref";
    rev = "cf55d1a5e304da707f2cc9a9ca793ee24503a0bc";
    hash = "sha256-Qx1dV5yHK6SxCEh+vXfeeBR8qU7GdFoH9n4dNl4PVbw=";
  };

  nativeBuildInputs = [];
  propagatedBuildInputs = [
    #exif  - python311Packages version, or other?
    pyqt6
    #pyqt6-qt6    ?
    #rectangle-packer
  ];

  meta = with lib; {
    description = "A simple reference image viewer";
    homepage = "https://github.com/rbreu/beeref";
    license = licenses.gpl3Only;
    #maintainers = with maintainers; [];
  };
}

One of the issues is that rectangle-packer, while it exists on PyPi, is not part of Nixpkgs, and I’m not sure what would be the idiomatic way to include this in my expression. Would it involve the use of the fetchPypi function?

Another issue is that the setup file specifies both pyQt6 and pyQt6-Qt6 as dependencies. These packages both exist on PyPi, but the latter does not seem to be in Nixpkgs. Is it perhaps contained within the first one? I basically know next to nothing about how Python stuff is packaged for Nixpkgs, nor do I know much about Python development in general. How would I go about referencing them independently, if it even matters?

The last problem is the exif dependency, which seems to refer to this program (an outdated version of which is in Nixpkgs). The problem is that this is not the base exif package contained in Nixpkgs (which is actually an unrelated program), but instead the one contained within python package sets - for example, python311Packages.exif. Is there a way of referencing this dependency more abstractly outside of the context of the package set for a specific Python version, or is that just the way I have to do it? Not sure which is the desirable solution, either.

Again, I really appreciate any help! Thanks.

Edit: Now that I look at this again, I see that I missed the fact that the pyqt6 package is also part of python311Packages or similar package sets, so the same issue as for exif would apply, I think.

One of the issues is that rectangle-packer, while it exists on PyPi, is not part of Nixpkgs, and I’m not sure what would be the idiomatic way to include this in my expression. Would it involve the use of the fetchPypi function?

I just recently packaged a Python app where I had to also package some dependencies, you can use this as reference: https://github.com/NixOS/nixpkgs/pull/220305/commits/e2b1e574da314e4aaf771f4dd9f522ac2b37bf56

I’ve done this before in some other PR and ultimately was asked by a reviewer to turn these dependencies into top-level packages. :man_shrugging:

Another issue is that the setup file specifies both pyQt6 and pyQt6-Qt6 as dependencies. These packages both exist on PyPi, but the latter does not seem to be in Nixpkgs. Is it perhaps contained within the first one?

I don’t know tbh, but I’d say try building it with pyqt6 and see what happens :slightly_smiling_face:

The last problem is the exif dependency, which seems to refer to this program (an outdated version of which is in Nixpkgs).

The current version of the Python exif package in nixpkgs is 1.3.5.

I see beeref doesn’t specify a version for exif so 1.3.5 might work. Otherwise you’ll probably have to upgrade exif in nixpkgs separately first.

1 Like

Hi, thanks for the input. I’ve made quite a bit of progress following your example. Here is what I have currently:

{ lib
, fetchFromGitHub
, python3Packages
}:

let
  rectangle-packer = python3Packages.buildPythonPackage rec {
    pname = "rectangle-packer";
    version = "2.0.1";

    # disabled = python3Packages.pythonOlder "3.6";

    src = python3Packages.fetchPypi {
      inherit pname version;
      sha256 = "sha256-3Y5wGE0q+KGV1WjgwsPMGX8MlMKkd0dUEeXql4xnsn0=";
    };

    propagatedBuildInputs = with python3Packages; [
      cython
    ];

    #pythonImportsCheck = ["rectangle-packer"];
    doCheck = false;
  };

in
python3Packages.buildPythonPackage rec {
  pname = "beeref";
  version = "0.3.0-dev";
  format = "setuptools"; # is this necessary?

  src = fetchFromGitHub {
    owner = "rbreu";
    repo = "beeref";
    rev = "cf55d1a5e304da707f2cc9a9ca793ee24503a0bc";
    hash = "sha256-Qx1dV5yHK6SxCEh+vXfeeBR8qU7GdFoH9n4dNl4PVbw=";
  };

  nativeBuildInputs = [];
  propagatedBuildInputs = with python3Packages; [
    exif
    pyqt6
    #pyqt6-qt6
    rectangle-packer
  ];

  doCheck = false;

  meta = with lib; {
    description = "A simple reference image viewer";
    homepage = "https://github.com/rbreu/beeref";
    license = licenses.gpl3Only;
    #maintainers = with maintainers; [];
  };
}

This code, as is, gives the error:

error: builder for '/nix/store/hr8lb6sg2k3z392ganahnnwk2w9pbmwh-python3.10-beeref-0.3.0-dev.drv' failed with exit code 1;
       last 10 log lines:
       > installing
       > Executing pipInstallPhase
       > /build/source/dist /build/source
       > Processing ./BeeRef-0.3.0.dev0-py3-none-any.whl
       > Requirement already satisfied: exif in /nix/store/hx9fdccpc0k89088ffak5ais6za3kasg-python3.10-exif-1.3.5/lib/python3.10/site-packages (from BeeRef==0.3.0.dev0) (1.3.5)
       > Requirement already satisfied: pyQt6>=6.2.0 in /nix/store/wl9vm76xw93d49z0w9fchq02rdglrxjw-python3.10-PyQt6-6.4.2/lib/python3.10/site-packages (from BeeRef==0.3.0.dev0) (6.4.2)
       > Requirement already satisfied: rectangle-packer>=2.0.1 in /nix/store/fgdrpf4vldyqzipzfxyxb73p20jlyhxi-python3.10-rectangle-packer-2.0.1/lib/python3.10/site-packages (from BeeRef==0.3.0.dev0) (2.0.1)

       > ERROR: Could not find a version that satisfies the requirement pyQt6-Qt6>=6.2.0 (from beeref) (from versions: none)
       > ERROR: No matching distribution found for pyQt6-Qt6>=6.2.0

However, if I uncomment pyqt-qt6 from the propagatedBuildInputs, I get:

       error: undefined variable 'pyqt6-qt6'

       at /nix/store/cd5jj3dz3i7njm776cvn1w1zbpqnv3k5-source/pkgs/beeref/default.nix:43:5:

           42|     pyqt6
           43|     pyqt6-qt6
             |     ^
           44|     rectangle-packer

So, I dunno, what is the solution here? Do you think it would be best to just pull it in from Pypi in the same way I’m doing for rectangle-packer? Speaking of which, I was trying to follow your example with the pythonImportsCheck variable defined, but when I uncomment it I get:

error: builder for '/nix/store/68bma2lyvz90wh9rlcczcsa4ah0iz2jp-python3.10-rectangle-packer-2.0.1.drv' failed with exit code 1;
       last 10 log lines:
       > Check whether the following modules can be imported: rectangle-packer
       > Traceback (most recent call last):
       >   File "<string>", line 1, in <module>
       >   File "<string>", line 1, in <lambda>
       >   File "/nix/store/iw1vmh509hcbby8dbpsaanbri4zsq7dj-python3-3.10.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
       >     return _bootstrap._gcd_import(name[level:], package, level)
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'rectangle-packer'

That is the name of the package on Pypi, though. Any idea what is wrong? The tests fail as well if doCheck is commented out for rectangle-packer, but quite vaguely:

error: builder for '/nix/store/c4hm1vm6pm11ly94kyvlas104pr8rzz4-python3.10-rectangle-packer-2.0.1.drv' failed with exit code 2;
       last 10 log lines:
       >
       > test failed (uncaught exception)
       >
       > == Tests result: FAILURE ==
       >
       > 1 test failed:
       >     test
       >
       > Total duration: 53 ms
       > Tests result: FAILURE

Would really appreciate if you or anyone else has any further suggestions. :slightly_smiling_face:

About pyqt6:
It seems beeref insists on https://pypi.org/project/PyQt6-Qt6/ . This seems to contain some actual qt binaries, so probably doesn’t make sense in the context of Nix. I’d try patching it out of https://github.com/rbreu/beeref/blob/cf55d1a5e304da707f2cc9a9ca793ee24503a0bc/setup.py#L13 .
Also check how other packages in nixpkgs use pyqt6 for reference : https://github.com/search?q=repo%3ANixOS%2Fnixpkgs%20pyqt6&type=code

ModuleNotFoundError: No module named ‘rectangle-packer’
That is the name of the package on Pypi, though.

It’s the name of the package but this is about its Python module.
You can see in https://github.com/Penlect/rectangle-packer#basic-usage that the name of the module is actually rpack

1 Like

Much obliged for the help. Implementing all of your suggestions, including a patch for the setup file, I have managed to get the program running fine. :+1:
I can’t say I understand how the PyQt-Qt6 package works, in general nor in the context of Nix, but removing that line from the setup file did indeed fix that particular issue. In addition, I had to make a patch based on a Github PR to fix a runtime error I was getting, but aside from that, so far so good! Thank you again. :grin:

Only problem left is that I was getting all sorts of test failures when they were actually enabled. Everything does seem to work with doCheck set to false (for both rectangle-packer and beeref), but I’d prefer take a more systematic approach to it then just “don’t run any of the tests”. Maybe not so important in this case, but for future reference as well. I suppose a lot of this stuff is failing due to the restrictive nature of the Nix build environment, but I’m not really equipped to untangle all those issues. Can you perhaps suggest any good resources that I might read up on about running Python tests in Nix, or the build environment more generally?

1 Like

For one, tests that try to use the network will fail on Nix.
Some packages patch such tests out so the rest of the tests can be run when building on Nix.

Having said that I just noticed you said you’re packaging this for your personal overlay. Why not build it for nixpkgs instead? If you create a PR anyone can see the test failures and it’ll gain more visibility so you’ll get more help. Plus when it gets merged everyone will benefit from your efforts. I’m sure many people want to use Beeref from Nix too! :slightly_smiling_face:

Yeah that might be a good idea and I have considered it. I don’t think the expression I ended up writing is up to nixpkgs standards and like you mentioned there are a ton of test issues that I’m kind of just ignoring at this point. I’ve never made a nixpkgs PR so I’m not really familiar with the process but if I can get some help with improving the code then that is something I might do after all.

Not gonna lie, making a PR up to nixpkgs standards can be quite a bit of work, especially the first one. Obviously it’s up to you if you want to / are able to put some time into it.

Having said that, it is the best way to collaborate, you’ll learn from the experience, and eventually other people might start submitting updates / fixes to your package i.e. maintenance is actually shared among people using the package (there’s also an updater bot that will create PRs automatically!)

Sometimes it does happen that PRs go ignored for some time. After all there are always thousands of open PRs and only so many people with write access.
Still, it seems to me that you’ve already done 90% of the work here so I’d say there’s little reason not to send a PR.
See Nixpkgs 23.11 manual | Nix & NixOS and https://github.com/NixOS/nixpkgs/blob/b7d1bf6ee1624507688030ef3b7e2bb3cbe0e83d/CONTRIBUTING.md