Building pyproject in FHS env

Help
1

First, context: I have a python project, built with poetry on Ubuntu, and try to build it with Nix. poetry.lock file contains urls and hashes of all 163 dependencies, in theory it’s enough for deterministic build. But many of them are not in Nixpkgs, and even more important, I’m tied to specific versions (of course different from what is in Nixpkgs) of numpy and some other packages, which requires patches on Nix. Both poetry2nix and mach-nix and dream2nix fails with my pyproject.toml. So I am going to use buildFHSUserEnv and install my packages in an environment with FHS (assumed by many of my dependencies).

However, even this is not straightforward. First, I was required to add a multitude of python build systems (and cmake python package I take from wheel, because it does not compile). Second then, opencv-contrib-python fails to install with fatal error: crypt .h: No such file or directory, and I don’t understand why it does not see glibc (I’ve compared with derivation from Nixpkgs, but don’t see what makes the difference).

Any help is appreciated, both with building opencv-contrib-python in FHS env and expected following multiple problems…

Here is the gist with my (a bit simplified) poetry project and shell.nix (nix-shell tries to install downloaded by Nix source of opencv-contrib-python in virtualenv): Build pyproject on Nix in FHS · GitHub

2

This is indeed complex! You’re building opencv via their python package, which is effectively just a homebrew build system dedicated to only building opencv with cmake. This means you’ll need to have all dependencies correctly prepared in the environment for cmake to pick them up without their build system breaking things. Since you’re building in the runScript, you’ll even have to have them available at runtime, which I’m pretty sure buildFHSUserEnv avoids by default (hence the need for all the -dev packages), so you’re going to have to fully understand the scikit build system, which environment variables/paths opencv needs, how cmake ultimately receives them, and which nixpkgs packages need to be put in them how. That’s bridging like three or four levels of build system abstractions that you need to fully grasp minute details of to make this work.

Naturally you’ll run into a bit of a mess here, you can be proud to have gotten this far.

So firstly, I’d suggest giving poetry2nix another look. You’d not be the only one to build opencv with it, which I think gives you more of a fighting chance than going alone. You can always use packages built by poetry2nix in an fhs env. Building packages in an fhs env is really a last resort you should only need for a truly insane build system - which this one might well be, but I haven’t yet seen evidence of your pain when using poetry2nix :wink:

As for the others, last I looked mach-nix was abandoned - around the time pyproject.toml was introduced - in favor of improving dream2nix, which in turn (currently) lacks the patch “database” that poetry2nix has, and is therefore much harder to use. Don’t use them for now, unless you have a very good reason to.

With that out of the way, can you give a bit more info, i.e. build logs (with poetry2nix as well as with your custom solution) and such? The build inputs are nice if I want to reproduce the failures, but it’s a lot of effort to actually run and debug the build for you, so this will probably lie around with no answers forever as-is.

Some wild guesses at the bit of information you’ve given so far:

  • opencv-contrib-python fails to install with fatal error: crypt .h: No such file or directory

    • I believe that’s a misspelled python header. Maybe some file got mangled somewhere? It should almost certainly be crypt.h.

  • I don’t understand why it does not see glibc

    • Assuming “it” is the opencv build script (how are you getting the above error without glibc?) and not an eldritch horror from a popular book (wouldn’t surprise me to find one in a build system though). As I mentioned earlier, I’m guessing it’s because you’re building it at runtime, rather than at build time with stdenv.mkDerivation, which usually makes gcc and other basic build requirements available in the environment. This runtime/buildtime split might also be related to the prior error, but I’m more concerned about the random space appearing in a filename there.
3

@le-chat greetings.

In theory poetry2nix should be preferred approach, I usually have better luck either with virtualenv + FHSUserEnv or manually package using pkgs.buildPythonPackage

in your example I am not sure why you are trying to install opencv-contrib-python from source, given its already in your pyproject.toml and poetry.lock

here is shell.nix I’ve used together with pyproject.toml and poetry.lock from your gist

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0874168639713f547c05947c76124f78441ea46c.tar.gz") { } }:
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    python38
    python38Packages.pip
    python38Packages.poetry
    python38Packages.virtualenv
    gcc
    pkg-config
    cairo.dev
    xorg.libxcb.dev
    xorg.libX11.dev
    xorg.xorgproto
    glib.dev
    gobject-introspection.dev
    libffi.dev
    cmake
    zlib
    libglvnd
  ]);
  runScript = "bash";
}).env


nix-shell
poetry install
poetry run python3 -c "import cv2; print(cv2.__version__)"
4.7.0

I’ve noticed that both opencv-python and opencv-contrib-python are listed in pyproject.toml
according to opencv-contrib-python · PyPI you only need one

this also installs without issues

poetry shell
pip install opencv-contrib-python

hopefuly this helps.

1 Like
4

Thank you for the response!
I will post logs with poetry2nix later, and here are build logs for my custom variant:

Regarding a space in crypt.h, I think it’s introduced by copy-paste from less in terminal to a firefox.

I would like to make a proper build derivation, not installing in runtime; I just haven’t yet figured out how to combine mkDerivation with FHS. Again, any suggestions/links are welcome.

@kirillrdy Thank you too. I haven’t used poetry install because I don’t see how to make poetry work without network, inside Nix’s build phase. May be you know way to achieve this?

5

I’ve also tried again with poetry2nix, here are the gist and the log.

It fails to build apriltag with a message ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found. Though there is setup.py in apriltag tarball…

6

while I would still like to produce an example without poetry2nix, i also tried poetry2nix with a clean project that only has opencv-python

i get

nix-shell 
error: evaluation aborted with the following error message: 'cannot find attribute `metadata.files''
(use '--show-trace' to show detailed location information)

lets keep trying :grinning:

7

I’ve updated my previous gist.
For building apriltag I’ve overriden

preBuild = "cd ..";
postBuild = "mv dist/ build/dist && cd build";

It looks a bit strange and probably I need to pass some parameter so that standard hooks could do this job.

And with pycairo I am getting an error about missing crypt.h after the same hack to work around «not installable directory». The full log cannot be posted due to anti-spam protection of discourse.

8

From a PR linked from Cannot build python38packages.pycairo on unstable · Issue #198036 · NixOS/nixpkgs · GitHub I’ve known that for python <3.9 I need also libxcrypt.

I’ve again updated the last gist. Now an environment build succesfully, though it cannot import some gstreamer modules:

import gi;
gi.require_version('Gst', '1.0');
gi.require_version('GstRtspServer', '1.0');
from gi.repository import GLib, Gst, GstRtspServer, GObject;

fails with gi.RepositoryError: Typelib file for namespace 'GstRtsp', version '1.0' not found

9

I’ve forgotten to mention that the problem with GstRtsp being not found explained just by missing gst-plugins-base.