Building pyproject in FHS env

le-chat · April 21, 2023, 7:31pm

First, context: I have a python project, built with poetry on Ubuntu, and try to build it with Nix. poetry.lock file contains urls and hashes of all 163 dependencies, in theory it’s enough for deterministic build. But many of them are not in Nixpkgs, and even more important, I’m tied to specific versions (of course different from what is in Nixpkgs) of numpy and some other packages, which requires patches on Nix. Both poetry2nix and mach-nix and dream2nix fails with my pyproject.toml. So I am going to use buildFHSUserEnv and install my packages in an environment with FHS (assumed by many of my dependencies).

However, even this is not straightforward. First, I was required to add a multitude of python build systems (and cmake python package I take from wheel, because it does not compile). Second then, opencv-contrib-python fails to install with fatal error: crypt .h: No such file or directory, and I don’t understand why it does not see glibc (I’ve compared with derivation from Nixpkgs, but don’t see what makes the difference).

Any help is appreciated, both with building opencv-contrib-python in FHS env and expected following multiple problems…

Here is the gist with my (a bit simplified) poetry project and shell.nix (nix-shell tries to install downloaded by Nix source of opencv-contrib-python in virtualenv): Build pyproject on Nix in FHS · GitHub

TLATER · April 21, 2023, 11:10pm

This is indeed complex! You’re building opencv via their python package, which is effectively just a homebrew build system dedicated to only building opencv with cmake. This means you’ll need to have all dependencies correctly prepared in the environment for cmake to pick them up without their build system breaking things. Since you’re building in the runScript, you’ll even have to have them available at runtime, which I’m pretty sure buildFHSUserEnv avoids by default (hence the need for all the -dev packages), so you’re going to have to fully understand the scikit build system, which environment variables/paths opencv needs, how cmake ultimately receives them, and which nixpkgs packages need to be put in them how. That’s bridging like three or four levels of build system abstractions that you need to fully grasp minute details of to make this work.

Naturally you’ll run into a bit of a mess here, you can be proud to have gotten this far.

So firstly, I’d suggest giving poetry2nix another look. You’d not be the only one to build opencv with it, which I think gives you more of a fighting chance than going alone. You can always use packages built by poetry2nix in an fhs env. Building packages in an fhs env is really a last resort you should only need for a truly insane build system - which this one might well be, but I haven’t yet seen evidence of your pain when using poetry2nix

As for the others, last I looked mach-nix was abandoned - around the time pyproject.toml was introduced - in favor of improving dream2nix, which in turn (currently) lacks the patch “database” that poetry2nix has, and is therefore much harder to use. Don’t use them for now, unless you have a very good reason to.

With that out of the way, can you give a bit more info, i.e. build logs (with poetry2nix as well as with your custom solution) and such? The build inputs are nice if I want to reproduce the failures, but it’s a lot of effort to actually run and debug the build for you, so this will probably lie around with no answers forever as-is.

Some wild guesses at the bit of information you’ve given so far:

opencv-contrib-python fails to install with fatal error: crypt .h: No such file or directory
- I believe that’s a misspelled python header. Maybe some file got mangled somewhere? It should almost certainly be crypt.h.
I don’t understand why it does not see glibc
- Assuming “it” is the opencv build script (how are you getting the above error without glibc?) and not an eldritch horror from a popular book (wouldn’t surprise me to find one in a build system though). As I mentioned earlier, I’m guessing it’s because you’re building it at runtime, rather than at build time with stdenv.mkDerivation, which usually makes gcc and other basic build requirements available in the environment. This runtime/buildtime split might also be related to the prior error, but I’m more concerned about the random space appearing in a filename there.

kirillrdy · April 22, 2023, 4:35am

@le-chat greetings.

In theory poetry2nix should be preferred approach, I usually have better luck either with virtualenv + FHSUserEnv or manually package using pkgs.buildPythonPackage

in your example I am not sure why you are trying to install opencv-contrib-python from source, given its already in your pyproject.toml and poetry.lock

here is shell.nix I’ve used together with pyproject.toml and poetry.lock from your gist

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0874168639713f547c05947c76124f78441ea46c.tar.gz") { } }:
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    python38
    python38Packages.pip
    python38Packages.poetry
    python38Packages.virtualenv
    gcc
    pkg-config
    cairo.dev
    xorg.libxcb.dev
    xorg.libX11.dev
    xorg.xorgproto
    glib.dev
    gobject-introspection.dev
    libffi.dev
    cmake
    zlib
    libglvnd
  ]);
  runScript = "bash";
}).env

nix-shell
poetry install
poetry run python3 -c "import cv2; print(cv2.__version__)"
4.7.0

I’ve noticed that both opencv-python and opencv-contrib-python are listed in pyproject.toml
according to opencv-contrib-python · PyPI you only need one

this also installs without issues

poetry shell
pip install opencv-contrib-python

hopefuly this helps.

le-chat · April 22, 2023, 3:30pm

Thank you for the response!
I will post logs with poetry2nix later, and here are build logs for my custom variant:

output of nix-shell, stdout and stderr
verbose output of pip (added to gist, 10 MB)

Regarding a space in crypt.h, I think it’s introduced by copy-paste from less in terminal to a firefox.

I would like to make a proper build derivation, not installing in runtime; I just haven’t yet figured out how to combine mkDerivation with FHS. Again, any suggestions/links are welcome.

@kirillrdy Thank you too. I haven’t used poetry install because I don’t see how to make poetry work without network, inside Nix’s build phase. May be you know way to achieve this?

le-chat · April 25, 2023, 6:38am

I’ve also tried again with poetry2nix, here are the gist and the log.

It fails to build apriltag with a message ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found. Though there is setup.py in apriltag tarball…

kirillrdy · April 25, 2023, 7:47am

while I would still like to produce an example without poetry2nix, i also tried poetry2nix with a clean project that only has opencv-python

i get

nix-shell 
error: evaluation aborted with the following error message: 'cannot find attribute `metadata.files''
(use '--show-trace' to show detailed location information)

lets keep trying

le-chat · April 25, 2023, 10:40am

I’ve updated my previous gist.
For building apriltag I’ve overriden

preBuild = "cd ..";
postBuild = "mv dist/ build/dist && cd build";

It looks a bit strange and probably I need to pass some parameter so that standard hooks could do this job.

And with pycairo I am getting an error about missing crypt.h after the same hack to work around «not installable directory». The full log cannot be posted due to anti-spam protection of discourse.

le-chat · April 26, 2023, 7:57am

From a PR linked from Cannot build python38packages.pycairo on unstable · Issue #198036 · NixOS/nixpkgs · GitHub I’ve known that for python <3.9 I need also libxcrypt.

I’ve again updated the last gist. Now an environment build succesfully, though it cannot import some gstreamer modules:

import gi;
gi.require_version('Gst', '1.0');
gi.require_version('GstRtspServer', '1.0');
from gi.repository import GLib, Gst, GstRtspServer, GObject;

fails with gi.RepositoryError: Typelib file for namespace 'GstRtsp', version '1.0' not found

le-chat · May 18, 2023, 5:53am

I’ve forgotten to mention that the problem with GstRtsp being not found explained just by missing gst-plugins-base.

le-chat · December 7, 2023, 2:21pm

My poetry2nix-based solution has been working for several months, but now I’ve faced with new problems:

First, with some recent upgrade of poetry2nix some python packages stopped to build;
Even worse, I need torch-tensorrt, which is not patched for Nix.

I think it’s too much effort to patch nvidia-related packages on my own. So I consider falling back to buildFHSUserEnv. I’ve found from FHS environments in a builder an example, but it does not work for me; it seems like inside of FHS environment $out is seen as read-only file system, so I cannot put their anything.

Am I right, and how can FHS be used inside of derivation builder?

In logs I have on attempt to run pip download:

ERROR: Exception:
Traceback (most recent call last):
  File "/nix/store/n8afz8jgi2mngpdw6y72k7a6v7g54rpi-python3-3.10.12-env/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/nix/store/n8afz8jgi2mngpdw6y72k7a6v7g54rpi-python3-3.10.12-env/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/nix/store/n8afz8jgi2mngpdw6y72k7a6v7g54rpi-python3-3.10.12-env/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 91, in run
    ensure_dir(options.download_dir)
  File "/nix/store/n8afz8jgi2mngpdw6y72k7a6v7g54rpi-python3-3.10.12-env/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 106, in ensure_dir
    os.makedirs(path)
  File "/nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/134fm1rs4qpfnxzqsig0kgpqg1fj5w6p-pip-download/packages'

Package definition looks like:

          packages.pyfhs = let
            deps = with pkgs; [
              jq gcc pkg-config zlib
              glib glib.dev
              libffi cairo openssl libxcrypt
              gst_all_1.gstreamer
              gobject-introspection
              gst_all_1.gst-rtsp-server
              gst_all_1.gst-plugins-base
              gst_all_1.gst-plugins-ugly
              gst_all_1.gst-plugins-good
              ffmpeg-headless
              curl
              poetry (python310.withPackages (ps: [ ps.pip ps.wheel ]))
              busybox util-linux # for debug
              ];
            sources = (nixpkgs.lib.cleanSource ./.);
            fhsWrapper = pkgs.buildFHSUserEnv {
              name = "pip-install-fhs-wrapper";
              runScript = "$@";
              targetPkgs = _: deps;
            };
            downloader = derivation (rec {
              name = "pip-download";
              inherit system;
              # builder = "${fhsWrapper}/bin/pip-install-fhs-wrapper";
              builder = pkgs.runtimeShell;
              args = [ /*pkgs.runtimeShell*/ "-e" (pkgs.writeText "pip-downloader" script) ];
              script = ''
                export PATH=${pkgs.lib.makeBinPath deps}:${pkgs.lib.makeBinPath [pkgs.busybox]}:$PATH
                mkdir $out/
                poetry export -C ${sources} -E main | grep -v '@ file://' > ./reqs1.txt
                ${fhsWrapper}/bin/pip-install-fhs-wrapper pip download --prefer-binary -r ./reqs1.txt --cache-dir $out/cache -d $out/packages
                rm -rf $out/cache/selfcheck
              '';
              outputHashAlgo = "sha256";
              outputHash = "";
              outputHashMode = "recursive";
            });
            installer = derivation rec {
              name = "pip-install";
              system = "x86_64-linux";
              builder = pkgs.runtimeShell;
              args = [ "-e" (pkgs.writeText "pip-installer" install-script) ]; 
              install-script = ''
                export PATH=${pkgs.lib.makeBinPath deps}:${pkgs.lib.makeBinPath [pkgs.busybox]}:$PATH
                mkdir $out/
                poetry export -C ${sources} --without-hashes > $out/reqs.txt
                python -m venv $out/venv
                . $out/venv/bin/activate
                pip install --no-index --cache-dir ${downloader}/cache ${downloader}/packages/*
              '';
            };
            runner = pkgs.buildFHSUserEnv {
              name = "pip-runner";
              runScript = "python";
              profile = ''. ${installer}/venv/bin/activate'';
              targetPkgs = _: deps;
            };
            in runner;

le-chat · December 26, 2023, 11:40am

To use buildFHSUserEnv inside of derivation, I was required to add extraBwrapArgs = [ "--bind" "$out" "$out" ];. It looks a bit hacky, but works. Current version is below (./python-build-systems.txt lists all used python build-systems like hatchling, flit-core, poetry-core and so on and also oldest-supported-numpy, which is sometimes needed for building but missed in poetry.lock):

{ python, poetry2nix, poetry, buildFHSUserEnv, runtimeShell, linkFarm, writeShellScript, writeText, busybox, lib, system }: rec {

  poetryPackages2Sources = ps:
    let
      hasUrls = p: (p?src) && (p.src?url || p.src?urls) && (p.pname != "python3");
      mkName = p: p.src.file or "${p.pname}-${p.version}.tar.gz";
      getRM = p: [ p ] ++ builtins.filter hasUrls (p.requiredPythonModules or [ ]);
      allPS = builtins.concatMap getRM (builtins.filter hasUrls ps.poetryPackages);
      srcs = map (p: { name = mkName p; value = p.src; }) allPS;
    in
    linkFarm "poetry-sources" /*ps.name?*/ (lib.listToAttrs srcs);

  poetryPackageSources = { projectDir, python, extras, preferWheels }:
    let
      pps = poetry2nix.mkPoetryPackages {
        inherit projectDir python extras preferWheels;
        overrides = [ ];
      };
    in
    poetryPackages2Sources pps;

  allPythonBuildSystems = derivation {
    name = "pip-build-systems";
    inherit system;
    builder = writeShellScript "download-pip-build-systems" ''
      set -e
      export PATH=${lib.makeBinPath [ python.pkgs.pip busybox ]}:$PATH
      mkdir $out
      pip download --prefer-binary -r $requirements --no-cache -d $out/
    '';
    requirements = ./python-build-systems.txt;
    outputHashMode = "recursive";
    outputHash = "sha256-3Hu+qQARM0mMgh27+wC5xJuasItXJ7BOUH9CLwUWtWs=";
  };

  poetryExport = { projectDir, extras }: derivation rec {
    name = "poetry-export";
    inherit system;
    builder = runtimeShell;
    args = [ "-e" (writeText "poetry-export" script) ];
    script =
      let extraString = lib.concatStringsSep " " (map (e: "-E ${e}") extras);
      in ''
        export PATH=${lib.makeBinPath [busybox poetry]}:$PATH
        mkdir $out
        poetry export -C ${projectDir} --with dev --without-hashes ${extraString} | grep -v '@ file://' > $out/main.txt
        poetry export -C ${projectDir} --without-hashes ${extraString} > $out/main-local.txt
      '';
  };

  buildPoetryProject = { buildDeps, runDeps, projectDir, extras }:
    let
      buildDeps' = buildDeps ++ [
        (python.withPackages (ps: with ps; [
          pip
          wheel
          poetry-core
          flit
          flit-core
          pbr
          hatchling
          hatch-vcs
          setuptools
          setuptools-scm
        ]))
      ];
      fhsBuilder = buildFHSUserEnv {
        name = "fhs-builder";
        runScript = "$@";
        targetPkgs = _: buildDeps';
        extraBwrapArgs = [ "--bind" "$out" "$out" ];
      };
      requirements = poetryExport { inherit projectDir extras; };
      wheels = poetryPackageSources {
        inherit projectDir extras python;
        preferWheels = true;
      };
      tarballs = poetryPackageSources {
        inherit projectDir extras python;
        preferWheels = false;
      };
      installer = derivation rec {
        name = "pip-install";
        system = "x86_64-linux";
        builder = writeShellScript "pip-install" ''
          set -e
          export PATH=${lib.makeBinPath (buildDeps' ++ [busybox])}:$PATH
          mkdir $out/
          ${fhsBuilder}/bin/fhs-builder ${installInFHS}
        '';
        installInFHS = writeShellScript "pip-install-in-fhs" ''
          set -e
          python -m venv $out/venv
          . $out/venv/bin/activate
          pip install --no-index --no-cache-dir -f ${wheels} -f ${tarballs} -f ${allPythonBuildSystems} \
            -r ${requirements}/main-local.txt \
            --log ./pip.log
        '';
      };
      runner = buildFHSUserEnv {
        name = "fhs-python";
        runScript = "python";
        profile = ''. ${installer}/venv/bin/activate'';
        targetPkgs = _: runDeps ++ [ python ];
      };
    in
    runner // { inherit fhsBuilder requirements wheels tarballs installer runner; };
}

Far from perfect, but frees you from nix-specific patching of huge number of packages. To use cuda, you need cuda’s shared libraries in /run/opengl-driver/lib/ (copy or bind them there inside of docker, for example)