Mixed Python/C++ with Python runtime dependencies

Help
1

Dear all,
i am trying to write some nix derivations for some computational chemistry software, including some python applications and libraries. Namely i have a problem with an application, that is written in a mixture of C++ and python, called Psi4.
I have written a derivation for some of its python dependencies, e.g. for the python module qcelemental

{ buildPythonPackage, lib, fetchPypi
# Other dependencies
, cacert
# Python dependencies
, numpy
, pydantic
, pint
, networkx
, pytestrunner
, pytestcov
}:
buildPythonPackage rec {
    # Skipping the rest here for clarity

    propagatedBuildInputs = [
      numpy
      pydantic
      pint
      networkx
      cacert
    ];
}

and i call it from my default.nix as

let
  nixpkgs = import <nixpkgs> { config = packageOverrides; };
  allPkgs = nixpkgs // pkgs;
  callPackage = nixpkgs.callPackage;
  pkgs = with allPkgs; rec {
    psi4 = python3Packages.callPackage ./pkgs/apps/psi4/default.nix { inherit qcelemental; };
    qcelemental = python3Packages.callPackage ./pkgs/libs/python/qcelemental/default.nix { inherit pydantic; pint = pint-0_10; };
    # ...
  }
in pkgs

The first thing that puzzles me, is that i can’t import the qcelemental module from a nix-shell, where my shell.nix something simple as :

let pkgs = import ./default.nix;
in pkgs.qcelemental

A python interpreter within that nix-shell only tells me, that a qcelemental package could not be found. Its propagatedBuildInputs can be found without any problems and a nix-shell -p python3Packages.numpy for example also behaves as i would expect, allowing me to import numpy.
Now if i use my qcelemental derivation as a propagatedBuildInput for my Psi4 derivation

{ stdenv, buildPythonApplication, buildPackages, fetchFromGitHub
# Build time dependencies
, cmake
, perl
, gfortran
, makeWrapper
# Runtime dependencies
, python3
, mkl
, gau2grid
, libint
, libxc
, qcelemental
}:
let
  python = python3.withPackages (ps : with ps; [ pybind11 ]);
in stdenv.mkDerivation rec {
    pname = "psi4";
    version = "1.3.2";

    nativeBuildInputs = [
      cmake
      perl
      gfortran
    ];

    buildInputs = [
      gau2grid
      libint
      libxc
      mkl
    ];

    propagatedBuildInputs = [
      python
      qcelemental
    ];

   # Rest skipped 
}

and try this one in a nix-shell, the import of qcelemental in a python interpreter works, and the Psi4 executable also works fine. However, installing the psi4 derivation with nix-env -f default.nix -iA psi4 gives a program, that cannot find the qcelemental module (and i suspect also other python modules).

So i am puzzled by two things:

  • Why are python packages available in a nix-shell, that were given with -p but not from a shell.nix?
  • How would i tell a mixed application of C++ and Python like Psi4, how and where to find all its python runtime modules, when building with stdenv.mkDerivation?

Thank you for any help and suggestions in advance
Best wishes
Phillip

2

thanks for bringing nix to science!

I think your shell.nix isn’t correct. The way you wrote it it will give you development environment to build a package, not to use. If you want to use, you should do

with import <nixpkgs> {};
mkShell {
  buildInputs = with import ./.; [ 
    qcelemental 
  ];
}

Alternatively, nix run -f default.nix qcelemental will also give you a quick temporary shell with your package available. This is 2.0 alternative for nix-shell -p

In this case you should debug, where does it try to search the python package in runtime. Maybe it requires some proper configure arg? In case you stuck, the last resort is to add a so called “wrapper”, thing which puts correct PYTHONPATH variable inside the binary.

Nixpkgs is all about wrappers :slight_smile:

2 Likes
3

I made a video about exposing python packages through nix-shell, i think it would help you a lot:

4 Likes
4

Thank you both for your advice. nix-shell does what i wanted to with the mkShell. This makes actually a lot of sense.

For the Psi4 application the wrapper suggestion was the solution:

python = python3.withPackages (ps : with ps; [ pybind11 qcelemental numpy pylibefp ]);
preFixup = ''
      wrapProgram $out/bin/psi4 \
        --prefix PYTHONPATH : ${python}/lib/${python.executable}/site-packages \
    '';

and the python dependencies are available at runtime to the executable now. :slight_smile:

2 Likes
5

@sheepforce I have started to package QC software in an overlay. Maybe you find some inspiration there.

1 Like
6

To make it a little cleaner, you can use:

${python}/${python.sitePackages} \
1 Like
7

(I found this page by googling for NixOS + Psi4)

It would be great to have Psi4 in nixpkgs or in the NixOS-QChem overlay.

8

I mean nix is a pretty good foundation to do what ever you want. You can take a look at GitHub - lopsided98/nix-ros-overlay: ROS overlay for the Nix package manager which seems to be a similar goal.

Flakes should also make it easier for people to “hook” into what you’re doing.

9

I have this in the overlay for our group since some time. It might not be as nice as the Q-Chem overlay, but gets its job done and has a few other packages, that are not in Q-Chem. Here is mine, if you need it now: Chemix.

If the Nixpkgs developers are interested I can try to submit a PR, but the Psi4 dependencies are heavy. If @markuskowa is interested I can also try to submit a PR to his Q-Chem overlay.

10

@sheepforce I am happy to take it into QChem overlay or review it if you make a nixpkgs pull request.
We could start with QChem, since I already have chemps2 packaged, what other libs does it require that are not upstream yet?

On a general note: my philosophy for the QChem repository is the following. I use it mainly to mature derivations, keep derivations that are not yet upstream quality (or never will be of upstream quality), and to make performance related overrides of upstream packages. However, I try to upstream as much as possible and keep it lean otherwise since it is easier to maintain that way.
I am rewriting the top-level structure right now, which hopefully makes overrides of system libraries less painful and will also allow to upgrade the repository to flakes more easily. Once PR #108983 (feel review to leave a review) is merged I will push an update to NixOS-QChem/master.

11

In this case I think it is a good way to first submit to your Q-Chem repo :slight_smile: I am not too familiar with the Nixpkgs internals and, as I said, it will probably require multiple additions.

Dependencies that I use for Psi4, but that are missing in Nixpkgs:

  • QCElemental (Python library with chemical data)
    • QCEngine
  • LibEFP and PyLibEFP (enables the effective fragment potential calculations in Psi4)
  • LibInt (v1 for Psi4 1.3.x, v2 for new Psi4 1.4.x) (but I can see you have it in QChem)
  • Gau2Grid (required for DFT)
  • LibXC 4.3.3
  • DKH program (for relativistic hamiltonians)
  • DFTD3 (required for DFT. I have no idea which license it has …)
  • PCMSolver
  • (completely optionally: MRCC)

One can drop most dependencies but this makes Psi4 a little unfunctional.
I had some struggle with many of the dependencies, as Psi4 often pins a single commit from some Git repo, that contains some very specific CMake fixes and I could really not get it to work with the standard releases of those packages. I also needed to tune many CMake flags of Psi4’s dependencies, as otherwise Psi4’s cmake does not find them. It required a lot of pinning and overrides specific for Psi4, but it is very similar to what Psi4 is doing in their CMake.

Two comments on Psi4 itself (for both see this thread in the Psi4 forum):

  • the latest stable version is 1.3.2, which contains a serious bug (affecting correctness of results). Psi4 1.4 is not released yet, therefore I am using some snapshot from Psi4 master
  • Psi4 has serious threading problems with any BLAS/LAPACK implementation besides MKL. Upstream is aware of it but it doesn’t look as if this is going to change soon.

That being said, I will make a PR to QChem later and maybe we can discuss there if there is a cleaner way to build Psi4.

12

It is merged into QChem now: Merge Request #6 :slightly_smiling_face:

1 Like