Need help with packaging python lib `postal` [solved]

Help
1

I’m trying to package a python package from pypi. The package use c99 and failed to compile. How do I specify -std=c99 in buildPythonPackage?

buildPythonPackage rec {
    pname = "postal";
    version = "1.1.9";
    
    src = py.fetchPypi {
      inherit pname version;
      sha256 = "2158fde6872d57c342b384ca1d0eadd562dc466a8a36f1160919e8189864b1b1";
    };
    doCheck = false;
    propagatedBuildInputs = [ pkgs.libpostal py.nose];
    
    meta = with lib; {
      homepage = "https://github.com/openvenues/pypostal";
      description = "Python bindings to libpostal for fast international address parsing/normalization";
      license = licenses.mit;
      maintainers = with maintainers; [  ];
    };
  };
2

What do they use internally to build the native components?

It probably needs to get fixed upstream to pass the correct flags to the compiler.

3

there’s a setup.py: pypostal/setup.py at 0e9f2d6b5ef72da796c446e15e10cc54596af44d · openvenues/pypostal · GitHub
You can see lines extra_compile_args=['-std=c99'] there.

4

Then the std is already set correctly, I think the bigger problem is the hardcoded library and include pathes. You might need to patch them to use correct locations

5

why do you think std is correct? It’s upstream build script and I don’t think buildPythonPackage will take that into account anyway. I believe there’s no hardcoded library problem and this will work once I find out how to tell buildPythonPackage to use c99.

6

The buildlog seems to support my claim:

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include -I/nix/store/8dxxjbiyxwkvh53q5kh6nydla2anacgi-python3-3.9.6/include/python3.9 -c postal/pyneardupe.c -o build/temp.linux-x86_64-3.9/postal/pyneardupe.o -std=c99

The -std is set correctly.

Can’t dig deeper as I’m on phone and building on remote PC via SSH.

7

Really appreciate your time. That’s a bug in my knowledge :wink: Looking at the log is helpful - thanks a lot for the tip! I’ll try to make a patch

8

I’ve added this patch:

    prePatch = ''
      substituteInPlace setup.py \
          --replace /usr/local/include "${pkgs.libpostal}/include" \
          --replace /usr/local/lib "${pkgs.libpostal}/lib"
''

so some modules got compiled. However, the last two remain failing. Here’s the error:

building 'postal._near_dupe' extension
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/nix/store/m4k48m5krs4prrp7jzwzi7ylw5h4y9pk>
postal/pyneardupe.c: In function ‘py_near_dupe_hashes’:
postal/pyneardupe.c:26:5: error: unknown type name ‘libpostal_near_dupe_hash_options_t’
   26 |     libpostal_near_dupe_hash_options_t options = libpostal_get_near_dupe_hash_default_options();
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

not sure how I can fix this…

9

looks like postal/pyneardupe.c is missing a type definition. Most likely <postal/libpostal.h> couldn’t be found, as that’s where this is defined

Do you mind printing the whole line with gcc?

10

here you are:

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC 
-I/nix/store/m4k48m5krs4prrp7jzwzi7ylw5h4y9pk-libpostal-1.1/include 
-I/nix/store/k0z9n599k02hab8qjjp3ljw065iwjcvg-python3-3.9.6/include/python3.9 
-c postal/pyneardupe.c -o build/temp.linux-x86_64-3.9/postal/pyneardupe.o 
-std=c99
11

I’m looking at the libpostal.h in nix store (from nixpkgs!) and it doesn’t seem to contain the string libpostal_near_dupe_hash_options_t…

12

The version in nixpkgs is 1.1 if I understand correctly. The corresponding tag is 5 years old. Perhaps there are newer releases without tags that have been overseen therefore?

Sadly documentation of both libpostalaas well as pypostal us very bad in that regard and doesn’t communicate versions/requirements well.

13

yeah, upstream seems to have overwritten that 1.1 tag, and nixpkgs is using older revision which is now tagged 1.1-alpha by upstream.

14

After switching to newer version of libpostal, the build seems to have no errors. Thank you all for helping out.

15

… I checked nixpkgs to see if it was at the latest release… I guess that’s fun that release management debt gets to appear in this fashion…

16

@myguidingstar do you have a working example? I am running into similar issues trying to build pypostal with poetry2nix. First the error that it can not include libpostal.h which I could resolve but now the error that setuptools is not in the environment. I currently have the following:

                postal = pyprev.postal.overridePythonAttrs (old: {
                  nativeBuildInputs = ((old.nativeBuildInputs or [ ]) ++ [
                    pyfinal.setuptools
                    final.libpostal
                  ]);
                  propagatedBuildInputs = [ pyfinal.nose];
                  prePatch = ''
                      substituteInPlace setup.py \
                          --replace /usr/local/include "${final.libpostal}/include" \
                          --replace /usr/local/lib "${final.libpostal}/lib"
                  '';

                    # preConfigure = ''
                    #   export CFLAGS="$CFLAGS -I${final.libpostal.src}/src"
                    # '';
                });
17

I no longer use it, and this piece is extracted from a big blob from the time I even don’t understand overlay, so please help yourself. Also, the revision must be quite old…

{ lib, py, pkgs }:
let libpostal = pkgs.libpostal.overrideAttrs (_: {
      src = pkgs.fetchFromGitHub {
        owner = "openvenues";
        repo = "libpostal";
        rev = "9c975972985b54491e756efd70e416f18ff97958";
        sha256 = "sha256-7G/CjYdVzsrvUFXGODoXgXoRp8txkl5SddcPtgltrjY=";
      };
    });
in
{ postal = py.buildPythonPackage rec {
    pname = "postal";
    version = "1.1.9";

    src = py.fetchPypi {
      inherit pname version;
      sha256 = "2158fde6872d57c342b384ca1d0eadd562dc466a8a36f1160919e8189864b1b1";
    };
    doCheck = false;
    propagatedBuildInputs = [ libpostal py.nose py.six];
    prePatch = ''
      substituteInPlace setup.py \
          --replace /usr/local/include "${libpostal}/include" \
          --replace /usr/local/lib "${libpostal}/lib"
    '';
    meta = with lib; {
      homepage = "https://github.com/openvenues/pypostal";
      description = "Python bindings to libpostal for fast international address parsing/normalization";
      license = licenses.mit;
      maintainers = with maintainers; [  ];
    };
  };
...
1 Like