yrd
November 22, 2022, 12:20pm
1
I am trying to package pillow-avif-plugin from Pypi, which includes some C extensions depending on libavif . This is what I’ve got so far:
{ buildPythonPackage
, fetchFromGitHub
, cython
, libavif
, libjpeg
, pkgconf
, pytestCheckHook
, pillow
}:
buildPythonPackage rec {
pname = "pillow-avif-plugin";
version = "1.3.1";
src = fetchFromGitHub {
owner = "fdintino";
repo = "pillow-avif-plugin";
rev = "v${version}";
sha256 = "vmVu19WqThe9T4QXTJ9VSqSvQwxTKcuPBUC6An8q0oU=";
};
nativeBuildInputs = [ pkgconf ];
buildInputs = [ cython libavif ];
propagatedBuildInputs = [ pillow ];
doCheck = false;
}
This builds alright, but running it gives me this error (which is the reason doCheck is off):
>>> import pillow_avif._avif
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /nix/store/cgk8nbxafs8l1igpyvnx0nrymv5hbabk-libyuv-1787/lib/libyuv.so: undefined symbol: jpeg_resync_to_restart
If anyone more experienced in Python Packaging then me has some pointers here that would be greatly appreciated.
voydus
November 22, 2022, 1:46pm
2
I added the following flake.nix to make it buildable
{
description = "A very basic flake";
outputs = { self, nixpkgs }:
let
pkgs = nixpkgs.legacyPackages.x86_64-linux;
in {
packages.x86_64-linux = {
default =
pkgs.python310Packages.callPackage ./default.nix {};
python = pkgs.python310.withPackages (ps: [self.packages.x86_64-linux.default]);
};
};
}
default.nix is your code. I could reproduce your error.
The symbol that’s missing is from libjpeg , which nix-tree tells me is pulled in by libyuv as libjpeg-turbo.
libjpeg-turbo seems to re-implement the libjpeg api and should probably have the symbol: jpeg_resync_to_restart (libjpeg.jpeglib.jpeg_resync_to_restart)
weird that it’s not found.
voydus
November 22, 2022, 1:55pm
3
The symbol does exist in the .so file though:
$ nm -gD /nix/store/ip2qihjd99hcdnr1vpp53dxz5dfsl5fd-libjpeg-turbo-2.1.4/lib/libjpeg.so | grep resync
00000000000390c0 T jpeg_resync_to_restart@@LIBJPEG_6.2
So I’d assume it’s not loaded properly. Let’s follow the chain
> nix build . && ldd result/lib/**/*.so
linux-vdso.so.1 (0x00007ffd28e5f000)
libavif.so.14 => /nix/store/k3mva734wph7s6bwlvin9h2amgaq4kjx-libavif-0.10.0/lib/libavif.so.14 (0x00007f244e317000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f244e10e000)
libdav1d.so.6 => /nix/store/xpj1x7rx0xnk36rg4g2lm8imvwa28f9v-dav1d-1.0.0/lib/libdav1d.so.6 (0x00007f244df47000)
libaom.so.3 => /nix/store/lxpnl9hf8hdqvc6sqfw5l30wi4ia2p91-libaom-3.5.0/lib/libaom.so.3 (0x00007f244d6eb000)
libvmaf.so.1 => /nix/store/6m7mhkm4y3f2yd070sckb3ldf1lj3lqm-libvmaf-2.3.1/lib/libvmaf.so.1 (0x00007f244d657000)
libjxl.so.0.6 => /nix/store/dz8dk7ls09km01xmivi41hzpqyazpc9r-libjxl-0.6.1/lib/libjxl.so.0.6 (0x00007f244d315000)
libm.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libm.so.6 (0x00007f244d235000)
libyuv.so => /nix/store/14i7g28wynjzx492q0d28i16acycgrvw-libyuv-1787/lib/libyuv.so (0x00007f244d158000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f244d151000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f244e347000)
libstdc++.so.6 => /nix/store/mdck89nsfisflwjv6xv8ydj7dj0sj2pn-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007f244cf3b000)
libgcc_s.so.1 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libgcc_s.so.1 (0x00007f244cf21000)
libbrotlidec.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlidec.so.1 (0x00007f244cf13000)
libbrotlicommon.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlicommon.so.1 (0x00007f244cef0000)
libbrotlienc.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlienc.so.1 (0x00007f244ce4d000)
> ldd /nix/store/k3mva734wph7s6bwlvin9h2amgaq4kjx-libavif-0.10.0/lib/libavif.so.14
linux-vdso.so.1 (0x00007fff00bd0000)
libdav1d.so.6 => /nix/store/xpj1x7rx0xnk36rg4g2lm8imvwa28f9v-dav1d-1.0.0/lib/libdav1d.so.6 (0x00007f4245652000)
libaom.so.3 => /nix/store/lxpnl9hf8hdqvc6sqfw5l30wi4ia2p91-libaom-3.5.0/lib/libaom.so.3 (0x00007f4244df8000)
libvmaf.so.1 => /nix/store/6m7mhkm4y3f2yd070sckb3ldf1lj3lqm-libvmaf-2.3.1/lib/libvmaf.so.1 (0x00007f4244d64000)
libjxl.so.0.6 => /nix/store/dz8dk7ls09km01xmivi41hzpqyazpc9r-libjxl-0.6.1/lib/libjxl.so.0.6 (0x00007f4244a22000)
libm.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libm.so.6 (0x00007f4244940000)
libyuv.so => /nix/store/14i7g28wynjzx492q0d28i16acycgrvw-libyuv-1787/lib/libyuv.so (0x00007f4244863000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f424485e000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f4244655000)
libstdc++.so.6 => /nix/store/mdck89nsfisflwjv6xv8ydj7dj0sj2pn-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007f424443f000)
libgcc_s.so.1 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libgcc_s.so.1 (0x00007f4244423000)
libbrotlidec.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlidec.so.1 (0x00007f4244415000)
libbrotlicommon.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlicommon.so.1 (0x00007f42443f2000)
libbrotlienc.so.1 => /nix/store/9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib/lib/libbrotlienc.so.1 (0x00007f4244351000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f4245842000)
> ldd /nix/store/14i7g28wynjzx492q0d28i16acycgrvw-libyuv-1787/lib/libyuv.so
linux-vdso.so.1 (0x00007ffc8b3df000)
libstdc++.so.6 => /nix/store/mdck89nsfisflwjv6xv8ydj7dj0sj2pn-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007ff364644000)
libm.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libm.so.6 (0x00007ff364564000)
libgcc_s.so.1 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libgcc_s.so.1 (0x00007ff36454a000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007ff364341000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007ff364939000)
Hm. No libjpeg, that seems wrong?
I’m starting to go out of my depth here but my gut says this is the right direction. Does somebody see something?
1 Like
trofi
November 22, 2022, 10:45pm
4
Nice debugging! Indeed it’s an underlinking bug in libyuv. I usually validate those by passing --no-undefined to the linker when building libraries:
$ nix build -L --impure --expr 'with import <nixpkgs> {}; libyuv.overrideAttrs (oa: { NIX_LDFLAGS = "--no-undefined"; })'
...
libyuv> [ 98%] Built target yuv
libyuv> [ 99%] Building CXX object CMakeFiles/yuvconvert.dir/util/yuvconvert.cc.o
libyuv> /nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld: CMakeFiles/yuv_shared.dir/source/mjpeg_decoder.cc.o: in function `libyuv::MJpegDecoder::MJpegDecoder()':
libyuv> mjpeg_decoder.cc:(.text+0xfc): undefined reference to `jpeg_resync_to_restart'
libyuv> /nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld: mjpeg_decoder.cc:(.text+0x136): undefined reference to `jpeg_std_error'
libyuv> /nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld: mjpeg_decoder.cc:(.text+0x194): undefined reference to `jpeg_CreateDecompress'
libyuv> /nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld: CMakeFiles/yuv_shared.dir/source/mjpeg_decoder.cc.o: in function `libyuv::MJpegDecoder::UnloadFrame()':
libyuv> mjpeg_decoder.cc:(.text+0x3a1): undefined reference to `jpeg_abort_decompress'
It clearly needs a form of libjpeg.
libyuv: fix libyuv.so not linking against libjpeg by MidAutumnMoon · Pull Request #201857 · NixOS/nixpkgs · GitHub has a fix lying around to handle it. Let’s pull it as is.
Meanwhile you can try to add a NIX_CFLAGS_LINK = [ "-ljpeg" ]; (and provide according build input) to your derivation to work around lack of library in depends.
2 Likes
yrd
November 23, 2022, 9:49am
5
Awesome - thanks both of you!
Adding NIX_CFLAGS_LINK as suggested above as well as including pkgs.libjpeg in buildInputs indeed seems to work.
Alternatively, I also had success with LD_PRELOAD=/nix/store/pn0sgk4i283n2vzbrspnmwky0b1fwj6c-libjpeg-turbo-2.1.4/lib/libjpeg.so.62.3.0 at runtime.
1 Like
