MacOS, dylib and install_name_tool

I’m working on fixing darwin support for the pjsip package. It mostly works, but for the python library (in ${pjsip.py}/lib you still need DYLIB_LIBRARY_PATH=${pjsip.out}/lib.

Without DYLIB_LIBRARY_PATH I currently get an error like:

    File "<string>", line 1, in <module>
    File "/nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env/lib/python3.10/site-packages/pjsua2.py", line 17, in <module>
      _pjsua2 = swig_import_helper()
    File "/nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env/lib/python3.10/site-packages/pjsua2.py", line 16, in swig_import_helper
      return importlib.import_module('_pjsua2')
    File "/nix/store/0c9ksx9i8mfx6n3011scyh80waz0dfc4-python3-3.10.9/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ImportError: dlopen(/nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env/lib/python3.10/site-packages/_pjsua2.cpython-310-darwin.so, 2): Library not loaded: ../lib/libpjsua2.dylib.2
    Referenced from: /nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env/lib/python3.10/site-packages/_pjsua2.cpython-310-darwin.so
    Reason: image not found

Notice the ../lib/libpjsua2.dylib.2 that it tries to load.

Using otool I found that relative paths are still being used even if fixDarwinDylibNames was added to the nativeBuildInputs. For example:

$ otool -L result/lib/libpjsua2.dylib
result/lib/libpjsua2.dylib:
	../lib/libpjsua2.dylib.2 (compatibility version 0.0.0, current version 0.0.0)
	../lib/libpjsua.dylib.2 (compatibility version 0.0.0, current version 0.0.0)
	../lib/libpjsip-ua.dylib.2 (compatibility version 0.0.0, current version 0.0.0)

Still ../lib references instead of /nix/store/ ones.

Same for the python library:

otool -L result-py/lib/python3.10/site-packages/_pjsua2.cpython-310-darwin.so
result-py/lib/python3.10/site-packages/_pjsua2.cpython-310-darwin.so:
	../lib/libpjsua2.dylib.2 (compatibility version 0.0.0, current version 0.0.0)
	/nix/store/q4ipnw0nq98dff6xli2wv0w7wag49a8q-libcxx-11.1.0/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	../lib/libpjsua.dylib.2 (compatibility version 0.0.0, current version 0.0.0)
	../lib/libpjsip-ua.dylib.2 (compatibility version 0.0.0, current version 0.0.0)
	../lib/libpjsip-simple.dylib.2 (compatibility version 0.0.0, current version 0.0.0)

When I manually use install_name_tool to fix the names it won’t change anything:

install_name_tool -change libpjsip-ua.dylib /nix/store/9dm86lxa0cicn92zw68brxni99wqfxan-pjsip-2.13/lib/libpjsip-ua.dylib _pjsua2.cpython-310-darwin.so

Only when I use the exact relative path that is used in the .so it’ll work:

install_name_tool -change ../lib/libpjsip-ua.dylib.2 /nix/store/9dm86lxa0cicn92zw68brxni99wqfxan-pjsip-2.13/lib/libpjsip-ua.dylib _pjsua2.cpython-310-darwin.so

This doesn’t seem to be something that fixDarwinDylibNames supports. And manually doing this for all pjsip libraries also doesn’t seem like a robust way to fix it.

I was thinking this would be a common problem with fixDarwinDylibNames, but I’m not sure.

One alternative that I thought was going to help was adding a rpath. Like:

install_name_tool -add_rpath /nix/store/9dm86lxa0cicn92zw68brxni99wqfxan-pjsip-2.13/lib _pjsua2.cpython-310-darwin.so

Using otool I see the rpath being correct:

$ otool -l _pjsua2.cpython-310-darwin.so
...
          cmd LC_RPATH
      cmdsize 72
         path /nix/store/9dm86lxa0cicn92zw68brxni99wqfxan-pjsip-2.13/lib (offset 12)

But this doesn’t seem to make the dynamic loader use the path as it still fails to load ../lib/libpjsua2.dylib.2 even though /nix/store/9dm86lxa0cicn92zw68brxni99wqfxan-pjsip-2.13/lib/../lib/libpjsua2.dylib.2 does exist.

To reproduce the error, you can use:

nix build github:bobvanderlinden/nixpkgs/c2b482ea30328fadcba4db51a2205476f6aca4e0#pjsip.tests.python-pjsua2

Does anyone know a good way to make this work for packaging this for darwin?