Python3 and openjdk use different glibc (packaging jep fails)

Help
1

Hello everyone,

I try to build the jep package, which is a a bridge library between java and python. The goal is in the end to try to make a polynote (notebook framework written in scala) service for NixOS. It has jep as a dependency:

https://github.com/scalavision/nixpkgs/blob/c5e1a03205f6661ac35c8c2e184af88a19a20f7d/pkgs/development/python-modules/jep/default.nix

First I build the package with: nix-build -A pythonPackage.jep

Then I try to run jep with this command from nixpkgs folder:

java -Djava.library.path=./result/lib/python3.8/site-packages/jep \
  -cp ./result/lib/python3.8/site-packages/jep/jep-3.9.1.jar \
 -jar ./result/lib/python3.8/site-packages/jep/jep-3.9.1.jar \
 jep.Run

I get this error:

java.lang.UnsatisfiedLinkError: /nix/store/s78py3n2pzhiyi8qy9hwjngkv89n88qj-python3.9-jep-3.9.1/lib/python3.9/site-packages/jep/libjep.so: /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6: version `GLIBC_2.32' not found (required by /nix/store/0ipynhzs670xbc9hixy05wgfd2xad2v2-python3-3.9.0/lib/libpython3.9.so.1.0)
        at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
        at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452)
        at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508)
        at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2669)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:807)
        at java.base/java.lang.System.loadLibrary(System.java:1907)
        at jep.MainInterpreter.initialize(MainInterpreter.java:128)
        at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
        at jep.Jep.<init>(Jep.java:256)
        at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)
        at jep.Run.run(Run.java:49)
        at jep.Run.main(Run.java:146)

I have tried with python 3.6 and python 3.7 using both openjdk 8 and openjdk 11, but none of these have matching glibc versions. This is a bit surprising to me. Is there a way to override the glibc version?

I also wonder how I actually set that only python3 should be used. I didn’t find any other way than using python3.pkgs.buildPythonPackage, that doesn’t feel correct.

2

I’m assuming some odd pinning is going on, because on unstable, it’s currently at glibc-2.32, and python3 points to python3.8 not python3.9

[10:40:01] jon@nixos ~/projects/nixpkgs (master)
$ ldd $(nix-build -A python39)/bin/python3.9
	linux-vdso.so.1 (0x00007ffff6b22000)
	libpython3.9.so.1.0 => /nix/store/0ipynhzs670xbc9hixy05wgfd2xad2v2-python3-3.9.0/lib/libpython3.9.so.1.0 (0x00007f4573cf6000)
	libpthread.so.0 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libpthread.so.0 (0x00007f4573cd5000)
	libdl.so.2 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libdl.so.2 (0x00007f4573cd0000)
	libcrypt.so.1 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libcrypt.so.1 (0x00007f4573c96000)
	libncursesw.so.6 => /nix/store/jxk39sgwpfnsfb00dsff5p96mxaj6m33-ncurses-6.2/lib/libncursesw.so.6 (0x00007f4573c22000)
	libutil.so.1 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libutil.so.1 (0x00007f4573c1b000)
	libm.so.6 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libm.so.6 (0x00007f4573ad8000)
	libgcc_s.so.1 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libgcc_s.so.1 (0x00007f4573abe000)
	libc.so.6 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/libc.so.6 (0x00007f45738fd000)
	/nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/ld-linux-x86-64.so.2 => /nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib64/ld-linux-x86-64.so.2 (0x00007f45740ba000)
3

Also, when adding packages to python-modules, please follow https://github.com/NixOS/nixpkgs/blob/8496b8426e104b79986713fb0b070f0f62abfaff/doc/languages-frameworks/python.section.md#building-packages-and-applications

I also go over python packaging with nix in a video, located: https://www.youtube.com/watch?v=jXd-hkP4xnU

EDIT:
namely, you should not have python3 as an import, you should be importing the python packages individually. using the python3 package will fix the python interpreter to python3.8, which is likely why you’re hitting these issues.

1 Like
4

Thanks for the feedback!

I think I might mixed up pasting output from trying with python 3.9, whereas the sources are showing python 3.8. I think both python 3.8, and 3.9 required glibc 1.32, while openjdk uses 1.31 it seems.

I will read through the docs again and try more tomorrow :slightly_smiling_face:

5

Where does the java you’re executing the jar with come from? If it’s from your system and you’re e.g. on 20.09 you’ll have an openjdk here with glibc-2.31 (or in other words: you don’t have the one from your buildInputs).

To confirm my hypothesis can you please open a nix-shell with an openjdk from your branch (nix-shell -I nixpkgs=$(pwd) -p openjdk) and run the java -D... command from that shell?

6

master is, but AFAICS currently nixpkgs-unstable (and nixos-unstable) aren’t: https://github.com/NixOS/nixpkgs/blob/cfed29bfcb28259376713005d176a6f82951014a/pkgs/development/libraries/glibc/common.nix#L44 - or am I looking at this the wrong way?

7

you’re probably right, I’ve largely abandoned channels as I’m constantly just using the branch to verify something works

8

Thanks a lot to everyone!

Yes @Ma27, you where right. I did use openjdk from my channel, and not from the branch I was in. Thank you for the tip of using nixpkgs=$(pwd) that solved the issue! Hehe, I should have thought about that, my nix skills are probably a bit rusty!

I encountered another issue now with java, but I think that is solvable.

1 Like