with import <nixpkgs> {};
let
python = python36;
in
pkgs.mkShell {
name = "pip-env";
buildInputs = with pkgs; [
python36Packages.ipython
python36Packages.virtualenv
git
bash
docker
];
src = null;
shellHook = ''
# Allow the use of wheels.
SOURCE_DATE_EPOCH=$(date +%s)
VENV=dev36env
if test ! -d $VENV; then
virtualenv $VENV
fi
source ./$VENV/bin/activate
export PYTHONPATH=`pwd`/$VENV/${python.sitePackages}/:$PYTHONPATH
'';
}
I use direnv so after entering my project directory it seems that I have everything and I can install the project in my virtualenv:
$ which python
/home/little-dude/xain-fl/dev36env/bin/python
$ pip install -e '.[dev]'
...
things seem to install correctly
...
Things look good, my virtualenv is here, and my package is installed. However, at runtime, it falls apart:
$ pytest -v
ImportError while loading conftest '/home/little-dude/xain-fl/tests/conftest.py'.
tests/conftest.py:3: in <module>
import grpc
dev36env/lib/python3.6/site-packages/grpc/__init__.py:23: in <module>
from grpc._cython import cygrpc as _cygrpc
E ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
So here I’m not sure what to do. I think that somehow I need to include my runtime dependency with propagateBuildInput, but I think that I also need to use patchelf although I don’t really understand how and on which binary exactly. Could someone enlighten me please?
you could do two things, either allow for grpc to link against libstdc++, or package the grpc package (not available in nixpkgs, and is abandoned upstream).
Thank you @jonringer. That is much simpler than I expected. You’re right, I’m actually using the grpcio package, not grpc. However I’m not sure whether you made a typo, but I get:
while evaluating the attribute 'lib.makeLibPath' at /nix/store/xxjjwq2askazv4d4q72gh4jqhz0jypcx-nixos-20.03pre209250.7184df6beb8/nixos/pkgs/top-level/all-packages.nix:61:10:
attribute 'makeLibPath' missing, at /home/little-dude/xain-fl/shell.nix:27:30
Yeah but if I do that, grpcio won’t be installed in the same virtualenv than the rest of the package. It should not be a problem but just in case I prefer your solution with LD_LIBRARY_PATH.
Thank you so much @jonringer without your help and the rest of the community I’d probably have given up on NixOS Seeing how to fix these small issues is really helpful for learning and I’m starting to really enjoy my new setup.
i resolved this issue by a simple installation of the clangStdenv or gccStdenv package globally (or in the nix shell you work in). Choose one you like, you are not able to install both since they clash each other. This also solved ton of other problems i had where programs were linking to libstdc++
Also if someone tries the export LD_LIBRARY_PATH version from @jonringer, which is also working btw, make sure to use lib.makeLibraryPath instead of lib.makeLibPath. Guess it is just a typo
I’ve been using the same technique in a flake.nix, and it was fine until recently where I updated my nixpkgs in my flake w/o updating the underlying system (which is also NixOS). Now I get the following error when using git inside the nix develop environment:
git: /nix/store/4s21k8k7p1mfik0b33r2spq5hq7774k1-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.34' not found (required by /nix/store/8dn12i3d7harw8g7dzk6dy7c5diz5ibp-gcc-11.3.0-lib/lib/libgcc_s.so.1)
So I guess one really should be specifying Python packages inside flake.nix/shell.nix as much as possible, and the virtualenv is only useful for python packages that doesn’t require ANY additional libraries other than python itself. Otherwise there might be surprises when one updates the flake/system.
We should really “solve” the glibc issue. It’s the only case in which nix isn’t freely able to pick and choose binaries because of how dynamic linking is done
maybe we should do something similar for NixOS? and as part of a generation’s activate, that we create a /etc/ld-nix.so.preload with the system’s glibc. Wouldn’t help people using stable and picking from unstable, but it would help with people upgrading their system and then getting the error.
Arch never gets into trouble of loading two libc.so.6 versions. Old and new version are both present on disk as a symlink with the same name: /lib64/libc.so.6. ELF binaries use SONAME to load library and thus avoid encoding specific libc version. Dynamic loader does not have a reason to load libc.so.6 twice.
The ldconfig call you linked is needed to regenerate ld.so.cache binary cache. It’s used to speed up resolution of library names (like libc.so.6) to full path (like /lib/libc.so.6). For this specific library ldconfig run is not needed as path never changes. ldconfig run is important when glibc adds or removes other .so libraries (like recently removed libpthread.so).
nixpskgs’ glibc does not populate cache at all and resolves libraries afresh every time. Multiple libc.so.6 versions available at different paths is a nixpkgs-specific issue. But I think it’s only an issue when you use LD_LIBRARY_PATH to load a library not compatible to the binary you are about to run. It should not happen on it’s own (If we ignore /run/opengl-driver and /run/current-system/sw/lib/kde4/plugins).
Seeing how to fix these small issues is really helpful for learning and I’m starting to really enjoy my new setup.