How to set LD_LIBRARY_PATH for gcc and co according to the default of the current nix generation or do I have to set all path variables by hand for each python environment?
You will probably want to look into poetry2nix for setting up local environments
glibc is exported by default, but libstdc++.so is part of gcc’s lib, and not usually present.
A few possible solutions: nix-shell -p steam-run --command "steam-run bash" is a quick and dirty way to get an FHS.
poetry2nix is probably a better solution that you’re looking for.
Nixpkgs has it own python packages as well:
$ nix-shell --pure -p python3Packages.pandas
$ python
Python 3.8.3 (default, May 13 2020, 19:59:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>>
Looks like another approach is to include stdenv.cc.cc.lib as a build input with autoPatchelfHook
I would set that up in a shell.nix file or build derivation file for your Python application.
buildInputs = [
stdenv.cc.cc.lib
pam
];
Searching through discourse.nixos.org also proves a lot of interesting responses; but to be honest mucking with LD_LIBRARY_PATH outside of Nix isn’t very “nixy”
Using ld_library_path to override the version of libstd to be used can affect other applications as well. Some will not operate correctly because it is now using an incompatible version of libstd. It is a good stopgap though, but when working with a team it will cause problems eventually.
The reason pandas needs libstd is because poetry will download a prebuilt library. That library was built against Debian(?) or some other distro. It is prone to become incompatible when libstd is placed somewhere else or when other versions of libstd are available. libstd not being found is the result of that.
With pip you can skip downloading prebuilt binaries and thus force building all packages with c bindings locally. It is done using --no-binary.
When building the package you do need to add all build tools and libraries to your shell. This means adding gcc, make, zlib and maybe others to your shell environment.
Poetry2nix does all of this automatically for many python packages. It is generally a good solution, but you cannot use the poetry cli as freely anymore. Still though, it does seem like the most stable solution.
In your example, none of poetry2nix is being used inside the devShell. You could try to add the mkPoetryApplication package to packages inside mkShell. That will make your application available as an executable. It should allow you to run myapp in your Nix shell. This is how your use your project as a standalone application.
I completely agree with you that mkPoetryEnv is a more elegant solution and ld_library_path may have side effects.
However, I often needed to override packages to add a build system dependency or to add it to build-systems.json when I tried to use mkPoetryEnv. It also takes time to compile as there is no binary cache (often had to recompile mypy, for example), at least to my knowledge.
installer.no-binary didn’t work for me when installing numpy immediately. Since it’s kind of slow I didn’t want to debug build errors and went with this option:
...
}: let
customPoetry = pkgs.writeShellScriptBin "poetry" ''
# Python packages come pre built and sometimes assume there are c libraries available on the system
# Usually because they are but the nix way is to isolate packages by default so we need to provide
# the right LD_LIBRARY_PATH to get it to work
# (Previously I did this for my whole environment but that predictably breaks stuff so don't do that)
# links
# https://www.reddit.com/r/NixOS/comments/1dexghb/im_just_about_done_with_nixos_how_do_i_get_a/
# https://discourse.nixos.org/t/what-package-provides-libstdc-so-6/18707/3
export LD_LIBRARY_PATH=${pkgs.stdenv.cc.cc.lib}/lib/
export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath [ pkgs.zlib ]}:$LD_LIBRARY_PATH"
exec ${pkgs.poetry}/bin/poetry $@
'';
in {
home.packages = [
customPoetry
...
To use the LD_LIBRARY_PATH trick just for poetry. This is a snippet from my home.nix setup but you could also expose customPoetry in a nix shell or some other way.