Nixos with poetry installed pandas - libstdc++.so.6: cannot open shared object file

Help
1

Hello community,

I have set up nixos and would like to start a project via poetry.

When

poetry new "${proj}"
cd              "${dirProj}"
poetry add pandas
poetry shell
python
import pandas

I get the error

ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

so there should be

/nix/store/dwc2rzj2k5fkh1xm79q3mn8hl2i3h4w1-gcc-9.2.0-lib/lib/libstdc++.so.6

It is only a path issue?

How to set LD_LIBRARY_PATH for gcc and co according to the default of the current nix generation or do I have to set all path variables by hand for each python environment?

currently there is only

echo $LD_LIBRARY_PATH
/nix/store/rwixw1zgn7s6amn5kwwrsg3pz6rf0jfi-sane-config/lib/sane

How to do it right?

4 Likes
2

You will probably want to look into poetry2nix for setting up local environments

glibc is exported by default, but libstdc++.so is part of gcc’s lib, and not usually present.

A few possible solutions:
nix-shell -p steam-run --command "steam-run bash" is a quick and dirty way to get an FHS.

poetry2nix is probably a better solution that you’re looking for.

Nixpkgs has it own python packages as well:

$ nix-shell --pure -p python3Packages.pandas

$ python
Python 3.8.3 (default, May 13 2020, 19:59:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>>
3 Likes
3

Looks like there is an entry: Packaging/Quirks and Caveats - NixOS Wiki
&
Packaging/Binaries - NixOS Wiki

Looks like another approach is to include stdenv.cc.cc.lib as a build input with autoPatchelfHook
I would set that up in a shell.nix file or build derivation file for your Python application.

 buildInputs = [
   stdenv.cc.cc.lib
   pam
 ];

Searching through discourse.nixos.org also proves a lot of interesting responses; but to be honest mucking with LD_LIBRARY_PATH outside of Nix isn’t very “nixy”

8 Likes
4

Hello jonringer,

I try to build a data science env.

well, if I wouldn’t want to do it witth nixos I can use ubuntu/FHS (with nix), right?

If I try to use poetry2nix there are errors I have no idea what to to with it:

error: --- EvalError ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-shell
in file: /nix/store/dzk0krx7hylcm14wcbhpmqw7pi0z6ll3-nixpkgs-20.03.2652.076c67fdea6/nixpkgs/pkgs/development/tools/poetry2nix/poetry2nix/mk-poetry-dep.nix (132:28)

attribute 'typing_extensions' missing

I’m not sure if I understood poetry2nix right … how to convert a project when python pkgs are not available in nixpkgs ?

5

i tried mach-nix
and it fails quite strange as well:

resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=Requirement.parse('dask==2.22.0'), parent=None), RequirementInformation(requirement=Requirement.parse('dask[complete]>=0.18.0'), parent=Candidate(name='datashader', ver=<Version('0.10.0')>, ex
tras=()))]
builder for '/nix/store/d5dnw42d7jfhdqmzh35cybak8w91jpkh-mach_nix_file.drv' failed with exit code 1
error: --- Error --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
build of '/nix/store/d5dnw42d7jfhdqmzh35cybak8w91jpkh-mach_nix_file.drv' failed
6

Hello jonringer

I found your post
https://www.reddit.com/r/NixOS/comments/fvfsig/pythonjupyter_development_environment_in_nixos/

how would adopt it for jupyterlab and you make jupyter lab build succeed?

  • it’s not working in postShellHook (neither with sudo )
1 Like
7

Seems that it fails because it tries to write to a nix store path

[LabBuildApp] Building in /nix/store/a4s5qgbakdzqfh8j9c9pyhzy6v9hici2-python3.8-jupyterlab-2.1.5/share/jupyter/lab
Build failed.

I’m unfortunately not aware of a way to pass another build directory.

8

I’m using poetry2nix with flakes and still getting this error? I’m not sure what to try at this point but this is my flake.nix

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.poetry2nix = {
    url = "github:nix-community/poetry2nix";
    inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        inherit (poetry2nix.legacyPackages.${system}) mkPoetryApplication;
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        packages = {
          myapp = mkPoetryApplication { projectDir = self; };
          default = self.packages.${system}.myapp;
        };

        devShells.default = pkgs.mkShell {
          buildInputs = [pkgs.stdenv.cc.cc.lib];
          packages = [ poetry2nix.packages.${system}.poetry ];
        };
      });
}
9

For the shell, I add

  shellHook = ''
    export LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath [
      pkgs.stdenv.cc.cc
    ]}
  '';

to mkShell. (with stdenv.cc.cc and other libs that my python dependencies need)

I don’t know if it’s the best way to do it, though

3 Likes
10

Using ld_library_path to override the version of libstd to be used can affect other applications as well. Some will not operate correctly because it is now using an incompatible version of libstd. It is a good stopgap though, but when working with a team it will cause problems eventually.

The reason pandas needs libstd is because poetry will download a prebuilt library. That library was built against Debian(?) or some other distro. It is prone to become incompatible when libstd is placed somewhere else or when other versions of libstd are available. libstd not being found is the result of that.

With pip you can skip downloading prebuilt binaries and thus force building all packages with c bindings locally. It is done using --no-binary.

Poetry has something similar, but it is a config value. See https://python-poetry.org/blog/announcing-poetry-1.2.0/#opting-out-of-binary-distributions

When building the package you do need to add all build tools and libraries to your shell. This means adding gcc, make, zlib and maybe others to your shell environment.

Poetry2nix does all of this automatically for many python packages. It is generally a good solution, but you cannot use the poetry cli as freely anymore. Still though, it does seem like the most stable solution.

In your example, none of poetry2nix is being used inside the devShell. You could try to add the mkPoetryApplication package to packages inside mkShell. That will make your application available as an executable. It should allow you to run myapp in your Nix shell. This is how your use your project as a standalone application.

If you want to develop on your application you’d generally want to use the mkPoetryEnv instead of mkPoetryApplication.
The poetry2nix readme unfortunately do not show a full example of using mkPoetryEnv inside mkShell, but this comment does: Add doc on how to create a shell that includes dev-dependencies · Issue #186 · nix-community/poetry2nix · GitHub. You can see that it refers to the mkPoetryEnv from within the mkShell.

1 Like
11

I completely agree with you that mkPoetryEnv is a more elegant solution and ld_library_path may have side effects.
However, I often needed to override packages to add a build system dependency or to add it to build-systems.json when I tried to use mkPoetryEnv. It also takes time to compile as there is no binary cache (often had to recompile mypy, for example), at least to my knowledge.