ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Hi,

I am trying to writing some data cleaning scripts using python + pandas on my machine that I’ve installed nixOS on. I’m new nix, and when importing pandas I get:

ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

After running my script in the console.

I presume I need to add a package to my configuration.nix file, but I haven’t found one that works. If anyone has encountered this problem and knows a fix that would be very helpful!

Thanks :slight_smile:

1 Like

can you show code example of what your setup looks like (just nix files). For example how are you making the python available in your shell? (my guess is you’re using virtualenv)

Hi thank you for replying.

I have python3 as a package in my dev packages in my configuration.nix file:

environment.systemPackages = with pkgs; [
    # dev packages
    emacs
    vscode
    python3
    gcc-unwrapped
    clojure
    leiningen

    #gaming
    lutris-free
    steam

   # Audio
   pavucontrol
  ];

I am just using python as I always have done on linux in the bash shell with the python -m ... command. I do use venv to make virtual environments to segment my code.

I have read a bit about nix-shell but not sure where my shell.nix file is, if it is anywhere at all. I just use bash in the KDE Konsole to navigate and run code.

Not sure if that’s helpful - let me know!

ok so the issue is when you create a virtual-env, a copy of python binary is copied and this does not know where to get its glibc and certain other libraries from. I faced this issue previously as well. You have 2 choices

  1. Provide a python with the required packages using nix
  2. Provide the libs needed for python

Provide python with required packages using nix

I’ll give you a direct example and you can figure out how to put it in a nix.shell file. If you run bellow command, you get a shell with a python that has pandas:

$ nix shell --impure --expr '(import (builtins.getFlake "nixpkgs/23.11") {}).python310.withPackages (pyPkgs: [pyPkgs.pandas])'
$ python
Python 3.10.13 (main, Aug 24 2023, 12:59:26) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> exit()

the key part is this python310.withPackages (pyPkgs: [pyPkgs.pandas]). This will return a python which has the pandas package available. You can search for packages here. Its easy to package new ones if its not already available. You can put the produced derivation in buildInputs of a shell.nix or simply run a nix-shell as above to get it

Provide the libs to python binary via virtual env

This is a lot messier. You need to provide certain libs and maybe even put them in your LD_LIBRARY_PATH. Bellow is an example derivation I make in my python projects which use virtualenv

  localPython = writeScriptBin "local-python"
  ''
  .venv/bin/python "$@"
  '';

  # Wrap only python with the required lib files
  python =
    runCommand "python" { 
      nativeBuildInputs = [ makeWrapper  ];
      buildInputs = [ libGL libGLU localPython];
    } ''
    makeWrapper ${localPython}/bin/local-python $out/bin/py \
     --prefix LD_LIBRARY_PATH : /usr/lib/wsl/lib \
     --prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath [pkgs.stdenv.cc.cc pkgs.cudaPackages.cudatoolkit pkgs.cudaPackages.cudnn libGL libGLU]}
    '';

In my examples I’ve also added cuda packages and wsl to library path but for your case its not needed. You just need pkgs.stdenv.cc.cc in LD_LIBRARY_PATH to make the error go away. Note that if you export that on your shell for all programs, it could cause issues so would recommend only wrapping the python runner with it.

I would recommend you always create shell.nix files per project so you can lock its dependencies for future use. Can use flakes, or simply use niv to lock nixpkgs sources as well with an easy interface to update later. Anytime in doubt, search github for nix files and a few keywords and more often than not you’ll find other examples. This has been the main method I’ve learned nix over the years

5 Likes

This worked great in the console, but had more difficulty with the rest of my dependencies using venv + pip + requirements.txt. How can I define these in a shell.nix file instead using your import statement?

Here’s an example shell.nix. Please check the docs for more attributes, additionally I would recommend using niv or flakes to make updating upstream refs easier:

{ pkgs ? import (builtins.fetchTarball
    {
      url = "https://github.com/NixOS/nixpkgs/archive/nixos-23.11.tar.gz";
      sha256 = "065jy7qivlbdqmbvd7r9h97b23f21axmc4r7sqmq2h0j82rmymxv";
    }) { }
}:
let
  python = pkgs.python310.withPackages (pyPkgs: [
    pyPkgs.pandas
  ]);
in
pkgs.mkShell {
  buildInputs = [
    python
  ];
}

Above will make python available after running nix-shell which has pandas already installed. You can use the same format to install additional ones. A good way of finding examples is using Github’s search function. Filter for nix files and a command you think the file should have.

I had a same problem and resolved!!
but I’m trying to use in Pycharm as Python Interpreter and didn’t work!!
Some idea please!!

You may try out buildFHSUserEnv wrapper for python environment that depend on libraries. In my cases, I only managed to get them running with FHS. In a FHS you can make a virtual environment and run pip -r requirements.txt as usual. See an example here, make sure to set the export LIBRARY_PATH.

(it’s a realtively old thread but it came up to the surface and may come up in search results, so a minor correction is in order)

Python interpreter in Nixpkgs links libc “directly” via RUNPATH, unaffected by copying. So neither libc nor the interpreter are the issue. The issue is in pandas, presumably installed from PyPi. PyPi ships incomplete packages, in particular it assumes things like libstdc++ preinstalled and “globally available” (the contract for this assumption is called “manylinux”). This is why python3.withPackages (ps: [ ps.pandas ]), mentioned by @arijoon, is preferable

1 Like

Thanks @SergeK, my bad I was misinformed then when it comes to the core of the libstdc issue. Given this is the case, is there a way to patch the site-packages folder within a venv even when its not part of a nix derivation? I understand this is risky and the dependent derivations may get deleted, but a script to run auto patch over a mutable folder could be a much simpler addition than having to wrap the python executable with LD_LIBRARY_PATH everytime (from time to time I have had to give up on trying to package my python env with nix and simply go for a venv, it can become very time consuming and hard to upgrade especially in ML cases)

There was a script (or a few) going around for patchelfing venvs automatically, not sure how to find it now. Personally I’d rather 1) avoid venvs, 2) use programs.nix-ld.enable and devshells with nix-direnv than patchelf the venv if I have to use a venv, 3) use something like dream/poetry2nix to dump patchelfed pypi stuff into the store, 4) overrideAttrs nixpkgs’ python wrapper’s makeWrapperFlags as a last resort. I’m fairly certain that (2) and (3-4) are less unreliable than mutating a directory that is routinely mutated by another tool

I’m getting this issue when I try to run an Interactive Terminal (with the Jupyter extension on VS Code). I’m connected via SSH to an EC2 instance running Ubuntu 22.04 where I have a nix development environment and packages installed via Poetry. I’ve followed a similar solution noted earlier (mine was set up by a colleague) to run Python scripts with poetry run python <script>.py but I can’t run <script>.py in an Interactive Terminal in VS Code.

 {
  
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
 };
  
  outputs = { self, nixpkgs, flake-utils
            }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system;
                              };

        python-env = pkgs.python311.withPackages (ps: with ps; [pandas geopandas ipython jinja2 seaborn scipy]);


      our-poetry = pkgs.writeShellScriptBin "poetry" ''
        # Wrap in libraries expected by numpy etc
        export LD_LIBRARY_PATH=${pkgs.stdenv.cc.cc.lib}/lib/
        export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath [ pkgs.zlib ]}:$LD_LIBRARY_PATH"
        exec ${pkgs.poetry}/bin/poetry $@
        '';

        
      in rec {
        packages = { };
        
        devShell =  pkgs.mkShell {
          buildInputs = with pkgs; [
            python-env
            awscli
            our-poetry
          ] ++ lib.optionals (stdenv.isDarwin) [
            darwin.apple_sdk.frameworks.CoreServices # needed for all?
            zlib
            hdf5
          ]
          ; 
          shellHook = ''
          '';
        };        
      }
    );
}