Sqlalchemy python fails to find libstdc++.so.6 in virtualenv

Hi, I’m having trouble understanding how gcc libraries are being linked when Python virtual environments are being used in nixos. I have this in my configuration.nix:

environment.systemPackages = with pkgs; [
     (python311.withPackages(ps: with ps; [ 
       ipython
       setuptools
       sqlalchemy
       greenlet
       aiosqlite
       virtualenv
     ]))
     poetry
     pdm
     gcc

then any virtualenvironments created with poetry or pdm with the same packages throw this error when sqlalchemy is used with asyncio:

<sqlalchemy.orm.session.AsyncSession object at 0x7f33a43d8410>
Traceback (most recent call last):
  File "/home/pythonapp/sqa.py", line 12, in <module>
    asyncio.run(get_db())
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/pythonapp/sqa.py", line 9, in get_db
    async with async_session() as session:
  File "/home/pythonapp/__pypackages__/3.11/lib/sqlalchemy/ext/asyncio/session.py", line 1082, in __aexit__
    await asyncio.shield(task)
  File "/home/pythonapp/__pypackages__/3.11/lib/sqlalchemy/ext/asyncio/session.py", line 1027, in close
    await greenlet_spawn(self.sync_session.close)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pythonapp/__pypackages__/3.11/lib/sqlalchemy/util/concurrency.py", line 64, in greenlet_spawn
    _not_implemented()
  File "/home/pythonapp/__pypackages__/3.11/lib/sqlalchemy/util/concurrency.py", line 44, in _not_implemented
    raise ValueError(
ValueError: the greenlet library is required to use this function. libstdc++.so.6: cannot open shared object file: No such file or directory

I have gcc installed and the c++ file exists in multiple nix stores:

❯ : fd -g libstdc++.so.6 /nix/store/
/nix/store/r7wq6fn906m2fr5jnmhg7774h2cs0440-gcc-12.3.0-lib/lib/libstdc++.so.6
.... more similar locations

To replicate this a simple script can be used:

import asyncio
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker, declarative_base

engine = create_async_engine("sqlite+aiosqlite://./test.db", echo=True)
async_session = sessionmaker(engine, expire_on_commit=False, class_=AsyncSession)
async def get_db():
    async with async_session() as session:
        print(session)

asyncio.run(get_db())

What’s the process to expose my virtualenv to libstdc++.so.6 here?
I’ve tried using poetry2nix through nix flakes but there are other bugs that are getting in the way there (like this one)

Another alternative I’ve seen is to edit LD_LIBRARY_PATH explicitly which doesn’t seem very sustainable.

Figured this out. There’s this detailed write up here on github how to get this done using nix-ld but here’s a quick summary:

  1. Enable nix-ld in your configuration.nix:
  programs.nix-ld.enable = true;
  programs.nix-ld.libraries = with pkgs; [
    zlib # numpy
    libgcc  # sqlalchemy
    # that's where the shared libs go, you can find which one you need using 
    # nix-locate --top-level libstdc++.so.6  (replace this with your lib)
    # ^ this requires `nix-index` pkg
  ];
  1. create a nix shell in your project to expand LD_LIBRARY_PATH with the shared libs. e.g. default.nix
with import <nixpkgs> {};
mkShell {
  NIX_LD_LIBRARY_PATH = lib.makeLibraryPath [
    stdenv.cc.cc
  ];
  NIX_LD = lib.fileContents "${stdenv.cc}/nix-support/dynamic-linker";
  shellHook = ''
    export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
  '';
}
  1. Then nix-shell and voila, the python scripts should work!

Still not sure I understand the whole process but this seems like the nix way of doing this.

1 Like