Pip/uv/poetry in NixOS

Help
1

I use heavily pip/uv/poetry with python packages like pytorch (machine learning packages which use CUDA libraries). All these tools use venv below.

My current solution to make such packages work is to export LD_LIBRARY_PATH in my configuation.nix:

environment.variables = {
LD_LIBRARY_PATH =
  "/run/opengl-driver/lib:" +
  pkgs.lib.makeLibraryPath [
    pkgs.stdenv.cc.cc.lib
    pkgs.zlib
    pkgs.cudaPackages.cudatoolkit
  ];
};

I have tried using nix-ld, but it seems to be ignored by these binaries installed inside a venv.

Interested to know - anyone in the same situations who does it differently?

2

Have you considered adding the libraries just to your shells packages list, rather than tampering with LD_LIBRARY_PATH which you should barely ever touch yourself, especially at this scope.

3

The thing is that I am an admin of a small HPC cluster, so I would prefer a setup that avoids the user having to create any .nix file or anything. I would like it to be completely invisible to the user…

But yeah, for my personal computer, I have different shell .nix files when I work with opencv or whatever and I use that… I was trying to have a solution where the users would use the power of nix if they wanted, but if they didn’t want, they didn’t need to bother with it either…

4

If you want to allow users to run software as if it was Ubuntu, install Ubuntu.

If you want to use NixOS, educate your users.

5

I appreciate the concern, but it’s an HPC that is used by both power users and novice users. What we could do it make a default shell for novice users and let power users build their own shell scripts… That’s not a bad thought!

In any case, if you use pip/uv/poetry (some packages are unfortunately only available in pypi so we need to use one of them), it would be really helpful, if you would like to share what you do. :slightly_smiling_face:

6

I’m assuming you’re aware of uv2nix? It doesn’t solve your issue of making libraries transparently available to non-Nix users but might help create per-project environments.

2 Likes
7

I did not know that. I will explore it - thanks!