how do i enable a python pip environment on nixos?
https://nixos.wiki/wiki/Packaging/Python
the shell.nix from this site is not helping, there is clearly no pip after running nix-shell
how do i enable a python pip environment on nixos?
https://nixos.wiki/wiki/Packaging/Python
the shell.nix from this site is not helping, there is clearly no pip after running nix-shell
looks like there are something more that is even more trickier to do python development on nixos:
[nltk_data] Error downloading 'punkt_tab' from
[nltk_data] <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data] pages/packages/tokenizers/punkt_tab.zip>: [Errno 30]
[nltk_data] Read-only file system: '/nix/store/y7kc3xb8ashq7ics6i8
[nltk_data] qp7yi9m0vwkz7-python3-3.12.8-env/lib/python3.12/site-p
[nltk_data] ackages/llama_index/core/_static/nltk_cache/tokenizers
[nltk_data] /punkt_tab.zip'
read only file system? i am not completely sure what i am dealing with here
Not sure how official this source is but I’ve seen it referenced a few times (and have set up the normal python flake above this link for myself as my main template) and this part is designed specifically with a .venv file in mind
this might come very handy thank you, i actually found there is ‘pip’ that you could install with python3.12 in a flake, such that, after nix develop, you can do pip install, but clearly that is not the preferred way. and indeed i have other issues like,
**NixOS dynamic linker issue**: The error message mentions that NixOS cannot run dynamically linked executables intended for generic Linux environments out of the box.
i am thinking about moving to my ubuntu machine if i cant figure out. in case you wonder, this is the package that is giving me trouble.
%pip install llama-index-embeddings-huggingface
%pip install llama-index-llms-llama-cpp
the second one, first one has been packaged into nixpkgs already.
this is my latest attempt, sorting out every other trouble but nix itself.
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/nix/store/a2vi447kcqbfbi5q7sdcvqr9vj7l0yvw-python3-3.12.8-env/lib/python3.12/site-packages/diskcache'
because i used pip install, nixos did not like it, unless someone convince me otherwise, i think best thing is for the package i am trying to use to be officially included in the nixpkgs.
Looks like XY Problem to me. Is there any reason you must install packages with means other than Nix?
If you wish, try to package it yourself. Otherwise, you may want to use a virtual environment (where you can use pip).
See also this wiki page:
https://wiki.nixos.org/wiki/Python
You can install Conda like this (add it to your packages and rebuild). With Conda, you no longer need to deal with Nix.
(python3.withPackages (
python-pkgs: with python-pkgs; [
conda
]
))
I do, because nixpkgs does not have it, and this one is in particularly tricky, example code:
from llama_index.llms import OpenAI
you cant just install llama-index, it will trigger another error, because it conflicts with the llama-index-cli.
package it myself on this one will not be trivial. and i tried virtual environment with uv,
uv venv
× Querying Python at `/home/alice7/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/python3.13` failed with exit status exit status: 127
│ --- stdout:
│ --- stderr:
│ Could not start dynamically linked executable: /home/alice7/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/python3.13
│ NixOS cannot run dynamically linked executables intended for generic
│ linux environments out of the box. For more information, see:
│ https://nix.dev/permalink/stub-ld
│ ---
this one will be real tricky.
you mean, within conda environment, i will be able to install any python packages using pip or uv without interacting with nix file system?
yes, this is how I use python in nix
and all the packages installed that way will be with conda environment, what happens after nix-collect-garbage or reboot? will those packages persist within conda, whenever i enter conda-shell
?
this is just a conda, like other linux, so the environments with be in $HOME/.conda/envs and it will not be collected by GC.
conda-shell
there is no conda-shell when you install conda like this,
if you need to use conda run conda activate <env>
(I assume you already know how to use conda like on any other Linux system.) .
yes there is, in fact, i must use conda-shell
, there is no conda
command.
i am actually very new, so this is what i found:
but again, when i enter the “shell”, i didnt do conda activate <env>
, i just did pip install something
right away, because it is already there, and you bet, it went through without giving me any nix errors, but, did I mess up? by not specifying the conda venv
?
Because you installed conda using packages.conda
and not the method I provided, there are some differences. The packages.conda
approach builds conda from the binary using fhsenv, while python3.withPackages
builds it from source. I prefer building from source, so that’s the method I use. However, there shouldn’t be much difference in functionality.
I don’t use this method to install conda, so I’m not entirely sure what conda-shell
do. However, I think it might enable a default environment. This means that if you don’t explicitly run conda activate , pip install will install packages into the default environment. This setup is fine if you only need one environment.
now i understand the situation, thank you.
i came across this link: https://gist.github.com/ChadSki/926e5633961c9b48131eabd32e57adb2
which is better because it integrates both cuda and conda together, such that, any python libraries that needs cuda/gpu support can be easily installed, but i am having this error:
exec: Failed to execute process '/home/alice7/.conda/bin/conda': The file specified the interpreter '/home/alice7/.conda/bin/python', which is not an executable command.
triggered when i try to run conda init
, how did you run this “local conda” exactly?
triggered when i try to run
conda init
, how did you run this “local conda” exactly?
Install conda like this (add this to packages
), not using the other methods you provided.
(python3.withPackages (
python-pkgs: with python-pkgs; [
conda
]
))
which is better because it integrates both cuda and conda together, such that, any python libraries that needs cuda/gpu support can be easily installed,
run export LD_LIBRARY_PATH="/run/opengl-driver/lib"
in your shell and python will work with cuda
to test whether cuda is avaliable
# install pytorch first
# test.py
import torch
is_cuda = torch.cuda.is_available()
print(is_cuda)
Note that this is not the “nix way” that many people recommend.
as it turned out, i used this shell.nix to test it, and it returned “false”, and I had to do this:
# fixes libstdc++ issues and libgl.so issues export LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/
otherwise i will run into issues with using libstdc and python libraries. and I cant export this at the same time:
export LD_LIBRARY_PATH="/run/opengl-driver/lib"
export LD_LIBRARY_PATH=“${stdenv.cc.cc.lib}/lib:/run/opengl-driver/lib”
not sure why you need to export stdenv.cc.cc.lib
, might be an issue with conda-shell
I will try that!
thank you!
the reason why i needed to do that was because if i dont, the .venv
will throw me some c++ file that are missing error, and subsequently none of the python libraries can be imported.
i ditched conda
this time, and went with nix-shell completely, since i have successfully established .venv
, that was the whole purpose, i think.
i want python interpreter will never go into my /nix/store
, while providing cuda
support in .venv
without any errors. that is trickiest part.
but let me get this straight, you have already done, python development with cuda enabled in a virtual environment on nixos correct?
by the way, this is the shell.nix
that is currently i am using:
with import <nixpkgs> { };
let
pythonPackages = python3Packages;
in pkgs.mkShell rec {
name = "impurePythonEnv";
venvDir = "./.venv";
buildInputs = [
# A Python interpreter including the 'venv' module is required to bootstrap
# the environment.
pythonPackages.python
# This executes some shell code to initialize a venv in $venvDir before
# dropping into the shell
pythonPackages.venvShellHook
# Those are dependencies that we would like to use from nixpkgs, which will
# add them to PYTHONPATH and thus make them accessible from within the venv.
pythonPackages.numpy
pythonPackages.requests
pythonPackages.pygobject3
# For PDF production in mkdocs
# pythonPackages.venvShellHook
# python311Packages.weasyprint
python312Packages.stdenv
# this is from unstable channel
# python312Packages.llama-cpp-python
# In this particular example, in order to compile any binary extensions they may
# require, the Python modules listed in the hypothetical requirements.txt need
# the following packages to be installed locally:
gobject-introspection
gtk3
taglib
openssl
git
libxml2
libxslt
libzip
gnused
httplz # for serving up the static site while testing
# ???
gitRepo
gnupg
autoconf
curl
procps
gnumake
util-linux
m4
gperf
unzip
cudatoolkit
linuxPackages.nvidia_x11
libGLU libGL
xorg.libXi xorg.libXmu freeglut
xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib
ncurses5
stdenv.cc
binutils
];
# Run this command, only after creating the virtual environment
postVenvCreation = ''
unset SOURCE_DATE_EPOCH
pip install -r requirements.txt
'';
# Now we can execute any commands within the virtual environment.
# This is optional and can be left out to run pip manually.
postShellHook = ''
# allow pip to install wheels
unset SOURCE_DATE_EPOCH
export CUDA_PATH=${pkgs.cudatoolkit}
# fixes libstdc++ issues and libgl.so issues
export LD_LIBRARY_PATH=${stdenv.cc.cc.lib}/lib/
# enable CUDA? currently 'is_cuda' returns false
# export LD_LIBRARY_PATH="/run/opengl-driver/lib"
# export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib
export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
export EXTRA_CCFLAGS="-I/usr/include"
'';
}