I have to work on Python projects. However I am on NixOS, so I have two choices:
My own experience is that (1) is hard, and actually not necessarily desirable.
This led me to achieve (2). It’s not the best, but I think it’s a good enough start for the moment.
Gist: How to: make Python dependencies installed via pip work on NixOS
Happy to get feedback and chat about your experiences and solutions!
The “nixify everything” alternative is GitHub - DavHau/mach-nix: Create highly reproducible python environments, which can take a requirements.txt and make a nix shell for you, sourcing packages from pip or anywhere else you like 
I’m personally partial to that, because it leaves the python in tact and nobody except you ever needs to know that it was nixified, but you personally get all the benefits of nixifying.
@Arsleust Thanks, this is really helpful. Does it work with PyCharm which requires a normal venv directory? If yes, it would be amazing, since I cannot use PyCharm with poetry2nix or mach-nix.
Hi! You can try it out, this works on any virtual environment, and Pycharm should pick it up.
You can try to follow this tutorial from @jonringer ttps://www.youtube.com/watch?v=jXd-hkP4xnU
I would also look at Nixpkgs 23.11 manual | Nix & NixOS. This will allow you to use a mix of nix, and leverage venv to pull down the remaining packages. This should work fine for pure python packages, also has the benefit of you being able to do pip install -e for local development
@jonringer is it possible to call autopatchelf inside postVenvCreation or postShellHook? In this test it is not running
with import <nixpkgs> { };
mkShell {
name = "ttest";
buildInputs = [
python3Packages.venvShellHook
];
nativeBuildInputs = [
autoPatchelfHook
];
venvDir = "./.my_env";
postVenvCreation = ''
unset SOURCE_DATE_EPOCH
pip install torch
autopatchelf ./.my_env
'';
}
Hi all, thanks for your feedbacks!
@brogos I haven’t watched https://www.youtube.com/watch?v=jXd-hkP4xnU entirely, but it seems aimed towards packaging Python for nixpkgs.
@jonringer unfortunately using the packages from nixpkgs forces you to use the version available in nixpkgs. Plus there are many packages not available on nixpkgs. In such cases, the manual does not, in my opinion, provide enough help.
This guide is meant to help fix the local Python virtual env when using NixOS. The whole point is to not use Nix to manage any of my Python dependencies.
I’m not saying it’s meant for everyone or that this solution is “the best”. My goal was to document and share what most likely some other people will need.
I really appreciate this pragmatic approach. I tried
for a python development setup for machine learning, and none could be used with an IDE like PyCharm.
I agree that vim and emacs are nice, but some work environments and university courses require IDEs and specific versions of dependencies. At the moment, the nix ecosystem does not provide a working solution for them, and they are left alone.
I love NixOS as a user, but not as a developer.
For me a nix-shell providing poetry works sufficiently well.
I do this on work a lot. I have to always use poetry run for a lot of stuff.
Also you can use the in folder venv, to make it easier discoverable for tools like intellij.
Out of interest, what is the problem with pycharm? Is it that it tries to do virtualenvs itself with the virtualenv module, downloading raw python binaries and whatnot?
My experience has been that people using emacs, vim, intellij and vscode have found things satisfactory with some plugins (and notably direnv), I’m curious if we could make things better for pycharm.
There absolutely are cases where you just need to accept that whatever you are hacking on isn’t prepared to deal with your special snowflake system, and that you’ll be more productive working in a ubuntu VM (e.g., when hacking on upstream bazel or other sandboxed build tools that assume ubuntu 18.04), but I don’t think just normal python dev has to be one of those cases.
I did not really want to go too deep into that discussion, but it might have been a mistake on my side not to provide more context.
If you want to use the standard Python workflow (with pip, poetry and the such), you’ll use a virtual env in your project and install the python dependencies with those tools.
Most of the time, they’ll download the wheel. The issue is that those wheels provide binaries executables and binaries that depend on dynamic linking, and on NixOS it does not work well.
Any lib without binaries is fine, but anything like numpy, pandas, or pytorch will most likely crash when you try to use them.
VSCode, Pycharm and many tools rely on such a venv. You could say “just enter a nix shell” but a lot of tooling wont support that (hello VSCode). And I can’t wait for a patch that might never come: I need to get stuff done today at my job.
So I just patch the binaries rpath and interpreter.
There is also the fact that I think most of the time it is not a good idea to use Nix to manage the Python dependencies of your local environment, but I need more time to open a discussion about it, and would prefer not to pollute the conversation here with a debate. 
Please help me in that regard 
@TLATER Sorry for the late reply. Take, for example, this toy project: GitHub - brian-dawn/nix-flake-poetry-example: Python + Poetry + Nix Flakes
I was not able to find a way to make it work with PyCharm as wants a venv or a path to an interpreter.