How do you python?

Help
1

Hello,

I am again facing issues with the nix wonderful world…
Here is the point: I’m working in a computer science school, on different projects with different teams. From times to times, I need to work on python projects or with python tools made by other teams and it is ALWAYS a pain in the … to be able to run theses things.

Today I need to work with a python tool built by poetry and no matter what I do, I can’t have it to work.
I tried to simply nix shell nixpkgs#poetry to then call poetry commands from my shell, but it does not work as poetry tries to modify things in the store.
Then I heard about poetry2nix, and tried to build a quick flake to run my tool: Dead end too, after a few hours of fails, I am unable to build my tool and I’m running out of time to continue debugging that way.

So here is my question: Is it possible to run quick python things without porting the world to nix ? I mean, I love nix, but I can’t afford to take hours and hours of my time to port a simple tool to nix every time I need to work with another team. Also I am on nixos, but none of my colleague are, and porting everything to nix seems like a big time waste.
Am I going to be forced to install a quick debian VM to be able to run python things ?

2 Likes
2

Usually when spinning up python environments that you want to be separate from what’s actually on the system you’d use a virtualenv of some kind (many other implementations exist).

Poetry has native support for this, so I’m a bit confused as to why it’s not working ootb for you. Maybe you need to explicitly use poetry env first, or the project you’re working on has silly configuration?

3

Actually I’m not a big python user, and maybe all of this is not even nix related. My poor python skills are limited to use things made by others and write small scripts.

The documentation of the tool I want to use (sorry I can’t link the tool as it is on a private repository) says I need to run the poetry self add 'poethepoet[poetry_plugin]' command before installing and building the tool, but this command fails with error: externally-managed-environment complaining about the nix store not being writeable.

Then I have to do:

poetry install -n
poetry build -n
pip3 install -q dist/cms-1.0.26.tar.gz

The first two commands are working, even without the poetry self command, but the pip3 install also fails because it is unable to write to the nix store.

4

Yeah, this is all ultimately because python packaging is a little nuts. If you don’t really understand how the tools work, and you’re trying to use nix which tries to impose some sanity on them, you can easily get stuck with the more quirky projects out there.

If you use pipx to install your python packages instead of nix most of these issues will magically go away (i.e., start with pipx install poetry before doing anything else, and use pipx install -q ... at the end).

Here’s me explaining the magic:

The poetry self add command adds plugins to poetry. Basically, you’re installing a build tool, but the build tool needs to then also install another component of itself before it can actually build your project.

This blurs the lines of data vs code, which is just brilliant, because it breaks when you make the build tool read-only (which nix imposes as a limitation to make it possible to have declarative configuration - imperatively changing applications on your system leads to insanity).

We can solve this with nix, by making nix install the plugin instead of letting poetry do it itself:

# poetrypoet.nix
let
  pkgs = import <nixpkgs> {};
in
  pkgs.poetry.withPlugins (_p: [
    pkgs.poethepoet
  ])

nix-shell -p 'import ./poetrypoet.nix'

Yes, this requires knowing all of the stuff I just told you about, and also about an obscure function hidden in the nixpkgs source. But the problem is ultimately that poetry blurs the line of what we call a “package” here.

Yep, pip installs to some global python modules directory, which is also a modification to your python environment. It used to just do that without complaining on other distros, which would have hilarious consequences like breaking ubuntu. Nowadays it doesn’t do that anymore, so at least we have some better tooling around.

The reason pipx solves all these issues is that it automatically creates an isolated virtualenv to install all your packages into. This virtualenv will be entirely declarative; The packages can be modified at runtime. The downside of this is that your environment may randomly break, the upside is that you can just blindly follow the instructions of random projects that expect your environment to be breakable.

This breaks apart when you need one virtualenv to hold a set of packages, of course, at that point you need to manually break out the virtualenv.

But yes, sometimes a VM or distrobox is the only reasonable option. Java projects, for example, are much worse than even the mess of python because of their complexity and the lack of sane build systems altogether.

2 Likes
5

I write a flake.nix in an adjacent directory, so it’s like:

.
├── theproj
│   ├── .envrc
│   ├── poetry.lock
│   └── pyproject.toml
└── theproj-flake
    ├── flake.lock
    └── flake.nix

.envrc contains:

use flake ../theproj-flake

The flake usually looks something like this:

{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:nixos/nixpkgs?ref=23.11";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; })  # maybe 
          mkPoetryApplication;                                   # maybe
        pkgs = import nixpkgs { inherit system; };
      in rec {

        theproj-poetry = mkPoetryApplication{ projectDir = ./.; };  # maybe

        devShells.default = pkgs.mkShell {
          inputsFrom = [ theproj-poetry ];  # maybe
          packages = [
            pkgs.poetry
            pkgs.python311
            pkgs.python311Packages.pylsp-mypy
            pkgs.postgresql_16 # or whatever else
          ];
        };
      });
}

If I’m lucky, then the project has dependencies which just work via poetry2nix, in which case I include the lines marked # maybe. Otherwise, I exclude them and just use poetry2nix as an overkill way to install poetry and use poetry install from the devshell that direnv sets up.

The aspiration here is that maybe one day I’ll port those dependencies, or maybe I can use poetry2nix on just a certain group of them, at any rate it’s nice to have a place to pin non-python dependencies, even if I’m still handling the python dependencies imperatively.

There are occasionally weird interactions between the devshell’s tweaks to PYTHONPATH which cause trouble inside of the venv, but I manage to dodge these most of the time. I think they can be avoiding by altering the package-generated wrapper scripts to be a bit less assertive (so that the venv is authoritative) but I have not yet mastered this art.

1 Like
6

I think I came to quite the same option.
I just got it to work, at least the poetry part, inside a python venv.

In my mind, poetry already creates a virtual environment so this should not have been necessary. But it seems I was wrong.

I first create a venv, then install poetry inside it, and finally I’m able to poetry self add without errors.
This looks like a ‘virtual env’-ception to me but it works. I will take a look at pipx to see what I can do with that.

It’s funny how nix has a tendency to highlight things that are buried under the rug by years of conventions.

Once again, thank’s TLATER for your patience !

1 Like