How to debug Python dependencies when using poetry2nix

Help
1

Let’s say I am on Ubuntu and I have Poetry installed. Then, let’s say I need to debug a dependency of my Poetry-based Python project. I can mark the dependency as develop=true and it will be checked out into .venv/src/ where I can edit its source directly to add a pdb.set_trace, fix a bug, etc.

➜  poetry install
Creating virtualenv my-app in /Users/zupo/work/poetry2nix_dependencies/.venv
Installing dependencies from lock file

Package operations: 5 installs, 0 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0)
  • Installing idna (2.10)
  • Installing urllib3 (1.26.4)
  • Installing requests (2.25.1 2c2138e)

Installing the current project: my-app (0.1.0)

➜  vim .venv/src/requests/requests/auth.py

However, when using poetry2nix, all dependencies are installed in Nix store, which is read-only, so I cannot edit the dependency’s source. What is the recommended way to “develop” a dependency in a poetry2nix project?

➜  which python
/nix/store/5wx6i7bpnm3a7hcz8xn2rzz6x5gpa1ki-python3-3.8.6-env/bin/python
➜  vim /nix/store/5wx6i7bpnm3a7hcz8xn2rzz6x5gpa1ki-python3-3.8.6-env/lib/python3.8/site-packages/requests/auth.py
# --> read-only file, can't save my change

Here is a repo with a minimal Poetry project that showcases the above: GitHub - zupo/poetry2nix_dependencies: Figuring out how to debug dependencies

2

Because I am on MacOS, I have a workaround that I am currently using: open the file up with Sublime Text 3, which asks for sudo password, then actually allows me to edit it.

➜  poetry2nix_dependencies git:(master) ✗ subl /nix/store/xva0lzhdyi06yvh7inypfpl33d0rb7l6-python3-3.8.8-env/lib/python3.8/site-packages/requests/api.py
➜  poetry2nix_dependencies git:(master) ✗ python src/my-app/foo.py
> /nix/store/xva0lzhdyi06yvh7inypfpl33d0rb7l6-python3-3.8.8-env/lib/python3.8/site-packages/requests/api.py(75)get()
-> return request("get", url, params=params, **kwargs)
(Pdb)

I’d really like to have a cleaner process though …

3

Yeah, editing files in the nix store is definitely not ideal, and not intentional, as the Nix store being read-only is both a feature and a requirement.

If you need to edit or debug a dependency, that should be done in its own project so as to allow for encapsulating your testing, and ensuring that you don’t touch a dependent app while keeping it modular.

4

95% of the cases I don’t need more than insert a pdb.set_trace into a dependency, to understand how it works so I can use it correcly, or in some rare case, monkey-patch it in my code.

Creating a new bespoke project for every dependency that I need to insert this one line into every time I need to do it seems very tedious and time consuming. Surely there is a better way?

5

This issue seems to be talking about the same thing: Enable "develop = true" option for local dependencies to be installed as editable packages · Issue #265 · nix-community/poetry2nix · GitHub

6

I found another approach, that should work on Linux too:

  1. Clone dependency into ./src directory: $ cd src/ && git clone git@github.com:niteoweb/pyramid_mixpanel.git
  2. Add dependency to editablePackageSources so it looks like so:
  devEnv = pkgs.poetry2nix.mkPoetryEnv (commonPoetryArgs // {
    editablePackageSources = {
      mypackage = ./src;
      pyramid_mixpanel = ./src/pyramid_mixpanel;
    };
  });
  1. Comment out the dependency from pyproject.toml:
# pyramid_mixpanel

Now nix-shell should pick up the locally checked out repo instead of taking the source from /nix/store.

1 Like
7

Note for the above: obviously, if pyramid_mixpanel depends on mixpanel lib, and you comment out pyramid_mixpanel from pyproject.toml, then you have to (temporarily) add mixpanel = "*".

8

As of Add path dependencies where develop=true as editable packages by default by adisbladis · Pull Request #487 · nix-community/poetry2nix · GitHub, debugging a Python dependency using poetry2nix is even more straightforward:

  1. Clone dependency into ./src directory: $ cd src/ && git clone git@github.com:teamniteo/pyramid_mixpanel.git
  2. In pyproject.toml, replace pyramid-mixpanel = "*" with { path = "./src/pyramid_mixpanel/", develop = true }

Now nix-shell should pick up the locally checked out repo instead of taking the source from /nix/store. No need to edit default.nix or manually add indirect dependencies to pyproject.toml any more!