Mach-nix, pip2nix, poetry2nix, or pynixify for Zulip provision

Development
1

Zulip is interested in replacing its provisioning system with Nix (see https://chat.zulip.org/#narrow/stream/3-backend/topic/provision.20via.20nix/near/1354345). The first step toward this goal is by making its Python environment setup using Nix.

This is the entirety of the script used by Zulip to prepare a Python virtualenv (where everything is cached in /srv/zulip-py3-venv): https://github.com/zulip/zulip/blob/d0a697fba73b1e0cdfd372b840b8fe4bda225723/scripts/lib/setup_venv.py. The requirements.txt files in https://github.com/zulip/zulip/tree/main/requirements are generated via pip-tools. The need to do hashing manually, to install system packages manually, will become obsolete once in Nix.

The progress is pending because there are several similar tools that do Python on Nix: mach-nix, pip2nix, poetry2nix, pynixify.

If anyone can give advice on which tool is best suited for this use case, it would be greatly helpful! I can provide additional context as needed.

6 Likes
2

Since you have requirements.txt files and from reading the thread there is a desire as well to migrate in steps, I suggest to use mach-nix for now. Note it’s database with metadata is quite big though.

There is also GitHub - on-nix/python: Extensive collection of Python projects from PyPI, for Nix!. It is my understanding that it has not indexed all Python packages though, so you need to check whether the packages you need are there.

1 Like
3

I’d suggest moving to poetry first, as using it is a lot less messy than good old setup.py and after that conversion has happened successfully, use poetry2nix.

1 Like
4

poetry is a lot more than “setup.py”. The latter is a build backend only, setuptools/distutils. poetry on the other hand has its own build backend, poetry-core, and the environment manager poetry. Note they can all be used independently from one another. And there are more build backends out there.

Now with PEP 621 declarative build backends such as flit and poetry-core will become more or less identical. Switching to a declarative build backend is, in my opinion, also preferable, but it is not needed for packaging with Nix.

5

I see, thank you! I have more questions.

I suppose there is a huge overlap between the package databases of GitHub - on-nix/python: Extensive collection of Python projects from PyPI, for Nix! and GitHub - DavHau/pypi-deps-db: Dependency DB for python packages on pypi. I suppose the content of pypi-deps-db is auto-generated, while on-nix/python is hand-curated? Does using mach-nix require always downloading the pypi-deps-db as a dependency, in all situations?

Does poetry2nix auto-generate Nix expressions that use the wheels from PyPI?

6

pypi-deps-db is auto-generated and is always needed in its entirety. on-python has an auto-generated part (which is significantly smaller) and some parts that are manual. poetry2nix uses poetry which creates a lock file and contains all needed Python information. poetry2nix then uses that along with its own database of overrides/fixes to nixify the environment created by poetry; no need for it to create/generate expressions.

7

It’s perhaps a bit off-topic, but related. As an user of poetry2nix I really enjoy the tool, because it addresses most of my needs. The biggest issue with it (of course it’s not poetry2nix’s fault) that a lot of python packages need overrides, because they need some extra dependencies or hacks around to make it work with nix.

I’m really glad that there’s an interest of python packages to use nix. I think what is greatly needed, and projects like poetry2nix or mach-nix would greatly benefit from, is a mechanism where author could define themselves how to build the package. I understand that Zulip here wants to build the application, and I think poetry2nix or mach-nix (did not use yet, I also see people like it), zulip appears to also provide a python package, which this would be applicable to.

Imagine providing a nix function, where you could pass python interpreter as an argument (with optional parameters to enable/disable functionality) and it would provide a derivation ready to use. No overrides and no pypi-deps-db would be needed anymore for such packages.

Would that be possible to standardize with flakes? I wouldn’t want to introduce a yet another file.

8

Does poetry2nix auto-generate Nix expressions that use the wheels from PyPI?

poetry2nix tries to be pure and builds everything from source, there’s an undocumented option to use wheels but I don’t know if it works anymore.

I use it at work and works well, from time to time requires PR to update overrides. As authors seem to frequently change their non python dependencies or even build systems (from setup to poetry or flit or hatch and others). The PRs are merged quickly and there’s also functionality adding own override for the time being.

9

Thank you for helping me understand the tools better.

I found a post comparing mach-nix and poetry2nix. Though this was in May 2020, and might be outdated.

For reference, the pypi-deps-db is ~74.9 MiB as of today, which is fine once downloaded.

poetry2nix tries to be pure and builds everything from source, there’s an undocumented option to use wheels but I don’t know if it works anymore.

I just discovered that the preferWheels feature was recently documented ~7 days ago: Document preferWheels parameter · nix-community/poetry2nix@c13ba70 · GitHub.

I need to re-mention that Nix-ifying the Python environment in Zulip is one of the steps toward moving the rest of the non-Python components to Nix. This means it will include nodejs, PostgreSQL, Redis, nginx, RabbitMQ, and so on as well (documentation can be found here). Is it possible to embed mach-nix.mkPython/poetry2nix.mkPoetryApplication as a part of a larger build system?

10

Could you elaborate on what you mean here? I think the answer for poetry2nix is probably yes, but I’d need a better understanding of what you mean.

11

I meant, the rest of the non-Python components are also prepared via Nix (which includes enabling daemons, logging, and so on). I was wondering if I have to specify e.g. postgresql, nodejs as dependencies within the closure of poetry2nix.mkPoetryApplication, or the other way around.

12

Right. That would be the other way around.

At the point where poetry2nix.mkPoetryApplication has returned a derivation you can pretty much forget the fact that it’s a Python application. It’s just another application packaged up in a derivation.

1 Like
13

Update: I have tried mainly mach-nix and poetry2nix to get a nix-shell env equipped with the Python dependencies. I started with a nix-shell env instead of a Python application, because this is the easiest way to incorporate Nix into Zulip’s provisioning, for now.

mach-nix

While mach-nix can read from a requirements.txt directly, it fails to read Zulip’s lockfile requirements files, because they contain explicit integrity hashes, as can be seen here

backoff-stubs==1.10.0 \
    --hash=sha256:03e995de0a70016c6fe758498e1ca811f1db517c00cbd06e3039c9e4f6ea2566

I can contribute upstream to mach-nix for the --hash=... feature, but this means that the feature is not available now.
Edit: it could be intentional that mach-nix doesn’t parse --hash=..., see Mach-nix, pip2nix, poetry2nix, or pynixify for Zulip provision - #16 by rht.

poetry2nix

I had to manually convert the requirements files to pyproject.toml and poetry.lock, but other than that, it works out of the box!

pip2nix

Uses wheels by default. But I am not able to even run pip2nix in a nix-shell due to this issue. I’m testing on a NixOS on the unstable channel, while pip2nix requires running pip2nix generate -r requirements.txt on the stable channels.

pynixify

Has the same problem as mach-nix: pynixify can’t parse --hash=.... Even if this is addressed, pynixify still has limitations with version pinning, as stated here. It recommends not using ==

# A great requirements.txt file
requests
tqdm>=4.47.0

pynixify relies on pinning the Nixpkgs version instead. And the Python package versions are constrained to be the versions specified by the pinned Nixpkgs. As such, I think using pynixify is a no-go.

3 Likes
14

I don’t know what you mean by “manually convert” but from the application venv, I generally run something such as:

poetry init -n
pip freeze | xargs poetry add
15

I was using the various methods suggested in Importing/ exporting from poetry in pip format. · Issue #663 · python-poetry/poetry · GitHub (found from Show how to convert from requirements.txt/Pipenv to Poetry · Issue #1701 · python-poetry/poetry · GitHub).

I thought the problem with pip freeze | xargs poetry add is that the pyproject.toml will contain indirect dependencies (where they should be only in the lockfile), and that the versions are pinned with == instead of >=.

2 Likes
16

Actually, with mach-nix, I can see why the --hash=... parsing of requirements.txt is not implemented. I can specify backoff-stubs==1.10.0, and mach-nix will do the integrity check using the hashes in pypi-deps-db, and it will still be reproducible.

17

At first, getting Zulip to use Nix is super interesting.

While I’m still mostly using pip2nix, I can confirm that pip2nix has a fundamental issue of depending on pip private APIs. That’s why it does not work on NixOS > 20.09. Also, it does not include “database” of non-Python dependencies, so in practice it needs complementary expression. I use it with a custom nix-functions to merge it with nixpkgs expressions, but possible issues with that are often too involving to debug. One more: updating generated expressions requires pip2nix to do pip install for all packages in requirements.txt, which is a bit much for updating a single package.

@FRidh Would you think, it’d be manageable for projects with size of Zulip to maintain it own Python package set? Start by copying packages from nixpkgs and then maintaining required versions with help of scripts? Does nixpkgs already use some scripts to help keep maintained Python packages up-to-date?

18

With poetry you indeed describe your abstract dependencies in the pyproject.toml and your concrete/resolved dependencies end up in your lock file. With requirements.txt it is entirely up to yourself how you use it

No, I would not recommend that. I think, especially in this case, that poetry2nix and mach-nix are preferable over manually maintaining expressions, even when using scripts for it. There aren’t any too weird dependencies. With poetry2nix it is especially clear what happens, and requires very few changes from non-Nix developers (at most some overrides).

The only good reason I think there is to use the Python packages set outside of Nixpkgs is if you find it important that you have tested packages and are willing to give up control over versions. For me that is worth it with what I use it for (mostly scientific computing), but for pure Python packages I would not bother.

Yes and no, but that is for in another thread.

1 Like
19

I should note that I have tested only on the mypy.txt of Zulip’s requirements. In total there are mypy.txt (Mypy-specific), dev.txt (development environment), and prod.txt (production environment).

I just tested this requirement in particular, which trips both mach-nix and poetry2nix:

# Forked to avoid pulling in scipy: https://github.com/mailgun/talon/pull/200
https://github.com/zulip/talon/archive/1711705c952806d4a704c7dbf58f21db8e11756a.zip#egg=talon-core==1.4.8.zulip1&subdirectory=talon-core

mach-nix failed with this error

SyntaxError: Unable to interpret marker syntax: //github.com/zulip/talon/archive/1711705c952806d4a704c7dbf58f21db8e11756a.zip: invalid expression: //github.com/zulip/talon/archive/1711705c952806d4a704c7dbf58f21db8e11756a.zip

I couldn’t make it work with poetry: I don’t know how to specify a subdirectory, like in the line above (i.e. subdirectory=talon-core). There are 2 related issues:

20

I have an update for the poetry2nix front. I discovered that as of Poetry 1.1.0a7, it has a feature to allow Git subdirectory (see this PR).

But this wasn’t fully working. I made a patch.

But this patched fork of Poetry causes problem on poetry2nix, with this log:

Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing wheel setup hook
Using wheelUnpackPhase
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
Executing wheelUnpackPhase
Finished executing wheelUnpackPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
applying patch /nix/store/g1vpfnjn785nq00qfpcmxjf8frzgxj65-e7869f05751561958b946b562093397027f6d5fa.patch
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/mypy/main.py
|+++ b/mypy/main.py
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
2 out of 2 hunks ignored
can't find file to patch at input line 27
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/mypy/modulefinder.py
|+++ b/mypy/modulefinder.py
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
5 out of 5 hunks ignored
can't find file to patch at input line 121
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/mypy/pyinfo.py
|+++ b/mypy/pyinfo.py
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
can't find file to patch at input line 179
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/mypy/test/testcmdline.py
|+++ b/mypy/test/testcmdline.py
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
1 out of 1 hunk ignored
can't find file to patch at input line 192
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/test-data/unit/cmdline.test
|+++ b/test-data/unit/cmdline.test
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
1 out of 1 hunk ignored

This is unrelated to the talon dependency, but is instead about Mypy. I think this is caused by me using the latest master version of Poetry, but not sure.