How to convert a Python `requirements.txt` file into a Nix script?

Help
1

Inherited a Django app with a non-standard requirements.txt that translates to the following:

Django == 2.1.5
psycopg2-binary
django-crispy-forms
django-debug-toolbar
django-tables2
openpyxl
django-mathfilters
django-audit-log
django-reversion
django-simple-history

I think I found the answer in the Python section of the Nixpkgs manual (in chapter 15.19.1.2.1. Python library packages in Nixpkgs to be precise), but before spending an inordinate amount of time, I would like to know whether this approach is indeed the most efficient/pragmatic/etc.

I got the first 3 lines so far but it is representative of the entire file because

  1. it requires pinning of Python packages (which I haven’t been able to figure out yet)
  2. has packages that are already part of the Python packages in Nixpkgs
  3. and packages that need to be added manually (e.g., django-crispy-forms)

I have been very lucky with django-crispy-forms because it is not part of the supported Python packages, but there is a derivation for it elsewhere in Nixpkgs so I could use it as a template.

The contents of the current shell.nix file are below, and my question is: should I continue along this vein?

# TODO pin this as soon as possible
{ pkgs ? import <nixpkgs> {} }:

let
  pythonEnv =
    let
      django-crispy-forms = pkgs.python38.pkgs.buildPythonPackage rec {
        pname = "django-crispy-forms";
        version = "1.10.0";

        src = pkgs.fetchFromGitHub {
          owner = "django-crispy-forms";
          repo = "django-crispy-forms";
          rev = version;
          sha256 = "0y6kskfxgckb9npcgwx4zrs5n9px159zh9zhinhxi3i7wlriqpf5";
        };

        # For reasons unknown, the source dir must contain a dash
        # for the tests to run successfully
        postUnpack = ''
          mv $sourceRoot source-
          export sourceRoot=source-
        '';

        checkInputs =
          [ pkgs.python38Packages.django
            pkgs.python38Packages.pytest-django
            pkgs.python38Packages.pytestCheckHook
          ];

        preCheck = ''
          export DJANGO_SETTINGS_MODULE=crispy_forms.tests.test_settings
        '';

        meta = with pkgs.lib; {
          description = "The best way to have DRY Django forms";
          homepage = "https://github.com/maraujop/django-crispy-forms";
          license = licenses.mit;
          maintainers = with maintainers; [ earvstedt ];
        };
      };
    in
      pkgs.python38.withPackages (
        packages:
          [ packages.django
            packages.psycopg2
            django-crispy-forms
          ]
      );
in
  pkgs.mkShell {
    buildInputs = [
      pythonEnv
    ];
  }

(I’m learning Python, Django, and more advanced Nix techniques at the same time so my replies may be a bit delayed to make sure I understood your suggestions… Thanks!)

I gave pypi2nix a try but it wouldn’t work unfortunately.

2 Likes
2

There is a lovely tool called mach-nix that you can use for this. It takes a requirements.txt as an input and can output:

  • A nix-shell shell.nix
  • A Python venv
  • An OCI container image that can be loaded into Docker and ran there, in the case that colleagues/friends are still working in that container mindset.

Just mach-nix gen -r requirements.txt and see what happens. The README.md does a better job of documenting the aforementioned functionality than I ever could, so I won’t repeat it here. :cowboy_hat_face:

1 Like
3

mach-nix is mighty impressive, I have to say. After traipsing around (and hitting on a bug), the script at the bottom worked like a charm. Not sure how to use the output of mach-nix gen -r requirements.txt but that is on me.

Thanks a lot!

# requirements.txt
# -------------------------
#   Django == 2.1.5
#   psycopg2-binary
#   django-crispy-forms
#   django-debug-toolbar
#   django-tables2
#   openpyxl
#   django-mathfilters
#   django-audit-log
#   django-reversion
#   django-simple-history
# -------------------------
let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/3.0.2";
  }) {};
in
  mach-nix.mkPythonShell {
    requirements = builtins.readFile ./requirements.txt;
  }
4

To make things more reproducible, I guess the following is the right approach, isn’t it?

  1. pip3 freeze > requirements.txt
  2. Let mach-nix do its thing on the above file
5

That seems right to me. How you want to manage that requirements.txt is outside the scope of Nix and Mach-Nix imo. The approach you’ve given is one, but so is pip-compile. There are a few other utilities that I forget the name of.

1 Like
6

Just wanted to say, that mach-nix is by its own admission “unmaintained” and dream2nix is supposed to be used. However, I tried and failed to make it work with just a requirements.txt file and I find the user experience over-all quite confusing, and don’t seem to be alone in that.

@matthewcroughan how does pip-compile solve the problem? It doesn’t generate any nix output, at least to my knowledge.

2 Likes
7

pip-compile produces a requirements.txt from a pyproject.toml which you could technically load into mach-nix via builtins.readFile in Nix evaluation, but I do not think this would work very well. Mach-nix is unmaintained, but is still quite good for some use-cases, though it will hopefully be obsoleted.

If I’m honest, I think my comment back then was incorrect, I think I believed pip-compile could create a sort of lock file for a given requirements.txt, but this is not the case, so it may not be as useful as I imagined in the original comment.

1 Like
8

It does create a sort of lock file, if you give it the right options, e.g. to tell it to include hashes. But the result is still just a requirements file. The harder part is the translation to nix.

1 Like