Building Python wheels in a virtualenv


#1

Hello,

I’ve used NixOS for 2 days and I’m learning from scratch. I’m trying to run the setup for a django project. I went on IRC and I got advice to read https://nixos.org/nixpkgs/manual/#how-to-consume-python-modules-using-pip-in-a-virtualenv-like-i-am-used-to-on-other-operating-systems I created this nix script with is almost 1 to 1 copy from the manual.

with import <nixpkgs> {};
with pkgs.python36Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    # these packages are required for virtualenv and pip to work:
    #
    python36Full
    python36Packages.virtualenv
    python36Packages.pip
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    stdenv
    zlib
    zlib.dev ];
  src = null;
  shellHook = ''
  # set SOURCE_DATE_EPOCH so that we can use python wheels
  SOURCE_DATE_EPOCH=$(date +%s)
  rm -fr venv
  virtualenv venv
  source venv/bin/activate
  pip install -r work/python/requirements/local.txt
  '';
}

It fails when building Pillow with missing zlib headers. Any advice on how to fix this?

Thanks


#2

venv python doesn’t know about zlib in your buildInputs.

Use python36Packages.pillow in buildInputs. In general, almost all that interops with native libraries, should be converted to Nix. So expect to maintain 2 sets of deps: one in requirements.txt, another (partial duplicate) in default.nix. To be fair, it shouldn’t be a big trouble, except when version in nixpkgs has completely new API than what you want. cryptography, psycopg2 and numpy are another examples of packages, that have to be explicitly set in Nix.

You don’t have to remove pillow from requirements after that change, it will be ignored when building in Nix virtualenv.


#3

And also, sorry to pushing my vision of things.

You should drop virtualenv altogether, because nix-shell replaces it. See https://nixos.wiki/wiki/Python#Emulating_virtualenv_with_nix-shell. I’m using this method here at work and I’m very satisfied with it.


#4

@danbst Thanks for the reply, I already solved it by doing just that. I thought that putting the python module in the dependencies would pull in the dependencies the wheel would require and it would build itself. I’m not sure if that’s the mechanism that’s behind it, but it works now. This is how my script looks now.

with import <nixpkgs> {};
with pkgs.python36Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    # these packages are required for virtualenv and pip to work:
    #
    python36Full
    python36Packages.virtualenv
    python36Packages.pip
    python36Packages.pillow
    python36Packages.psycopg2
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    stdenv
    libpqxx
    zlib
    zlib.dev
    libffi
    libffi.dev
  ];
  src = null;
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    virtualenv $PWD/venv
    export PATH=$PWD/venv/bin:$PATH
    export PYTHONPATH=$PWD/venv/lib/python3.6/site-packages/:$PYTHONPATH
    pip install -r python/requirements/local.txt
    pip install -r python/requirements/production.txt
  '';
}

I’m interested in using nix only for management, however I think that some of the python libs that I need would be missing from nixpkgs. If that’s the case, do I revert to nix + virtualenv?


#5

I’m interested in using nix only for management, however I think that some of the python libs that I need would be missing from nixpkgs. If that’s the case, do I revert to nix + virtualenv?

Nix+virtualenv (or only Nix, as described in wiki) is a fine solution, if you don’t want to build your package with Nix. You have to manage with Nix only libraries, which require native libraries. If library requires native libraries, and is missing in Nix, you have to package it (true only for NixOS, on Ubuntu you don’t have to do that).

For example, nanomsg-py library requires nanomsg libs, yet isn’t added to nixpkgs. Packaging it is 10 lines: https://gist.github.com/danbst/dd641c696f77f5465ef9c827fcbf1e2c#file-default-nix-L6-L16


#6

I’d like to use build my application with nix to make sure it’s portable and reproducable when deploying. Just as an experiment. Here’s the script I’m using right now to create a working nix-shell but I’m missing some of the dependencies.

{ pkgs ? import <nixpkgs> {} }:
  pkgs.mkShell {
    buildInputs = [
      pkgs.python36Packages.pip
      pkgs.python36Packages.pillow
      pkgs.python36Packages.psycopg2
      pkgs.python36Packages.pytz
      pkgs.python36Packages.python-slugify
      pkgs.python36Packages.pillow
      pkgs.python36Packages.argon2_cffi
      pkgs.python36Packages.redis
      pkgs.python36Packages.django_2_1
      pkgs.python36Packages.django_environ
      pkgs.python36Packages.django-allauth
      pkgs.python36Packages.django_redis
      #pkgs.python36Packages.django_rest-auth
      pkgs.python36Packages.djangorestframework
      #pkgs.python36Packages.coreapi
      #pkgs.python36Packages.django-cors-headers
      pkgs.python36Packages.boto3
      #pkgs.python36Packages.braintree
      pkgs.python36Packages.numpy
    ];
  }

The libraries that I’ve commented out are missing, do I have to package them now so that they’re available?

Thanks for the great responses.


#7

yes, you have to package those and their dependencies with Nix.