I’ve used NixOS for 2 days and I’m learning from scratch. I’m trying to run the setup for a django project. I went on IRC and I got advice to read Nixpkgs 23.11 manual | Nix & NixOS I created this nix script with is almost 1 to 1 copy from the manual.
with import <nixpkgs> {};
with pkgs.python36Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
# these packages are required for virtualenv and pip to work:
#
python36Full
python36Packages.virtualenv
python36Packages.pip
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
stdenv
zlib
zlib.dev ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
rm -fr venv
virtualenv venv
source venv/bin/activate
pip install -r work/python/requirements/local.txt
'';
}
It fails when building Pillow with missing zlib headers. Any advice on how to fix this?
venv python doesn’t know about zlib in your buildInputs.
Use python36Packages.pillow in buildInputs. In general, almost all that interops with native libraries, should be converted to Nix. So expect to maintain 2 sets of deps: one in requirements.txt, another (partial duplicate) in default.nix. To be fair, it shouldn’t be a big trouble, except when version in nixpkgs has completely new API than what you want. cryptography, psycopg2 and numpy are another examples of packages, that have to be explicitly set in Nix.
You don’t have to remove pillow from requirements after that change, it will be ignored when building in Nix virtualenv.
You should drop virtualenv altogether, because nix-shell replaces it. See Python - NixOS Wiki. I’m using this method here at work and I’m very satisfied with it.
@danbst Thanks for the reply, I already solved it by doing just that. I thought that putting the python module in the dependencies would pull in the dependencies the wheel would require and it would build itself. I’m not sure if that’s the mechanism that’s behind it, but it works now. This is how my script looks now.
with import <nixpkgs> {};
with pkgs.python36Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
# these packages are required for virtualenv and pip to work:
#
python36Full
python36Packages.virtualenv
python36Packages.pip
python36Packages.pillow
python36Packages.psycopg2
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
stdenv
libpqxx
zlib
zlib.dev
libffi
libffi.dev
];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv $PWD/venv
export PATH=$PWD/venv/bin:$PATH
export PYTHONPATH=$PWD/venv/lib/python3.6/site-packages/:$PYTHONPATH
pip install -r python/requirements/local.txt
pip install -r python/requirements/production.txt
'';
}
I’m interested in using nix only for management, however I think that some of the python libs that I need would be missing from nixpkgs. If that’s the case, do I revert to nix + virtualenv?
I’m interested in using nix only for management, however I think that some of the python libs that I need would be missing from nixpkgs. If that’s the case, do I revert to nix + virtualenv?
Nix+virtualenv (or only Nix, as described in wiki) is a fine solution, if you don’t want to build your package with Nix. You have to manage with Nix only libraries, which require native libraries. If library requires native libraries, and is missing in Nix, you have to package it (true only for NixOS, on Ubuntu you don’t have to do that).
For example, nanomsg-py library requires nanomsg libs, yet isn’t added to nixpkgs. Packaging it is 10 lines: Nix + Python + nanomsg · GitHub
I’d like to use build my application with nix to make sure it’s portable and reproducable when deploying. Just as an experiment. Here’s the script I’m using right now to create a working nix-shell but I’m missing some of the dependencies.