Installing python modules from pip: buildFHSUserEnv for python tutorial?

mjlbach · January 31, 2020, 8:49pm

I do research for ML, and rely on non-pure python libraries with C/Cuda dependencies (tensorflow/pytorch/ray). I need the ability to install nightlies (for my work) or legacy versions (for running codebases I’ve found online) simultaneously for different projects. Previously I’ve used virtual environments (which don’t work for this usecase, since you can’t pip install non-pure python modules in Nix) or containers (which still work, but can be annoying to develop in).

I’ve heard talk of using buildFHSUserEnv to mock a filesystem (see 1 and 2), but I’m wondering if there are other best practices/what solutions people have deployed.

timokau · January 31, 2020, 10:43pm

This might help you: https://nixos.wiki/wiki/Python

Note that for something like tensorflow you probably want to add

   export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:${pkgs.stdenv.cc.cc.lib}/lib"

To your shell.nix’s shellHook to make sure tensorflow finds libc.

alexv · January 31, 2020, 11:28pm

Note that support for CUDA in tf/pytorch is automatic, you only need to add

cudaSupport = true;

to your config.nix (or the nix file you use to build your environment). You can also use pinned versions of nixpkgs to install older versions of anything side by side. There are plenty of examples on how to do that in this forum. If the nightly versions do not deviate too much from the versions in nixpkgs (so that the build process changes) you can use overlays to override src attribute.

mjlbach · January 31, 2020, 11:43pm

@timokau I’ve read the wiki page have used nix-shell/direnv to emulate virtual environments, however, this doesn’t particularly cover my issue of using non-pure python packages outside of the scope of what is covered by nixpkgs.

@alexv I should clarify, that I can currently run the tensorflow (with GPU support) in nixpkgs without any problems, however, I am currently using tf-nightly-gpu, and writing an overlay/building the latest tensorflow each time it updates is quite a bit of work.

timokau · February 1, 2020, 7:16pm

Why not? At least it worked for tensorflow, which definitely isn’t pure-python.

mjlbach · February 6, 2020, 4:04pm

Sorry for the delay! On a fresh NixOS install:

python -m venv ~/.virtualenvs/python
source ~/.virtualenvs/python/bin/activate
pip install tf-nightly

I get the following error when trying to install the scipy dependency:

      compiling C sources
      C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

      creating build/temp.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/numpy
      creating build/temp.linux-x86_64-3.7/numpy/core
      creating build/temp.linux-x86_64-3.7/numpy/core/src
      creating build/temp.linux-x86_64-3.7/numpy/core/src/npymath
      creating build/temp.linux-x86_64-3.7/build
      creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy
      creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy/core
      creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy/core/src
      creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy/core/src/npymath
      compile options: '-Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Inumpy/core/include -Ibuild/src.linux-x86_64-3.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/mjlbach/.virtualenvs/python/include -I/nix/store/k5rdcbcwwpvj7l9f1yvd5mfggcfz16kk-python3-3.7.5/include/python3.7m -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.7/numpy/core/src/private -Ibuild/src.linux-x86_64-3.7/numpy/core/src/npymath -c'
      gcc: numpy/core/src/npymath/npy_math.c
      gcc: build/src.linux-x86_64-3.7/numpy/core/src/npymath/ieee754.c
      gcc: build/src.linux-x86_64-3.7/numpy/core/src/npymath/npy_math_complex.c
      gcc: numpy/core/src/npymath/halffloat.c
      ar: adding 4 object files to build/temp.linux-x86_64-3.7/libnpymath.a
      zsh:1: command not found: ar
      zsh:1: command not found: ar
      error: Command "ar rcs build/temp.linux-x86_64-3.7/libnpymath.a build/temp.linux-x86_64-3.7/numpy/core/src/npymath/npy_math.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy/core/src/npymath/ieee754.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/numpy/core/src/npymath/npy_math_complex.o build/temp.linux-x86_64-3.7/numpy/core/src/npymath/halffloat.o" failed with exit status 127
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/mjlbach/.virtualenvs/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h4idr7hq/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h4idr7hq/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-gp_87zcu/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-obeei9p_/overlay --compile --install-headers /home/mjlbach/.virtualenvs/python/include/site/python3.7/numpy Check the logs for full command output.
  WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/mjlbach/.virtualenvs/python/bin/python /home/mjlbach/.virtualenvs/python/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-obeei9p_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- wheel setuptools 'Cython>=0.29.13' 'numpy==1.13.3; python_version=='"'"'3.5'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.5'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'pybind11>=2.4.0' Check the logs for full command output.

jonringer · February 6, 2020, 8:30pm

For packages that need natively built dependencies, i would just include them in your nix-shell:

with import <nixpkgs> {};
mkShell {
  name = "pip-env";
  buildInputs = with python3.pkgs; [
    numpy
    pandas
    scipy
  ];
}

you can also use the shellhook to add LD_LIBRARY_PATH as well:

mkShell {
   ...
  shellHook = ''
    export LD_LIBRARY_PATH=${some_package}/lib:$LD_LIBRARY_PATH
  '';
}

mjlbach · February 7, 2020, 4:11am

@jonringer, Thanks for the example shell.nix! I’m getting

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

Even though I have plenty of space.

This is while trying to install tensorflow from pip, which I need to do to get the latest version (tf 2.1 is currently not packaged for nix).

jonringer · February 8, 2020, 4:30pm

could you post what you were trying to run?

mjlbach · February 8, 2020, 6:45pm

Yes, with my nix-shell:

with import <nixpkgs> {};
mkShell {
  name = "pip-env";
  buildInputs = with python3.pkgs; [
    pip
    numpy
    pandas
  ];
  shellHook = ''
    export LD_LIBRARY_PATH=${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH
    alias pip="PIP_PREFIX='$(pwd)/_build/pip_packages' \pip"
    export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.7/site-packages:$PYTHONPATH"
    unset SOURCE_DATE_EPOCH
  '';
}

I ran:

$ nix-shell
$ pip install tf-nightly

And get

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

mjlbach · February 11, 2020, 5:16am

Interesting, manually setting tmpdir seems to solve it, however installing with the no-cache-dir flag via pip does not. See here.

TMPDIR=/home/mjlbach pip install tf-nightly

The following derivation seems to work! I had to reinstall the cuda drivers so they compiled against the right nvidia driver version.

with import <nixpkgs> {};
mkShell {
  name = "pip-env";
  buildInputs = with python3.pkgs; [
    pip
    numpy
    pandas
    setuptools
  ];
  shellHook = ''
    export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.stdenv.cc.cc.lib}/lib:${pkgs.cudatoolkit_10_1}/lib:${pkgs.cudnn_cudatoolkit_10_1}/lib:${pkgs.cudatoolkit_10_1.lib}/lib:$LD_LIBRARY_PATH
    alias pip="PIP_PREFIX='$(pwd)/_build/pip_packages' TMPDIR='$HOME' \pip"
    export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.7/site-packages:$PYTHONPATH"
    export PATH="$(pwd)/_build/pip_packages/bin:$PATH"
    unset SOURCE_DATE_EPOCH
  '';

mjlbach · February 11, 2020, 5:02pm

@jonringer @timokau How do you feel about me adding this to the nix wiki? I could do a separate tensorflow page or add it under the python page special modules.

jonringer · February 11, 2020, 5:06pm

no space left on device → not sure, to be honest, this sounds like your specific partitions created a scenario in which a given mount point didn’t have any space available.

cuda → There’s already a wiki page Nvidia - NixOS Wiki, but if you feel like this adds additional insight to a user’s workflow then feel free to add it. You could always just add another subsection which are tensorflow specific. I’m sure many are interested.

mjlbach · February 11, 2020, 5:13pm

What’s weird, is I checked my inodes and partitioning, and it doesn’t seem like there’s a space issue. TF is fairly huge (1.5 gb), so wondering if this is exposing something with nix-shell.

jonringer · February 11, 2020, 5:25pm

I’m not familiar with buildFHSUserEnv to be certain, but i know that chroot will mess with mountpoints, it’s probably not capturing one of yours.

mjlbach · February 11, 2020, 5:31pm

Ah, I’m no longer using FHSuserenv, as an alternative I wrote the nix shell above, however , I still have to do the pip TMPDIR hack

timokau · February 11, 2020, 10:16pm

Sounds good, feel free to add it (it’s a community wiki after all)

mjlbach · February 11, 2020, 11:02pm

Do you have any idea why I have to manually override tmpdir?

timokau · February 12, 2020, 9:32pm

I think that is because nix uses some dir under /run/user/1000 by default, which for some reason is a pretty small tmpfs. Not sure why.

mjlbach · February 13, 2020, 7:37am

This seems related to system-d, see this issue