Jupyter notebook dependency management with Poetry

Guides
1

Poetry is a declarative project manager for Python. However, if you install Jupyter notebook with poetry on NixOS, it might fail to find some .so files from the environment.

Combining the buildFHSUserEnv solution in another topic here with poetry2nix.mkPoetryEnv, we can get a working Jupyter environment with declarative dependency management portable to non-Nix environment.

Just place two files in the top-level directory of the Poetry project:

poetry-env.nix 

{pkgs ? import <nixpkgs> {} }:
let
  lib = pkgs.lib;
  poetry2nix = pkgs.poetry2nix;
  python37 = pkgs.python37;
in
  poetry2nix.mkPoetryEnv {
    python = python37;
    pyproject = ./pyproject.toml;
    poetrylock = ./poetry.lock;
  }

poetry-env-fhs.nix 

{
  pkgs ? import <nixpkgs> {},
  # This allows us to provide a command to run via `--argstr run COMMAND`.
  run ? "bash"
}:
let
  poetry-env = import ./poetry-env.nix { };
in
  with pkgs; (buildFHSUserEnv {
    name = "poetry-env-fhs";
    targetPkgs = pkgs: with pkgs; [
      # curl
      # git
      gcc
      gnumake
      python37Packages.poetry
      pandoc # for pdf conversion
      texlive.combined.scheme-full # for pdf conversion
      which # a convenient tool in vertualized environments
    ] ++ [
      poetry-env
    ];
    runScript = "${run}";
    profile = ''
      # export SSL_CERT_FILE=${cacert}/etc/ssl/certs/ca-bundle.crt
      # export GIT_SSL_CAINFO="$SSL_CERT_FILE"
      # export LANG=C.UTF-8
    '';
  }).env

Since it takes quite a while for mkPoetryEnv to build a package, you may want to
nix-build ./poetry-env.nix to keep the build result from being GC-ed.
If two files would to be too much, they can also be joined together:

{
  pkgs ? import <nixpkgs> {},
  # This allows us to provide a command to run via `--argstr run COMMAND`.
  run ? "bash"
}:
let
  lib = pkgs.lib;
  poetry2nix = pkgs.poetry2nix;
  python37 = pkgs.python37;
  poetry-env = poetry2nix.mkPoetryEnv {
        python = python37; # Python package to use
        pyproject = ./pyproject.toml; # Path to pyproject.toml
        poetrylock = ./poetry.lock; # Path to poetry.lock
  };
in
  with pkgs; (buildFHSUserEnv {
    name = "poetry-env-fhs";
    targetPkgs = pkgs: with pkgs; [
      # curl
      # git
      gcc
      gnumake
      python37Packages.poetry
      pandoc # for pdf conversion
      texlive.combined.scheme-full # for pdf conversion
      which # a convenient tool in vertualized environments
    ] ++ [
      poetry-env
    ];
    runScript = "${run}";
    profile = ''
      # export SSL_CERT_FILE=${cacert}/etc/ssl/certs/ca-bundle.crt
      # export GIT_SSL_CAINFO="$SSL_CERT_FILE"
      # export LANG=C.UTF-8
    '';
  }).env

I’m just a newbie here. Any advice will be appreciated.

2 Likes
2

I’ve never used poetry for an environment, but this is what i do (The environment is used for many things, not just notebooks):

# shell.nix
let
  pkgs = import <nixpkgs> {};
  python = pkgs.python37;
  pythonPackages = python.pkgs;
in

with pkgs;

mkShell {
  name = "pip-env";
  buildInputs = with pythonPackages; [
    azure-cli
    kubectl

    # Python requirements (enough to get a virtualenv going).
    psutil
    #tensorflow
    #pyarrow
    pandas
    ipykernel
    jupyter
    pytest
    setuptools
    wheel
    venvShellHook

    libffi
    openssl
    gcc

    unzip
  ];
  venvDir = "venv37";
  src = null;
  postVenv = ''
    unset SOURCE_DATE_EPOCH
    ./scripts/install_local_packages.sh
  '';
  postShellHook = ''
    # Allow the use of wheels.
    unset SOURCE_DATE_EPOCH

    # get back nice looking PS1
    source ~/.bashrc
    source <(kubectl completion bash)

    PYTHONPATH=$PWD/$venvDir/${python.sitePackages}:$PYTHONPATH
  '';
}

And I can largely use it as a vanilla virtualenv environment. So if needed, i can just do something like pip install requests, and it works for most packages that don’t have native dependencies. (I guess your buildFHSUserEnv is one way around this). Another benefit is that I get to download most of my python dependency through the nixpkgs cache, which is much faster than pip install.

For running the notebook I can just do:

$ nix-shell
$ jupyter notebook

NOTE: Jupyter notebook kernels reference a python interpreter path, make sure that interpreter is able to find the necessary packages it needs. I achieve this through shell-hooks from the packages, so my PYTHONPATH contains them, but mkPoetryEnv will probably install everything into a single site-packages, which should also be fine.

4 Likes
3

The wrappers seems to be more Nixy than buildUserFHSEnv.

I put it under FHS environment simply because Jupyter Notebook complained about missing .so files and failed to start, and that nbconvert and the “Download as” button in the notebook interface requires pandoc and texlive in the PATH to produce PDF files.

It would be great if there are some cleaner solution.