Adding custom ipykernel on jupyterlab

Help
1

Hi,

I’m a newbie in Nix, and I’m sure the following is my fault. Nowadays I have a computer managed with home-manager. For another reason, the computer is not running nixOS.

I’m installing Jupyterlab using Nix&home-manager, so I can run my Jupyterlab server on startup using user’s systemd service. Everything is working fine and I’m super happy about it. The main problem is when I try to work with some Python libraries, which need to install pip dependencies, I’m not able to install another ipykernel in Jupyterlab or create a new pip env.

What I understand, I need to create different flakes for each “notebook”, and manage all from different Flake. Even though I’m happy to do that for a real application, pip install would help me to iterate faster in multiple ways.

So my questions are:

  • Is there any way to create venv in each jupyterlab notebook, and install pip packages directly from it?
  • Could anyone install a new jupyterlab ipykernel from another flake? From the same flake is “easy”
  • Can a flake install pip requirements directly? To avoid the translation on each package?

The relevant code is the following:

  # The JupiterLab packages dependency. 
  python = (const.pythonVersion.withPackages (py:
    with py; [
      numpy
      jupyterlab
      pip
      ipython
      pandas
      ipykernel
      langchain
      pypdf
      # fastembed
      faiss
      emoji
      langdetect
      unstructured
      iso639_p
    ]));

    ...
    packages = const.extendedPkgs ++ [
      python
      pkgs.nginx
    ];

I guess that it’s going to be something related to python.override{}[0] but I’m not sure how to do that from a different flake or if it’s the right thing.

[0] Install Custom Kernels Into Jupyter Lab - #5 by douglas

Thanks

2

I want to make sure we get the abstractions right here, for Jupyterlab:
Kernel → execution environment (combination of runner and it’s environment, like virtualenv)
Notebook → an organized collection of files.

If you’ve ever used a tool like pyenv/direnv, you can use the same virtualenv with multiple source directories. This is similar to how jupyterlab notebooks work. You can, of course, also create a unique kernel to be used with each notebook.

When using nix to manage jupyterlab, you manage the kernels, not the notebooks, so it gives you a fixed set of execution environments. If you want to be able to add kernels outside of nix, then you shouldn’t use nix to manage jupyterlab.

Finally, there is a way to create a python execution environment using nix that is based on a virtualenv (and thus, you can use pip). You can find instructions how to do that using the nix python wiki:
https://nixos.wiki/wiki/Python

So, to answer your questions in order:

  1. Yes. Use nix to create a new kernel for each notebook, and make that kernel based on virtualenv
  2. No. If you’re using nix to control jupyterlab, the list of kernels is fixed in the nix config.
  3. Yes, but there are a bunch of solutions here, and each one I’ve tried has different limitations. I think the best solution at this time is poetry2nix.

** EDIT **
As remarked by @SergeK below, you can install additional kernels into jupyterlab that are unmanaged. See his post for details

3

Jupyterlab will pick up any kernels defined in ~/.local/share/jupyter/kernels/. E.g. you can make an environment created by python3.withPackages available in jupyterlab by including (nixpkgs’) ipykernel in it and running ipykernel --user --install (which you could even do from an mkShell’s shellHook). Jupyterlab would also pick up kernels installed from "venv"s and they will probably work in your FHS environment, even though they are incomplete (and thus “broken”) by construction (which is why they require extra services on NixOS). Including prebuilt PyPi packages in a python3.withPackages environment is fragile (everything about PyPi is) but there are tools which accommodate that, as mentioned by @douglas

1 Like
4

Hi!

Thanks, for sure, I know that it’s a fragile thing to use pip packages, but it’s useful when I try to run packages that I’m not sure If it’s going to work or are useful.

I created a custom bash script which creates a new venv, registers the kernel as a user and fixes the shared libraries. This allows me to have a pragmatic way to get something fast to test. I wrote my setup here:

1 Like