Jupyter notebook with batteries

How do I get a Python notebook with numpy, scipy, matplotlib and other essential libraries?

1 Like

You can run:
nix-shell -p pythonPackages.numpy pythonPackages.notebook then jupyter notebook from within the shell and add all other libraries you need if available. (use nix search to check availability)

It’s in the Nixpkgs manual: python3.withPackages(ps: with ps; [ numpy scipy matplotlib ])

Avoid using nix-shell -p pythonPackages.numpy pythonPackages.notebook. It “works” because it uses a setup hook for setting up PYTHONPATH, but it is not guaranteed to work. Also, not all packages will propagate the interpreter.

2 Likes

Is there a less ad-hoc way than nix-shell? Like installing a derivation with the required environment.

Have you looked yet in the manual?

Yes, I did. Should I read it once again?

Maybe you want to create an overlay with the following contents:

self: super: {
   jupyterWithBatteries = super.jupyter.override {
      python3 = super.python3.withPackages(ps: with ps; [ numpy scipy matplotlib ]);
   };
} 

then run

nix-env -iA nixpkgs.jupyterWithBatteries

to install the modified jupyter

2 Likes

I added the following into $HOME/.config/nixpkgs/config.nix:

  packageOverrides = pkgs: with pkgs; {
    jupyterWithBatteries = pkgs.jupyter.override {
      python3 = pkgs.python3.withPackages(ps: with ps; [
        numpy scipy matplotlib
      ]);
    };
  };

nix-env -iA nixos.jupyterWithBatteries installs jupyter but numpy and other libraries are not available in notebooks.

Oh sorry, my fault :frowning: reviewed the source, you should override the definitions attribute, which looks like this: nixpkgs/kernel.nix at beaf69cee298e092698dd2da2e4758b7811859ad · NixOS/nixpkgs · GitHub

I’m trying to install jupyter globally with some packages available but I don’t get the override right. It looks like this in my /etc/nixos/configuration.nix

environment.systemPackages = with pkgs;
    let jupyterWithStuff = pkgs.jupyter.override {
        definitions = pkgs.python3.withPackages( ps: with ps; [
          numpy matplotlib pandas
        ]);
      };
    in
    [
    jupyterWithStuff
    <other pkgs here>
    ];

The with nixos-rebuild switch is:

error: value is a boolean while a set was expected, at /nix/store/1rx1bb767clgdhqnhh083ryf35zg6ihh-nixos-19.03pre169895.fa82ebccf66/nixos/pkgs/applications/editors/jupyter/kernel.nix:46:20

Which is clearly a sign for a wrong override.

@Ninlives Can you tell me what I’m missing?

This is relevant to the question above:
https://www.tweag.io/posts/2019-02-28-jupyter-with.html

6 Likes

This seems nice for bigger projects or such but it feels a bit like overkill for me. It’s downloading and compiling a lot of stuff for 45 minutes now. (Only python with pandas, numpy and matplotlib is configured)

In the meanwhile I’m still trying to figure out the correct overwrite for the jupyter package…

@noir: jupyterWith usually takes long for the first build. If you later modify the python libraries it is very quick because of the local caching. But I agree that it might be overkill if you just want a few standard python libraries.

The long initial build times of jupyterWith are to a large extent due to the nixpkgs version and overrides that are used. The goal is to merge the overrides step-by-step into nixpkgs to have them cached and to be able to set the environment up in a way similar to the one you suggest above.

1 Like

I wrote a Nix derivation to run a Jupyter notebook with libraries like pytorch, numpy, matplotlib, etc. You might be able to get some ideas from this:

I also wrote up a blog post explaining it a little bit:

https://functor.tokyo/blog/2019-03-16-nix-derivation-for-fastai

I did this because I wanted to go through the fast.ai course on deep learning. I’ve gone through two weeks of the course and so far everything in the Jupyter notebooks from the course is working well.

The only hacky thing is support for CUDA. You can read the blog post to get a little more information about how CUDA support works.

Unfortunately I haven’t used jupyterWith, so I can’t compare it to my solution above.

6 Likes

nice @cdepillabout ! For clarification: jupyterWith’s main purpose is support for jupyterlab and also for various kernels other than Python.

2 Likes

@noir This may be a little too late, but the way to override the definitions attribute in jupyter is to override the whole definition’s attribute, and not just a part of it.

jupyterWithStuff = pkgs.jupyter.override {
      definitions = {
        python3 = let
          env = (pkgs.python3.withPackages(ps: with ps; [
            numpy
            matplotlib
            pandas
          ]));
        in {
          displayName = "Python 3";
          argv = [
            "${env.interpreter}"
            "-m"
            "ipykernel_launcher"
            "-f"
            "{connection_file}"
          ];
          language = "python";
          logo32 = "${env.sitePackages}/ipykernel/resources/logo-32x32.png";
          logo64 = "${env.sitePackages}/ipykernel/resources/logo-64x64.png";
        };
      };
    };

Installing jupyterWithStuff now makes all these packages available to jupyter-notebook.

5 Likes

To circumvent the problem with long initial build times and having to write lots of overrides, you can use mach-nix. It speeds up your build times by fetching pre-built binary releases from pypi or anaconda.
You can reproducibly build directly from a requirements.txt or environment.yml file.
For packages that still need to be built from source, it inherits the build recipe from nixpkgs and adapts it for the required package version which reduces the necessity for manual overrides.

Here an example with jupyterWith + mach-nix:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/3.0.2";  # update this version
  }) {
    python = "python37";
  };

  # load your requirements
  machNix = mach-nix.mkPython rec {
    requirements = builtins.readFile ./requirements.txt;
  };

  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    ref = "master";
    #rev = "some_revision";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "mach-nix-jupyter";
    python3 = machNix.python;
    packages = machNix.python.pkgs.selectPkgs;
  };

  jupyterEnvironment = jupyter.jupyterlabWith {
    kernels = [ iPython ];
  };
in
  jupyterEnvironment.env
3 Likes