Jupyter notebook with batteries


#1

How do I get a Python notebook with numpy, scipy, matplotlib and other essential libraries?


#2

You can run:
nix-shell -p pythonPackages.numpy pythonPackages.notebook then jupyter notebook from within the shell and add all other libraries you need if available. (use nix search to check availability)


#3

It’s in the Nixpkgs manual: python3.withPackages(ps: with ps; [ numpy scipy matplotlib ])

Avoid using nix-shell -p pythonPackages.numpy pythonPackages.notebook. It “works” because it uses a setup hook for setting up PYTHONPATH, but it is not guaranteed to work. Also, not all packages will propagate the interpreter.


#4

Is there a less ad-hoc way than nix-shell? Like installing a derivation with the required environment.


#5

Have you looked yet in the manual?


#6

Yes, I did. Should I read it once again?


#7

Maybe you want to create an overlay with the following contents:

self: super: {
   jupyterWithBatteries = super.jupyter.override {
      python3 = super.python3.withPackages(ps: with ps; [ numpy scipy matplotlib ]);
   };
} 

then run

nix-env -iA nixpkgs.jupyterWithBatteries

to install the modified jupyter


#8

I added the following into $HOME/.config/nixpkgs/config.nix:

  packageOverrides = pkgs: with pkgs; {
    jupyterWithBatteries = pkgs.jupyter.override {
      python3 = pkgs.python3.withPackages(ps: with ps; [
        numpy scipy matplotlib
      ]);
    };
  };

nix-env -iA nixos.jupyterWithBatteries installs jupyter but numpy and other libraries are not available in notebooks.


#9

Oh sorry, my fault :frowning: reviewed the source, you should override the definitions attribute, which looks like this: https://github.com/NixOS/nixpkgs/blob/beaf69cee298e092698dd2da2e4758b7811859ad/pkgs/applications/editors/jupyter/kernel.nix#L5


#10

I’m trying to install jupyter globally with some packages available but I don’t get the override right. It looks like this in my /etc/nixos/configuration.nix

environment.systemPackages = with pkgs;
    let jupyterWithStuff = pkgs.jupyter.override {
        definitions = pkgs.python3.withPackages( ps: with ps; [
          numpy matplotlib pandas
        ]);
      };
    in
    [
    jupyterWithStuff
    <other pkgs here>
    ];

The with nixos-rebuild switch is:

error: value is a boolean while a set was expected, at /nix/store/1rx1bb767clgdhqnhh083ryf35zg6ihh-nixos-19.03pre169895.fa82ebccf66/nixos/pkgs/applications/editors/jupyter/kernel.nix:46:20

Which is clearly a sign for a wrong override.

@Ninlives Can you tell me what I’m missing?


#11

This is relevant to the question above:


#12

This seems nice for bigger projects or such but it feels a bit like overkill for me. It’s downloading and compiling a lot of stuff for 45 minutes now. (Only python with pandas, numpy and matplotlib is configured)

In the meanwhile I’m still trying to figure out the correct overwrite for the jupyter package…


#13

@noir: jupyterWith usually takes long for the first build. If you later modify the python libraries it is very quick because of the local caching. But I agree that it might be overkill if you just want a few standard python libraries.

The long initial build times of jupyterWith are to a large extent due to the nixpkgs version and overrides that are used. The goal is to merge the overrides step-by-step into nixpkgs to have them cached and to be able to set the environment up in a way similar to the one you suggest above.


#14

I wrote a Nix derivation to run a Jupyter notebook with libraries like pytorch, numpy, matplotlib, etc. You might be able to get some ideas from this:

I also wrote up a blog post explaining it a little bit:

https://functor.tokyo/blog/2019-03-16-nix-derivation-for-fastai

I did this because I wanted to go through the fast.ai course on deep learning. I’ve gone through two weeks of the course and so far everything in the Jupyter notebooks from the course is working well.

The only hacky thing is support for CUDA. You can read the blog post to get a little more information about how CUDA support works.

Unfortunately I haven’t used jupyterWith, so I can’t compare it to my solution above.


#15

nice @cdepillabout ! For clarification: jupyterWith’s main purpose is support for jupyterlab and also for various kernels other than Python.