I was previously a big proponent of jupyterWith and I used it as a convenient way to serve up stable diffusion before it became jupyenv. But since the transition I can no longer recommend jupyenv.
jupyterWith
For me jupyterWith was the perfect way to enable jupyterLab with python packages from nixpkgs. Just throw whatever you want into packages and you’re good to go. eg:
iPython = pkgs.kernels.iPythonWith {
name = "Python-env";
packages = p: with p; [jax jaxlib-bin openai];
ignoreCollisions = true;
};
All of this also fit nicely in a single flake. Here’s what I use to get a minimal jax environment with access to my GPU. (ommiting inputs and outputs for brevity)
flake-utils.lib.eachSystem ["x86_64-linux"] (system: let
pkgs = import nixpkgs {
inherit system;
overlays = [nixgl.overlay] ++ (nixpkgs.lib.attrValues jupyterWith.overlays);
config.allowUnfree = true;
};
iPython = pkgs.kernels.iPythonWith {
name = "Python-env";
packages = p: with p; [jax jaxlib-bin openai];
ignoreCollisions = true;
};
jupyterEnvironment = pkgs.jupyterlabWith {
kernels = [iPython];
};
jupyterWrapped = pkgs.writeShellScriptBin "jupyter" ''
#!/bin/sh
${pkgs.nixgl.auto.nixGLDefault}/bin/nixGL ${jupyterEnvironment}/bin/jupyter-lab "$@"
'';
in rec {
apps.jupyterLab = {
type = "app";
program = "${jupyterWrapped}/bin/jupyter";
};
packages.default = jupyterWrapped;
apps.default = apps.jupyterLab;
devShells.default = jupyterEnvironment.env;
});
jupyenv
jupyenv has a different way of doing things. The recommended project structure is creating a separate kernels file (kernels.nix). Which is fine but not necessarily the greatest ergonomics when it comes to people like me who are used to putting everything in a flake, especially when your kernels file looks like this most of the time.
{pkgs, ...}: {
kernel.python.deep-learning = {
enable = true;
extraPackages = ps: with ps; [jax jaxlib-bin openai];
};
}
Although this is easy enough to remedy with the following in a single flake. Here’s what that flake looks like:
flake-utils.lib.eachSystem
[
flake-utils.lib.system.x86_64-linux
]
(
system: let
inherit (jupyenv.lib.${system}) mkJupyterlabNew;
pkgs = import nixpkgs {
inherit system;
overlays = [nixgl.overlay];
config.allowUnfree = true;
};
kernels = {
kernel.python.minimal = {
enable = true;
extraPackages = ps: with ps; [jax jaxlib-bin openai];
};
};
jupyterlab = mkJupyterlabNew ({...}: {
nixpkgs = inputs.nixpkgs;
imports = [(import kernels)];
});
in rec {
packages = {inherit jupyterlab;};
packages.default = jupyterlab;
jupyterWrapped = pkgs.writeShellScriptBin "jupyter-lab" ''
#!/bin/sh
${pkgs.nixgl.auto.nixGLDefault}/bin/nixGL ${jupyterlab}/bin/jupyter-lab "$@"
'';
apps.${system}.default = {
type = "app";
program = "${jupyterWrapped}/bin/jupyter-lab";
};
}
);
however this fails with the following error:
error: builder for '/nix/store/rfia0fx39qqic23a79xnlrplz86jd211-python3.10-openpyxl-3.0.10.drv' failed with exit code 1;
last 10 log lines:
> adding 'openpyxl-3.0.10.dist-info/RECORD'
> removing build/bdist.linux-x86_64/wheel
> Finished executing setuptoolsBuildPhase
> installing
> Executing pipInstallPhase
> /build/openpyxl-3.0.10/dist /build/openpyxl-3.0.10
> Processing ./openpyxl-3.0.10-py3-none-any.whl
> ERROR: Could not find a version that satisfies the requirement et-xmlfile (from openpyxl) (from versions: none)
> ERROR: No matching distribution found for et-xmlfile
>
For full logs, run 'nix log /nix/store/rfia0fx39qqic23a79xnlrplz86jd211-python3.10-openpyxl-3.0.10.drv'.
So it worked in the previous version of jupyterWith and now it doesn’t work. This is probably due to the fact that now jupyenv uses poetry2nix to manage python dependencies which is not great considering I don’t care about versioning in the case where I just want to play around with some libraries. I also have no idea how to diagnose this.
I can figure out that it’s tied to the openai dependency and I guess I could put everything into a poetry project and try from there but I don’t see a reason to when I could just change the revision back to jupyterWith and use that interface.
I would normally make this an issue on GitHub but there seems to only be radio silence there since march with the repository having no activity recently.
Addressing the Tweag announcement blog post
The jupyenv announcement post explains that the refactor of jupyterWith into jupyenv is designed to appeal to those who are unfamiliar with the nix ecosystem and who might already have a poetry project.
I would argue that exposing them to poetry2nix would alienate them from using nix considering the massive amount of build failure issues on poetry2nix. This problem isn’t jupyenv’s fault, or poetry2nix’s fault or even poetry’s fault but more of the fault of python’s packaging system. I’m arguing that jupyenv is inheriting poetry2nix’s issues without the ability to properly triage these issues.
Summary
The old interface for jupyterWith worked for users who just wanted to try out python packages in a juypter notebook. The new interface jupyenv brings a plague of issues associated with poetry with no ability to use the old way of doing things.
I’m also not the only one who misses the old interface: