Criticism of the new jupyterWith

I was previously a big proponent of jupyterWith and I used it as a convenient way to serve up stable diffusion before it became jupyenv. But since the transition I can no longer recommend jupyenv.

jupyterWith

For me jupyterWith was the perfect way to enable jupyterLab with python packages from nixpkgs. Just throw whatever you want into packages and you’re good to go. eg:

      iPython = pkgs.kernels.iPythonWith {
        name = "Python-env";
        packages = p: with p; [jax jaxlib-bin openai];
        ignoreCollisions = true;
      };

All of this also fit nicely in a single flake. Here’s what I use to get a minimal jax environment with access to my GPU. (ommiting inputs and outputs for brevity)

    flake-utils.lib.eachSystem ["x86_64-linux"] (system: let
      pkgs = import nixpkgs {
        inherit system;
        overlays = [nixgl.overlay] ++ (nixpkgs.lib.attrValues jupyterWith.overlays);
        config.allowUnfree = true;
      };

      iPython = pkgs.kernels.iPythonWith {
        name = "Python-env";
        packages = p: with p; [jax jaxlib-bin openai];
        ignoreCollisions = true;
      };

      jupyterEnvironment = pkgs.jupyterlabWith {
        kernels = [iPython];
      };

      jupyterWrapped = pkgs.writeShellScriptBin "jupyter" ''
        #!/bin/sh
        ${pkgs.nixgl.auto.nixGLDefault}/bin/nixGL ${jupyterEnvironment}/bin/jupyter-lab "$@"
      '';
    in rec {
      apps.jupyterLab = {
        type = "app";
        program = "${jupyterWrapped}/bin/jupyter";
      };
      packages.default = jupyterWrapped;
      apps.default = apps.jupyterLab;
      devShells.default = jupyterEnvironment.env;
    });

jupyenv

jupyenv has a different way of doing things. The recommended project structure is creating a separate kernels file (kernels.nix). Which is fine but not necessarily the greatest ergonomics when it comes to people like me who are used to putting everything in a flake, especially when your kernels file looks like this most of the time.

{pkgs, ...}: {
  kernel.python.deep-learning = {
    enable = true;
    extraPackages = ps: with ps; [jax jaxlib-bin openai];
  };
}

Although this is easy enough to remedy with the following in a single flake. Here’s what that flake looks like:

    flake-utils.lib.eachSystem
    [
      flake-utils.lib.system.x86_64-linux
    ]
    (
      system: let
        inherit (jupyenv.lib.${system}) mkJupyterlabNew;

        pkgs = import nixpkgs {
          inherit system;
          overlays = [nixgl.overlay];
          config.allowUnfree = true;
        };
        kernels = {
          kernel.python.minimal = {
            enable = true;
            extraPackages = ps: with ps; [jax jaxlib-bin openai];
          };
        };
        jupyterlab = mkJupyterlabNew ({...}: {
          nixpkgs = inputs.nixpkgs;
          imports = [(import kernels)];
        });
      in rec {
        packages = {inherit jupyterlab;};
        packages.default = jupyterlab;

        jupyterWrapped = pkgs.writeShellScriptBin "jupyter-lab" ''
          #!/bin/sh
          ${pkgs.nixgl.auto.nixGLDefault}/bin/nixGL ${jupyterlab}/bin/jupyter-lab "$@"
        '';

        apps.${system}.default = {
          type = "app";
          program = "${jupyterWrapped}/bin/jupyter-lab";
        };
      }
    );

however this fails with the following error:

error: builder for '/nix/store/rfia0fx39qqic23a79xnlrplz86jd211-python3.10-openpyxl-3.0.10.drv' failed with exit code 1;
       last 10 log lines:
       > adding 'openpyxl-3.0.10.dist-info/RECORD'
       > removing build/bdist.linux-x86_64/wheel
       > Finished executing setuptoolsBuildPhase
       > installing
       > Executing pipInstallPhase
       > /build/openpyxl-3.0.10/dist /build/openpyxl-3.0.10
       > Processing ./openpyxl-3.0.10-py3-none-any.whl
       > ERROR: Could not find a version that satisfies the requirement et-xmlfile (from openpyxl) (from versions: none)
       > ERROR: No matching distribution found for et-xmlfile
       > 
       For full logs, run 'nix log /nix/store/rfia0fx39qqic23a79xnlrplz86jd211-python3.10-openpyxl-3.0.10.drv'.

So it worked in the previous version of jupyterWith and now it doesn’t work. This is probably due to the fact that now jupyenv uses poetry2nix to manage python dependencies which is not great considering I don’t care about versioning in the case where I just want to play around with some libraries. I also have no idea how to diagnose this.

I can figure out that it’s tied to the openai dependency and I guess I could put everything into a poetry project and try from there but I don’t see a reason to when I could just change the revision back to jupyterWith and use that interface.

I would normally make this an issue on GitHub but there seems to only be radio silence there since march with the repository having no activity recently.

Addressing the Tweag announcement blog post

The jupyenv announcement post explains that the refactor of jupyterWith into jupyenv is designed to appeal to those who are unfamiliar with the nix ecosystem and who might already have a poetry project.

I would argue that exposing them to poetry2nix would alienate them from using nix considering the massive amount of build failure issues on poetry2nix. This problem isn’t jupyenv’s fault, or poetry2nix’s fault or even poetry’s fault but more of the fault of python’s packaging system. I’m arguing that jupyenv is inheriting poetry2nix’s issues without the ability to properly triage these issues.

Summary

The old interface for jupyterWith worked for users who just wanted to try out python packages in a juypter notebook. The new interface jupyenv brings a plague of issues associated with poetry with no ability to use the old way of doing things.

I’m also not the only one who misses the old interface:
https://github.com/tweag/jupyenv/issues/485
https://github.com/tweag/jupyenv/issues/328

8 Likes

I’m with you, and also am struggling to find a minimal stable jupyter environment with nix. Currently manually hijacking PYTHONPATH in a shell hook, but facing broken things like jupyter nbconvert etc.

I agree that usage of poetry2nix requires very significant knowledge of Nix, knowledge of poetry2nix and Python packaging.

I have always used just

nix-shell -p 'python3.withPackages(ps: with ps; [ numpy pandas jupyter ])' --run jupyter-notebook

etc. whenever I needed Jupyter. Is the issue with the packages which are not in nixpkgs? Or multiple kernels?

1 Like

this solution would be here: refactor/move-kernelfunc by djacu · Pull Request #462 · tweag/jupyenv · GitHub
but I’m still waiting for @djacu to deal with that.
then, you can use the option of env to declare a pure kernelEnv(without poetry2nix), such as:

{
  kernel.python.minimal = {
   enable = true;
 env  = python3.withPackages(ps: with ps; [ numpy pandas jupyter ]);
};
}

Right, It’s tough for users to solve the dependencies and add overrides for poetry2nix.
That’s why I added a new template Templates: enrich the template scenes by GTrunSec · Pull Request #452 · tweag/jupyenv · GitHub.

I think we should collect users’ use cases and scenario configurations to face various challenging problems.

experimental: GitHub - GTrunSec/desci-workflow-template: A collection/template of automation tasks(BPMN, DMN, prefect, julia,python,yaml,quarto).