Issues with Dependency Resolution in Custom Nix Derivations for Otter-Grader

Help
1

Hi, folks!

I’m working on creating a development environment using nix-direnv that includes otter-grader. However, otter-grader and two of its dependencies (fica and ipylab) are not currently available in nixpkgs, so I’ve had to create custom derivations for them.

The problem I’m encountering is that otter-grader does not seem to recognize ipylab during its pythonRuntimeDepsCheckHook. However, when I build the flake without otter-grader, it works correctly, and I can load ipylab as a Python module in the resulting environment.

To summarize:

Issue

The build process for otter-grader consistently fails at the pythonRuntimeDepsCheckHook, with the error message indicating that ipylab is not installed.

The resulting error from `direnv reload` 
error: builder for '/nix/store/vp51ffmi1lfbr7fa77qv37mdwlqvb6ij-otter-grader-5.5.0.drv' failed with exit code 1;
       last 10 log lines:
       > Running phase: pythonRuntimeDepsCheckHook
       > Executing pythonRuntimeDepsCheck
       > Checking runtime dependencies for otter_grader-5.5.0-py3-none-any.whl
       >   - ipylab not installed
       For full logs, run 'nix log /nix/store/vp51ffmi1lfbr7fa77qv37mdwlqvb6ij-otter-grader-5.5.0.drv'.
error: 1 dependencies of derivation '/nix/store/xlpb90a1n44bvymgnvkcfhnsgq3qklh5-nix-shell-env.drv' failed to build
flake.nix (used by direnv) 
{
  description = "A flake to manage dependencies for otter-grader and related packages";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = {
    self,
    nixpkgs,
    flake-utils,
  }:
    flake-utils.lib.eachDefaultSystem (
      system: let
        pkgs = import nixpkgs {inherit system;};
      in {
        packages = {
          fica = pkgs.callPackage ./pkgs/fica/default.nix {};
          ipylab = pkgs.callPackage ./pkgs/ipylab/default.nix {};
          otter-grader = pkgs.callPackage ./pkgs/otter-grader/default.nix {
            inherit (pkgs) lib python3 fetchFromGitHub;
            fica = self.packages.${system}.fica;
            ipylab = self.packages.${system}.ipylab;
          };
        };
        devShells.default = pkgs.mkShell {
          buildInputs = [
            pkgs.python3
            pkgs.nodejs
            pkgs.git
            pkgs.curl
            pkgs.wget
            pkgs.zsh
            pkgs.direnv
            self.packages.${system}.fica
            self.packages.${system}.ipylab
            self.packages.${system}.otter-grader
            pkgs.python3.pkgs.ipykernel
            pkgs.python3.pkgs.pandas
            pkgs.python3.pkgs.matplotlib
            pkgs.python3.pkgs.scipy
            pkgs.python3.pkgs.scikit-learn
            pkgs.python3.pkgs.jupyter
            pkgs.python3.pkgs.jupyterlab
            pkgs.python3.pkgs.numpy
            pkgs.python3.pkgs.notebook
            pkgs.python3.pkgs.plotly
            pkgs.python3.pkgs.seaborn
            pkgs.python3.pkgs.statsmodels
            pkgs.python3.pkgs.beautifulsoup4
            pkgs.python3.pkgs.requests
            pkgs.python3.pkgs.lxml
            pkgs.python3.pkgs.sqlalchemy
            pkgs.python3.pkgs.nbconvert
            pkgs.python3.pkgs.jupytext
            pkgs.python3.pkgs.tqdm
            pkgs.python3.pkgs.pytest
            pkgs.python3.pkgs.sphinx
          ];
        };
      }
    );
}
default.nix (otter-grader derivation) 
{
  lib,
  python3,
  fetchFromGitHub,
  fica,
  ipylab,
}:
python3.pkgs.buildPythonApplication rec {
  pname = "otter-grader";
  version = "5.5.0";
  pyproject = true;

  src = fetchFromGitHub {
    owner = "ucbds-infra";
    repo = "otter-grader";
    rev = "v${version}";
    hash = "sha256-RXdZDtTaSkkS0oahjOoMVXMbUlidC3EoP36FqIwdqvo=";
  };

  nativeBuildInputs = [
    python3.pkgs.setuptools
    python3.pkgs.wheel
  ];

  propagatedBuildInputs =
    (with python3.pkgs; [
      astunparse
      click
      dill
      fica
      ipylab
      ipython
      ipywidgets
      jinja2
      jupytext
      nbconvert
      nbformat
      pandas
      python-on-whales
      pyyaml
      requests
      wrapt
    ])
    ++ [
      fica
      ipylab
    ];

  pythonImportsCheck = ["otter_grader"];

  # Phases
  preBuild = ''
    export PYTHONPATH=$PYTHONPATH:${ipylab}/${python3.sitePackages}
  '';

  preCheck = ''
    echo "Checking runtime dependencies for otter_grader..."
    python3 -c "import ipylab; print(ipylab.__version__)"
  '';

  meta = with lib; {
    description = "A Python and R autograding solution";
    homepage = "https://github.com/ucbds-infra/otter-grader";
    changelog = "https://github.com/ucbds-infra/otter-grader/blob/${src.rev}/CHANGELOG.md";
    license = licenses.bsd3;
    maintainers = with maintainers; [];
    mainProgram = "otter-grader";
  };
}

Suspected Culprit

I suspect the issue may be related to how dependencies are being referenced or propagated, but I’m not sure.

Request for Help

  • Build Order: How can I ensure ipylab is built and available before otter-grader starts its build process? (I would have though the fact that it is a dependency would ensure this.)
  • Dependency Propagation: Are there any better ways to handle the dependency propagation in my setup? General ‘best practices’ derivation tips after reading my half-baked nix code would be wonderful too!
  • Debugging Tips: Any tips on how to better debug this issue would be appreciated.

This is my first foray into making a custom derivation. I’d like to eventually contribute these to nixpkgs if I can get all the bugs worked out :slight_smile:

Thanks for your help!