Python flake does not install package as editable

Help
1

Hi everyone,

I’m new to nix, so apologies if I’ve missed something obvious.

I am trying to develop a python package using flakes, but I can’t seem to get my package to install in editable mode. Both direnv with use flake and nix develop install my package / dependencies successfully, but if I edit the local source files the changes aren’t reflected unless I rebuild the dev shell.

I am using buildPythonPackage with format = "pyproject", and as far as I can tell that should trigger an editable install (see pip build hook).

Have I done something wrong here? (I’d also welcome any suggestions to improve my flake setup, this is all still very new to me).

Thank you!

Directory structure

.
├── flake.lock
├── flake.nix
├── pyproject.toml
├── python-overlay.nix  # Defines dependencies missing from nixpkgs
├── setup.cfg
├── src
│  └── mypackage
│     ├── __init__.py
│     ├── ...
│     └── mysubmodule
│        ├── __init__.py
│        └── ...
└── tests
   └── ...

flake.nix

{
  description = "My description";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    let out = system:
      let
        useCuda = system == "x86_64-linux";
        pkgs = import nixpkgs {
          inherit system;
          config.cudaSupport = useCuda;
          config.allowUnfree = true;
          overlays = [
            (import ./python-overlay.nix)
          ];
        };
        python = pkgs.python310;
        pythonPackages = python.pkgs;

        pythonEnv = python.withPackages (ps: with ps; [
          pytorch
          numpy
          pandas
          transformers
          tokenizers
          tqdm
          accelerate
          altair
          altair-saver
          selenium
          # Dev dependencies
          black
          ipdb
          ipython
          isort
          pytest
        ]);
        buildInputs = with pkgs; [
          pythonEnv
          geckodriver
        ] ++ pkgs.lib.optionals useCuda (with pkgs; [
          pkgs.linuxPackages.nvidia_x11
        ]);
        defaultPackage = pythonPackages.buildPythonPackage {
          name = "mypackage";
          format = "pyproject";
          src = ./.;
          inherit buildInputs;
        };
      in
      {
        inherit defaultPackage;
        devShell = pkgs.mkShell {
          nativeBuildInputs = [
            defaultPackage
          ] ++ buildInputs;
          shellHook = ''
            export PYTHONFAULTHANDLER=1
            export PYTHONBREAKPOINT=ipdb.set_trace
            set -o allexport
            source .env
            set +o allexport
          '' + pkgs.lib.optionalString useCuda ''
            export CUDA_PATH=${pkgs.cudatoolkit}
            export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.ncurses5}/lib
            export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
            export EXTRA_CCFLAGS="-I/usr/include"
          '';
        };
      }; in with utils.lib; eachSystem defaultSystems out;
}
2

Support for editable packages using pyproject has not been implemented yet.

You are probably better off for now to export explicitly PYTHONPATH.

3

Ah, I see. Thank you!
Do you have an example of the best way to do that? I assume I would put it in the shellHook, but what would I write exactly?

4

For anyone who has the same issue in the future: I’m not sure if this is the best way to do it, but here is what I got working. I have a script called .env that gets called in shellHook:

shellHook

set -o allexport
source .env
set +o allexport

I added a one-liner to it that gets the directory containing the script, and I use that to add my package’s parent path to PYTHONPATH:
.env

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]:-$0}"; )" &> /dev/null && pwd 2> /dev/null; )";
PYTHONPATH="$SCRIPT_DIR/src:$PYTHONPATH"

I tried to directly do export PYTHONPATH="${./.}/src:$PYTHONPATH" in shellHook, but ./. references a path to the nix store, not the local directory containing the flake.

If anyone knows of a cleaner way to do this, I’d love to know!

1 Like
5

related resources: