Python flake does not install package as editable

Help
1

Hi everyone,

I’m new to nix, so apologies if I’ve missed something obvious.

I am trying to develop a python package using flakes, but I can’t seem to get my package to install in editable mode. Both direnv with use flake and nix develop install my package / dependencies successfully, but if I edit the local source files the changes aren’t reflected unless I rebuild the dev shell.

I am using buildPythonPackage with format = "pyproject", and as far as I can tell that should trigger an editable install (see pip build hook).

Have I done something wrong here? (I’d also welcome any suggestions to improve my flake setup, this is all still very new to me).

Thank you!

Directory structure

.
├── flake.lock
├── flake.nix
├── pyproject.toml
├── python-overlay.nix  # Defines dependencies missing from nixpkgs
├── setup.cfg
├── src
│  └── mypackage
│     ├── __init__.py
│     ├── ...
│     └── mysubmodule
│        ├── __init__.py
│        └── ...
└── tests
   └── ...

flake.nix

{
  description = "My description";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    let out = system:
      let
        useCuda = system == "x86_64-linux";
        pkgs = import nixpkgs {
          inherit system;
          config.cudaSupport = useCuda;
          config.allowUnfree = true;
          overlays = [
            (import ./python-overlay.nix)
          ];
        };
        python = pkgs.python310;
        pythonPackages = python.pkgs;

        pythonEnv = python.withPackages (ps: with ps; [
          pytorch
          numpy
          pandas
          transformers
          tokenizers
          tqdm
          accelerate
          altair
          altair-saver
          selenium
          # Dev dependencies
          black
          ipdb
          ipython
          isort
          pytest
        ]);
        buildInputs = with pkgs; [
          pythonEnv
          geckodriver
        ] ++ pkgs.lib.optionals useCuda (with pkgs; [
          pkgs.linuxPackages.nvidia_x11
        ]);
        defaultPackage = pythonPackages.buildPythonPackage {
          name = "mypackage";
          format = "pyproject";
          src = ./.;
          inherit buildInputs;
        };
      in
      {
        inherit defaultPackage;
        devShell = pkgs.mkShell {
          nativeBuildInputs = [
            defaultPackage
          ] ++ buildInputs;
          shellHook = ''
            export PYTHONFAULTHANDLER=1
            export PYTHONBREAKPOINT=ipdb.set_trace
            set -o allexport
            source .env
            set +o allexport
          '' + pkgs.lib.optionalString useCuda ''
            export CUDA_PATH=${pkgs.cudatoolkit}
            export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.ncurses5}/lib
            export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
            export EXTRA_CCFLAGS="-I/usr/include"
          '';
        };
      }; in with utils.lib; eachSystem defaultSystems out;
}
2

Support for editable packages using pyproject has not been implemented yet.

You are probably better off for now to export explicitly PYTHONPATH.

3

Ah, I see. Thank you!
Do you have an example of the best way to do that? I assume I would put it in the shellHook, but what would I write exactly?

4

For anyone who has the same issue in the future: I’m not sure if this is the best way to do it, but here is what I got working. I have a script called .env that gets called in shellHook:

shellHook

set -o allexport
source .env
set +o allexport

I added a one-liner to it that gets the directory containing the script, and I use that to add my package’s parent path to PYTHONPATH:
.env

SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]:-$0}"; )" &> /dev/null && pwd 2> /dev/null; )";
PYTHONPATH="$SCRIPT_DIR/src:$PYTHONPATH"

I tried to directly do export PYTHONPATH="${./.}/src:$PYTHONPATH" in shellHook, but ./. references a path to the nix store, not the local directory containing the flake.

If anyone knows of a cleaner way to do this, I’d love to know!

2 Likes
5

related resources:

6

For those coming that want an actual example please check out my project sqlelf - It builds with traditional Python setuptools (pyproject.toml) and also via Nix.

I have nix build working and nix develop.
Please see my flake: sqlelf/flake.nix at ae0241d25eb8a16759af1eca5f0d4c3aca8b0e83 · fzakaria/sqlelf · GitHub

Thanks @jonringer for the Python .venv example earlier.

7

There is one annoying thing at the moment:

pyright seems to be not packaged in pythonpackages but at the top level of nixpkgs.
The .venv automatic setup in the shell is pulling it in from PyPI instead and it’s at the start of my $PATH.
The PyPI wheel has a non-NixOS compliant binary in it so … doesn’t work.