There are plenty of tools for using an already existing python package with nix but I was not finding much on how to develop a new python package with nix that could be easily worked on by non-nix developers. I set up a python package using a flake. My first attempt was to have the setup.py script under the buildPythonPackage. This works but then there is no setup.py in the repository (it’s generated in the /nix/store while building the package. This makes it difficult for a non-nix user to work on the project (especially if they are going to create a python virtualenv).
The easy solution would be to just write a setup.py but there’s a lot redundancy with the nix flake where I could easily miss bumping a version or adding a dependency. There is also no easy way to get the dependency version requirements from nix to specify in the setup.py.
To get around this I’ve attempted to have the flake write a pyproject.toml that works with poetry by running a shellHook that generates the pyproject.toml with provided values if the flake.nix is newer than the pyproject.toml or the pyproject.toml does not exist.
The, slightly stripped down, initial attempt is here:
# flake.nix
{
description =
"A python package for storing and working with publication data in graph form.";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-22.05";
flake-utils = {
url = "github:numtide/flake-utils";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = { self, nixpkgs, flake-utils, pubmedparser }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = nixpkgs.legacyPackages.${system};
python = pkgs.python3;
# Nix does not expose `checkInputs` attribute.
pubnetCheckInputs = (with python.pkgs; [ pytest pytest-snapshot ]);
pubnet = python.pkgs.buildPythonPackage rec {
pname = "pubnet";
version = "0.1.0";
src = ./.;
format = "pyproject";
buildInputs = (with python.pkgs; [ poetry ]);
propagatedBuildInputs = (with python.pkgs; [ numpy pandas scipy ]);
checkInputs = pubnetCheckInputs;
authors = [ "David Connell <davidconnell12@gmail.com>" ];
checkPhase = ''
python -m pytest
'';
};
nix2poetryDependency = list:
builtins.concatStringsSep "\n" (builtins.map (dep:
let
pname = if dep.pname == "python3" then "python" else dep.pname;
versionList = builtins.splitVersion dep.version;
major = builtins.elemAt versionList 0;
minor = builtins.elemAt versionList 1;
version = if pname == "python" then
''\"~${major}.${minor}\"''
else
''\"^${major}.${minor}\"'';
in pname + " = " + version) list);
in {
packages.pubnet = pubnet;
defaultPackage = self.packages.${system}.pubnet;
devShell = pkgs.mkShell {
packages = [
(python.withPackages (p:
with p;
[
ipython
python-lsp-server
pyls-isort
python-lsp-black
pylsp-mypy
] ++ pubnet.propagatedBuildInputs ++ pubnetCheckInputs))
];
shellHook = ''
export PYTHONPATH=.
if [ ! -f pyproject.toml ] || \
[ $(date +%s -r flake.nix) -gt $(date +%s -r pyproject.toml) ]; then
pname=${pubnet.pname} \
version=${pubnet.version} \
description='A python package for storing and working with publication data in graph form.' \
license=MIT \
authors="${
builtins.concatStringsSep ",\n "
(builtins.map (name: ''\"'' + name + ''\"'') pubnet.authors)
}" \
dependencies="${
nix2poetryDependency pubnet.propagatedBuildInputs
}" \
devDependencies="${nix2poetryDependency pubnetCheckInputs}" \
./.pyproject.toml.template
fi
'';
};
});
}
Which calls the shell script.
#!/usr/bin/env bash
# .pyproject.toml.template
cat >pyproject.toml <<_EOF_
[tool.black]
line-length = 79
[tool.poetry]
name = "$pname"
version = "$version"
description = "$description"
license = "$license"
authors = [
$authors,
]
packages = [
{ include = "$pname" },
]
[tool.poetry.dependencies]
$dependencies
[tool.poetry.dev-dependencies]
$devDependencies
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
_EOF_
And results in:
# pyproject.toml
[tool.black]
line-length = 79
[tool.poetry]
name = "pubnet"
version = "0.1.0"
description = "A python package for storing and working with publication data in graph form."
license = "MIT"
authors = [
"David Connell <davidconnell12@gmail.com>",
]
packages = [
{ include = "pubnet" },
]
[tool.poetry.dependencies]
numpy = "^1.21"
pandas = "^1.4"
scipy = "^1.8"
python = "~3.9"
[tool.poetry.dev-dependencies]
pytest = "^7.1"
pytest-snapshot = "^0.9"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
The devShell itself doesn’t actually contain the results of building the python package, instead it creates a python environment with all the runtime and testing dependencies plus all my development dependencies, then adds the flake to PYTHONPATH so python sees the working source code rather than the packaged code in the /nix/store allowing me to work on it with running python process (and IPython can be set to reload the modules as I write to them). Another flake, however, can use the resulting nix package in a python.withPackages call.
I’m not entirely sure about how to deal with dependency requirements. I don’t think python’s method for handling dependencies is compatible with nix since all python packages have access to the same copy of all runtime packages. Meaning if two packages depend on the same package, there has to be overlap in what versions they will accept. Nix wants to give each package the exact version it asks for, but only allowing a single dependency version would make it difficult for python to resolve dependencies. As a compromise I tried to set dependency to having at least the minor version but less than the next major version (i.e. the “^” notation) and for python the slightly stricter “~” notation since all packages you need must be compatible with all the python versions you promise the package to work with and many packages cap the python version to a minor version (say 3.12) which is stricter than the <=4.0 you get with the “^” notation. But this does not allow you to explicitly define which package versions your package works with. You could potentially define a seperate package for each python version it’s compatible be with, which should work for nix but I’m not sure how that would work with python/poetry since you’d need a seperate pyproject.toml for each python version.
May be relavent to: How to build python package for use in non-nix environments. Should be able to add poetry build to postBuild and move the resulting sdist or wheel to $out.