Hey,
I am trying to install packages from pypi that are not in nixos in my system environment. So far I have something like that:
my-python-packages = python-packages:
with python-packages; [
requests
beautifulsoup4
jupyter
pandas
numpy
matplotlib
];
python-with-my-packages = python3.withPackages my-python-packages;
environment.systemPackages = with pkgs; [ python-with-my-packages ];
I am trying to use either mach-nix or poetry2nix to add more packages easily but fail. I am using flakes and don’t understand how I can define the python version of mach-nix. It somehow uses python39 but since I am on nixos unstable I would like to set it to python310.
This is how I defined the mach-nix py env:
mach_nix_py_env = mach-nix.lib."x86_64-linux".mkPython {
requirements = ''
requests
beautifulsoup4
jupyter
pandas
numpy
matplotlib
'';
};
In the other approach poetry2nix tries to compile everything from scratch. Why is that? also It fails with: ModuleNotFoundError: No module named ‘setuptools’
Any help in either direction much appreciated!
I haven’t done this, but here’s an example of a flake where someone appears to be setting some options (including the python version) from mach-nix/examples.md at 913e6c16f986746ba5507878ef7ff992804d1fa8 · DavHau/mach-nix · GitHub .
I’m not certain, but I think it’s common enough for python packages to have undeclared dependencies on setuptools since it is typically present in a traditional python env. Try adding it to your requirements.
What are the names of the packages which exist in pypi but not in nixpkgs? I might try them with my poetry2nix flakes and test whether they work in poetry2nix! I’ll let you know whether they work and if they work I will share the configs!
Those packages are for example: pykson or ebooklib
This is how I try to define a systemwide python environment with python packages installed by poetry:
let
poetry_py_env = pkgs.poetry2nix.mkPoetryEnv { projectDir = ./poetry_pkgs; };
in {
environment.systemPackages = with pkgs; [
poetry_py_env
];
}
pykson and ebooklib can be packaged with the following files.
flake.nix
Summary
{
description = "Application packaged using poetry2nix";
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
inputs.poetry2nix = {
url = "github:nix-community/poetry2nix";
inputs.nixpkgs.follows = "nixpkgs";
};
outputs = { self, nixpkgs, flake-utils, poetry2nix }:
{
# Nixpkgs overlay providing the application
overlay = nixpkgs.lib.composeManyExtensions [
poetry2nix.overlay
(final: prev: {
poetry2nix = prev.poetry2nix.overrideScope' (p2nixfinal: p2nixprev: {
# pyfinal & pyprev refers to python packages
defaultPoetryOverrides = (p2nixprev.defaultPoetryOverrides.extend (pyfinal: pyprev:
{
## dodge infinite recursion ##
setuptools = prev.python310Packages.setuptools.override {
inherit (pyfinal)
bootstrapped-pip
pipInstallHook
setuptoolsBuildHook
;
};
setuptools-scm = prev.python310Packages.setuptools-scm.override {
inherit (pyfinal)
packaging
typing-extensions
tomli
setuptools;
};
pip = prev.python310Packages.pip.override {
inherit (pyfinal)
bootstrapped-pip
mock
scripttest
virtualenv
pretend
pytest
pip-tools
;
};
## dodge infinite recursion (end) ##
ebooklib = pyprev.ebooklib.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.setuptools
]);
});
pykson = pyprev.pykson.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.setuptools
]);
src = final.fetchFromGitHub {
owner = "sinarezaei";
repo = "pykson";
rev = "master";
sha256 = "sha256-hyW6cemGNTBD0HUFwBCTmt3WHcmAYZAb9rjNlrJY1hs=";
};
});
}
));
});
})
(final: prev: {
# The application
myapp = prev.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
# The env
myenv = prev.poetry2nix.mkPoetryEnv {
projectDir = ./.;
};
})
];
} // (flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
overlays = [ self.overlay ];
};
in
{
#packages.default = pkgs.myapp;
#packages.myenv = pkgs.myenv;
packages.poetry = nixpkgs.legacyPackages.${system}.python310Packages.poetry;
devShells.default = pkgs.mkShell {
packages = with pkgs; [
#(python310.withPackages (ps: with ps; [ poetry ]))
myenv
];
};
}));
}
pyproject.toml
Summary
[tool.poetry]
name = "sepiabrown"
version = "0.1.0"
description = ""
authors = ["Suwon Park"]
[tool.poetry.dependencies]
python = "^3.10"
pykson = "*"
ebooklib = "*"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Copy flake.nix and pyproject.toml in the same folder and try the following in the folder.
nix shell .#poetrypoetry update --locknix developpythonimport pyksonimport ebooklib
I’m currently testing the version with requests, beautifulsoup4, jupyter, pandas, numpy, matplotlib. They seem to require a little bit more overriding. I’ll share the config when it is done. In the meantime, try the above files!
Edit : So here is the version with requests, beautifulsoup4, jupyter, pandas, numpy, matplotlib.
flake.nix
Summary
{
description = "Application packaged using poetry2nix";
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
inputs.poetry2nix = {
url = "github:nix-community/poetry2nix";
inputs.nixpkgs.follows = "nixpkgs";
};
outputs = { self, nixpkgs, flake-utils, poetry2nix }:
{
# Nixpkgs overlay providing the application
overlay = nixpkgs.lib.composeManyExtensions [
poetry2nix.overlay
(final: prev: {
poetry2nix = prev.poetry2nix.overrideScope' (p2nixfinal: p2nixprev: {
# pyfinal & pyprev refers to python packages
defaultPoetryOverrides = (p2nixprev.defaultPoetryOverrides.extend (pyfinal: pyprev:
{
## dodge infinite recursion ##
setuptools = prev.python310Packages.setuptools.override {
inherit (pyfinal)
bootstrapped-pip
pipInstallHook
setuptoolsBuildHook
;
};
setuptools-scm = prev.python310Packages.setuptools-scm.override {
inherit (pyfinal)
packaging
typing-extensions
tomli
setuptools;
};
pip = prev.python310Packages.pip.override {
inherit (pyfinal)
bootstrapped-pip
mock
scripttest
virtualenv
pretend
pytest
pip-tools
;
};
## dodge infinite recursion (end) ##
comm = pyprev.comm.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.hatchling
]);
});
ebooklib = pyprev.ebooklib.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.setuptools
]);
});
nbclient = pyprev.nbclient.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.hatchling
]);
});
packaging = pyprev.packaging.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.flit-core
]);
});
pykson = pyprev.pykson.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.setuptools
]);
src = final.fetchFromGitHub {
owner = "sinarezaei";
repo = "pykson";
rev = "master";
sha256 = "sha256-hyW6cemGNTBD0HUFwBCTmt3WHcmAYZAb9rjNlrJY1hs=";
};
});
jupyter-events = pyprev.jupyter-events.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.hatchling
]);
});
jupyter-server-terminals = pyprev.jupyter-server-terminals.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.hatchling
]);
});
rfc3986-validator = pyprev.rfc3986-validator.overridePythonAttrs (old: rec {
propagatedBuildInputs = builtins.filter (x: ! builtins.elem x [ ]) ((old.propagatedBuildInputs or [ ]) ++ [
pyfinal.setuptools
]);
});
}
));
});
})
(final: prev: {
# The application
myapp = prev.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
# The env
myenv = prev.poetry2nix.mkPoetryEnv {
projectDir = ./.;
};
})
];
} // (flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
config.allowUnfree = true;
overlays = [ self.overlay ];
};
in
{
#packages.default = pkgs.myapp;
#packages.myenv = pkgs.myenv;
packages.poetry = nixpkgs.legacyPackages.${system}.python310Packages.poetry;
#packages.poetry = pkgs.myenv.pkgs.poetry;
devShells.default = pkgs.mkShell {
packages = with pkgs; [
#(python310.withPackages (ps: with ps; [ poetry ]))
myenv
];
};
}));
}
pyproject.toml
Summary
[tool.poetry]
name = "sepiabrown"
version = "0.1.0"
description = ""
authors = ["Suwon Park"]
[tool.poetry.dependencies]
python = "^3.10"
requests = "*"
beautifulsoup4 = "*"
jupyter = "*"
notebook = "6.4.2" # nixos-22.11
pandas = "*"
numpy = "*"
matplotlib = "*"
pykson = "*"
ebooklib = "*"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
3 Likes
Hey sepiabrown, thanks! I get a shell with all the packages so it works for this case. I am just surprised how complex the solution turns out to be. I thought I can just declare python packages via poetry but as it turns out I also have to figure out the proper overrides. I am not sure what poetry2nix gets me then.
Those overrides happened partly because I’m using the latest stable release nixos-22.11 and keeping all python packages except notebook to its latest version in pyproject.toml. Older nixos releases and package versions would have less overrides. Also, if you get used to those overrides, it is quite mechanical. You just add those python packages that the error tells you ('module not found : ') to propagatedBuildInputs.
But at the same time, I have to admit that the solution I gave you is not fully satisfying for me either. It’s python after all. Obscure dependencies making many lives miserable!
If you haven’t already you can add the overrides you found to this file via a pull request.
{
"aadict": [
"setuptools"
],
"aafigure": [
"setuptools"
],
"aardwolf": [
"setuptools"
],
"abodepy": [
"setuptools"
],
"absl-py": [
"setuptools"
],
"absolufy-imports": [
"setuptools"
],
"accuweather": [
show original
1 Like
robert1
December 15, 2022, 8:34am
11
Great, thanks for making me aware of this.
1 Like
