You need to parenthesize the input, otherwise python3.withPackages and the lambda are separate list elements.
@jtojnar, I don’t understand where you’re saying I should put parentheses. I’ve tried putting parentheses in a few different places, but I still get:
21| stdenv.mkDerivation ({
22| name = "getBook";
| ^
23| src = ./.;
cannot coerce a function to a string
Like this.
Though to avoid that problem I prefer:
- use
letbindings - do not use
with
let myPython = python3.withPackages (ps: [ ps.numpy libgenApi ]); in
{
# …
buildinputs = [ myPython ];
}
Though that will probably not work for a python as you won’t have any runtime dependencies to your python with packages without doing some additional wrapping, similar will probably happen to the libgenApi.
Therefore you should prefer the python helpers like buildPythonPackage, buildPythonApplication or poetry2nix which do some of the nasty details for you.
Which module contains buildPythonPackage? It’s telling me it’s not in stdenv.
It is in the “top-level”, as most of the builders are.
Depending or your level of experience with Nix, anonymous functions, the tendency to call the latter lambdas, etc.; the above advice might range from spot-on-helpful to totally useless (or even worse, by raising more confusing questions than it answers).
So, in case it helps you, or anyone else in the future, here is an explanation of what it means.
Lambda
Firlstly, as you’re writing in Python, chances are you know what a lambda is in general, but you might not appreciate what they look like in Nix. The following are equivalent
lambda a: a + 1 # Python
a: a + 1 # Nix
They are both anonymous functions with one parameter a which return a + 1 when they are called. The top one is in Python, the bottom one in Nix.
(In other languages these might look like
|a| a + 1
[](int a) {return a+1;}
\a -> a + 1
(lambda (a) (+ a 1))
etc., etc., etc. but they all mean the same thing (modulo polymorphism))
Hopefully this clears up why the advice talks about ‘the lambda’ while ‘lambda’ does not appear anywhere in the code.
Role of parentheses in Nix
The way you write withPackages(ps: ...) with the opening-parenthesis right up against the function, suggests that you maybe think that the parentheses play the role of the function call syntax, like they do in Python (and many other languages). Beware! this is not the case in Nix!
This is how to call a function fn with two arguments a and b, first in Python, then in Nix
fn(a, b) # Python
fn a b # Nix
I recommend that you change the style in which you write function calls in Nix, putting a space between the function and its argument, even if the argument is in parentheses. That should help you remember that the role of parentheses is to group things together rather than calling functions.
(A discussion of currying would probably be more confusing than helpful to the majority of the target audience at this point, so I’ll leave it out.)
Question
What is the meaning of
fn(a b)
in Nix?
Answer
- First, call the function
awith the argumentb… - then call the function
fnwith the result of the previous step
Moral of the story: the parentheses are there for explicit precedence, not as a part of the function call syntax.
To someone used to parsing C-family languages, this can be very misleading: you might have to work hard to suppress your natural parsing instincts when reading and writing Nix!
Make sure you understand these:
fn a b # Call `fn` with two arguments
fn (a b) # Call `fn` with one argument (the result of calling `a` with one argument)
fn a # Call `fn` with one argument
fn(a) # Exactly the same as the previous line!
Nix list syntax
Note that the Nix list syntax, unlike Python (and many other languages), does not use commas to separate elements. List elements are separated by whitespace (… or parentheses!). But function arguments are also separated by whitespace … so if you put a function call inside a list, who wins? Which syntax claims the spaces (… or parentheses)?
You might be surprised by the answer:
[1 2 3] # 3 integers in a list
[fn a b] # 3 elements in a list
[fn (a b)] # 2 elements in ...
[fn(a b)] # 2 elements !!!!!! (Don't be fooled by your Python parsing habits!)
[fn a] # 2 elements
[fn(a)] # 2 elements !!!!!!
[(fn a b)] # 1 element (compare to `[fn(a b)]`, three lines up)
[(fn a)] # 1 element, c.f. `[fn(a)]` two lines up
What’s this got to do with the original problem?
Armed with this knowledge, let’s look at your problem. When you wrote
buildInputs = [python3.withPackages(ps: with ps; [ numpy libgenApi ])]
Nix parsed it as
buildInputs = [
python3.withPackages # first element in the list
(ps: with ps; [ numpy libgenApi ]) # second element in the list
]
- element 1: named function
python3.withPackages - element 2: anonymous function (lambda)
ps: with ps; [ numpy libgenApi ]
So we should now be able to fully understand @jtojnar’s statement that
python3.withPackagesand the lambda are separate list elements.
and, hopefully, now we also understand that we wrote something of the form [fn(a)] when we meant [(fn a)].
A bit more explicitly, we had something of the form
[fn(parameter: body)] # list with 2 elements: 1. named function, 2. lambda
where we need
[(fn (parameter: body))] # list with one element: the result of calling fn with a lambda as its sole argument
Conclusion
I hope that this serves to clear up a possible source of confusion in Nix, for someone.
15 Likes
Thanks for the very thorough explanation.
Ok here’s what I have so far:
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils, buildPythonPackage }: {
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = buildPythonPackage {
name = "libgen-api";
src = fetchFromGitHub {
owner = "harrison-broadbent";
repo = "libgen-api";
sha = "1304a6f21df12e6ee843a9315faf4229908c3a00";
};
};
in
buildPythonPackage {
name = "getBook";
src = ./.;
buildInputs = [ (python3.withPackages (ps: with ps; [ numpy libgenApi ])) ];
};
};
}
But I’m still having trouble finding where buildPythonPackage is. I either get “undefined variable ‘buildPythonPackage’” or “cannot find flake ‘flake:buildPythonPackage’ in the flake registries”
pkgs.python3.pkgs.buildPythonPackage or pkgs.python3Packages.buildPythonPackage.
If you want to be more specific about the python version, then replace the 3s with 38s (or 37s or 39s).
For this sort of thing, it’s quite useful to use a Nix repl:
nix repl -I nixpkgs=fetchTarball "https://github.com/NixOS/nixpkgs/archive/15a64b2facc1b91f4361bdd101576e8886ef834b.tar.gz"
should give you a nix-repl> prompt, where you can use tab-completion to explore what is where. In this repl you will be inside pkgs already.
[REPL: Read-Eval-Print-Loop, aka interactive ‘shell’]
Ok great, so it’s not at the top level after all, but in pkgs.python3Packages.
Here’s what I have so far:
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils }: {
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = pkgs.python3Packages.buildPythonPackage {
name = "libgen-api";
src = pkgs.python3Packages.fetchPypi "libgen-api";
};
in
pkgs.python3Packages.buildPythonPackage rec {
name = "getBook";
src = ./.;
buildInputs = [ (python3.withPackages (ps: with ps; [ numpy libgenApi ])) ];
};
};
}
but now I’m getting a bizarre error:
error: --- TypeError --------------------------------------------------------------------------- nix
at: (69:16) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix
68| let
69| result = f origArgs;
| ^
70|
value is a string while a set was expected
(use '--show-trace' to show detailed location information)
The full trace is:
jon@jon-laptop ~/C/getBook (master) [1]> nix build --show-trace
warning: Git tree '/home/jon/Code/getBook' is dirty
error: --- TypeError --------------------------------------------------------------------------- nix
at: (69:16) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix
68| let
69| result = f origArgs;
| ^
70|
value is a string while a set was expected
-------------------------------------------- show-trace --------------------------------------------
trace: while evaluating 'makeOverridable'
at: (67:24) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix
66| */
67| makeOverridable = f: origArgs:
| ^
68| let
trace: from call site
at: (14:17) in file: /nix/store/izzqlvb6nyfy06pdpyq2yw3fd8fx44s5-source/flake.nix
13| name = "libgen-api";
14| src = pkgs.python3Packages.fetchPypi "libgen-api";
| ^
15| };
trace: while evaluating the attribute 'src.name'
at: (14:11) in file: /nix/store/izzqlvb6nyfy06pdpyq2yw3fd8fx44s5-source/flake.nix
13| name = "libgen-api";
14| src = pkgs.python3Packages.fetchPypi "libgen-api";
| ^
15| };
trace: while evaluating 'hasSuffix'
at: (234:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/strings.nix
233| # Input string
234| content:
| ^
235| let
trace: from call site
at: (122:25) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix
121| pythonRemoveBinBytecodeHook
122| ] ++ lib.optionals (lib.hasSuffix "zip" (attrs.src.name or "")) [
| ^
123| unzip
trace: while evaluating 'optionals'
at: (270:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/lists.nix
269| # List to return if condition is true
270| elems: if cond then elems else [];
| ^
271|
trace: from call site
at: (122:10) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix
121| pythonRemoveBinBytecodeHook
122| ] ++ lib.optionals (lib.hasSuffix "zip" (attrs.src.name or "")) [
| ^
123| unzip
trace: while evaluating 'chooseDevOutputs'
at: (493:22) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/attrsets.nix
492| /* Pick the outputs of packages to place in buildInputs */
493| chooseDevOutputs = drvs: builtins.map getDev drvs;
| ^
494|
trace: from call site
trace: while evaluating the attribute 'nativeBuildInputs' of the derivation 'python3.8-libgen-api'
at: (111:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix
110|
111| name = namePrefix + name;
| ^
112|
trace: while evaluating the attribute 'outPath'
at: (164:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix
163| drvPath = assert condition; drv.drvPath;
164| outPath = assert condition; drv.outPath;
| ^
165| };
trace: while evaluating anonymous lambda
at: (645:24) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/lists.nix
644| */
645| unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
| ^
646|
trace: from call site
at: (64:6) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/top-level/python-packages.nix
63| modules = filter hasPythonModule drvs;
64| in unique ([python] ++ modules ++ concatLists (catAttrs "requiredPythonModules" modules));
| ^
65|
trace: while evaluating 'requiredPythonModules'
at: (62:27) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/top-level/python-packages.nix
61| # Get list of required Python modules given a list of derivations.
62| requiredPythonModules = drvs: let
| ^
63| modules = filter hasPythonModule drvs;
trace: from call site
at: (15:13) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/wrapper.nix
14| env = let
15| paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
| ^
16| pythonPath = "${placeholder "out"}/${python.sitePackages}";
trace: while evaluating the attribute 'passAsFile' of the derivation 'python3-3.8.7-env'
at: (7:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/build-support/trivial-builders.nix
6| stdenv.mkDerivation ({
7| name = lib.strings.sanitizeDerivationName name;
| ^
8| inherit buildCommand;
trace: while evaluating the attribute 'buildInputs' of the derivation 'python3.8-getBook'
at: (111:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix
110|
111| name = namePrefix + name;
| ^
112|
trace: while evaluating the attribute 'drvPath'
at: (163:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix
162| outputUnspecified = true;
163| drvPath = assert condition; drv.drvPath;
| ^
164| outPath = assert condition; drv.outPath;
The error message says it all, even though the position is confusing:
fetchPypi requires attribute set as an argument:
pkgs.python3Packages.fetchPypi {
pname = "libgen-api";
version = "something";
sha256 = "0000000000000000000000000000000000000000000000000000000000000000";
}
Another useful trick for finding where things are, is to clone (or fetch and unpack a tarball of) the nixpkgs repo. Then search that for occurrences of what you want, making sure to include a leading dot.
If you’re not already doing so, do yourself a favour and start using ripgrep. It’s in nixpkgs, so you can use Nix to install it. For instance (with a flake-enabled Nix[*])
nix shell nixpkgs#ripgrep -c rg -i '\.buildpythonpackage' path/to/your/nixpkgs/clone
shows me lots of valid, qualified, uses of buildPythonPackage, and it would do so even if I didn’t have ripgrep installed on my machine. Given that I do have ripgrep installed in my nix profile, I can omit the nix shell nixpkgs#ripgrep part, and get the same result, with just
rg -i '\.buildpythonpackage' path/to/your/nixpkgs/clone
[*] With a pre-flakes Nix, I might use
nix-shell -p --command "rg -i '\.buildpythonpack' path/to/your/nixpkgs/clone"
Edit:
I didn’t suggest searching directly in the GitHub web interface, because it gave me zero hits and made me question my sanity. This thread has both given me some peace of mind about my mental health, as well as pointing out another way of finding at you want, without the need to get a local copy of nixpkgs: https://search.nix.gsc.io/. Be sure to click on the “Advanced …” bar under the search box, and check the Ignore Case box (otherwise you’re bound to make some CamelCase error, and end up with zero hits), and then type \.builtpythonpackage in the search box.
1 Like
Hmm, I beg to differ. I think that there is plenty more than needs to be said, to make the required course of action clear to a Nix-non-expert.
1 Like
Good tips, thanks.
Now it builds, at long last. But when I enter the shell, it can’t find the Python modules I declared. (ModuleNotFoundError). What am I missing?
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils }: {
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = pkgs.python3Packages.buildPythonPackage rec {
pname = "libgen-api";
version = "0.3.0";
doCheck = false;
propagatedBuildInputs = with pkgs.python3Packages; [
setuptools
wheel
beautifulsoup4
];
src = (pkgs.python3Packages.fetchPypi {
inherit pname version;
sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
});};
in
pkgs.python3Packages.buildPythonPackage rec {
name = "getBook";
src = ./.;
propagatedBuildInputs = [ (python3.withPackages (ps: with ps; [ requests libgenApi ])) ];
};
};
}
If you are using nix shell, that puts you into a shell with your defaultPackage on PATH. If you want to develop with the package’s dependencies, you need nix develop. For that, you will also need a devShell output. devShell = self.defaultPackage; should probably be enough.
Hm, that doesn’t seem to work, I don’t think. Unless it’s just that I’m not putting that line in the right place? Here’s what I have so far:
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils }: {
devShell = self.defaultPackage;
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = pkgs.python3Packages.buildPythonPackage rec {
pname = "libgen-api";
version = "0.3.0";
doCheck = false;
propagatedBuildInputs = with pkgs.python3Packages; [
setuptools
wheel
beautifulsoup4
];
src = (pkgs.python3Packages.fetchPypi {
inherit pname version;
sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
});};
in
pkgs.python3Packages.buildPythonPackage rec {
name = "getBook";
src = ./.;
propagatedBuildInputs = [ (python3.withPackages (ps: with ps; [ requests libgenApi ])) ];
};
};
}
And here’s what happens when I try to run it inside nix develop or nix shell:
[jon@jon-laptop:~/Code/getBook]$ python getBook.py
Traceback (most recent call last):
File "getBook.py", line 10, in <module>
from libgen_api import LibgenSearch
ModuleNotFoundError: No module named 'libgen_api'
That should almost work, only you are mixing two different paradigms:
- python packages are created using
buildPythonPackageand take python dependencies inpropagatedBuildInputs - projects that need
Pythonwith some extra Python modules needpython.withPackages …in inputs
So I would replace
with either
pkgs.python3Packages.buildPythonPackage rec {
name = "getBook";
src = ./.;
propagatedBuildInputs = with python3Packages; [
requests
libgenApi
];
};
or
pkgs.stdenv.mkDerivation rec {
name = "getBook";
src = ./.;
buildInputs = [
(python3.withPackages (ps: with ps; [
requests
libgenApi
]))
];
};
depending on whether getBook is one or the other. Possibly buildPythonApplication, if it is application and not a module.
2 Likes
It seems I still get the same error, with both of those variations. Here’s the one I’m using now:
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils }: {
devShell = self.defaultPackage;
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = pkgs.python3Packages.buildPythonPackage rec {
pname = "libgen-api";
version = "0.3.0";
doCheck = false;
propagatedBuildInputs = with pkgs.python3Packages; [
setuptools
wheel
beautifulsoup4
];
src = (pkgs.python3Packages.fetchPypi {
inherit pname version;
sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
});};
in
pkgs.python3Packages.buildPythonPackage rec {
name = "getBook";
src = ./.;
propagatedBuildInputs = with pkgs.python3Packages; [
requests
libgenApi
];
};
};
}
Do I need to declare the Pypi package as an input or something?
Looks like you are using wrong hash (of toolz package), so that gets used instead.
If you change the hash to sha256 = "";, you will get complain:
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
builder for '/nix/store/7j90gbgzcj6cq3c8b43rlfrq6vqz7614-libgen-api-0.3.0.tar.gz.drv' failed with exit code 1; last 10 log lines:
curl: (22) The requested URL returned error: 404
trying https://pypi.io/packages/source/l/libgen-api/libgen-api-0.3.0.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122 100 122 0 0 1119 0 --:--:-- --:--:-- --:--:-- 1119
100 280 100 280 0 0 853 0 --:--:-- --:--:-- --:--:-- 1958
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
error: cannot download libgen-api-0.3.0.tar.gz from any mirror
That is because PyPI only has that package as wheel
Looking at
you need to pass it format = "wheel";.
I got it to work by using GitHub instead of PyPi, but now I’m getting another error:
{
description = "Just some python scripts to download books.";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, utils }: {
devShell = self.defaultPackage;
defaultPackage.x86_64-linux =
with import nixpkgs { system = "x86_64-linux"; };
let
libgenApi = pkgs.python3Packages.buildPythonPackage rec {
pname = "libgen-api";
version = "0.3.0";
doCheck = false;
propagatedBuildInputs = with pkgs.python3Packages; [
setuptools
wheel
beautifulsoup4
];
src = (pkgs.fetchFromGitHub {
owner = "harrison-broadbent";
repo = "libgen-api";
rev = "v0.3.0";
sha256 = "GwomZduHpF/vqT9h0kw9Df6bG0enGPWwDlvdKIS85Ww=";
});};
in
pkgs.python3Packages.buildPythonPackage rec {
pname = "getBook";
version = "0.1.0";
src = ./.;
propagatedBuildInputs = with pkgs.python3Packages; [
requests
libgenApi
];
};
};
}
returns:
error: --- Error --- nix-daemon
builder for '/nix/store/jcpgpifbmjvc8rr8f2dglflad9m534i8-python3.8-libgen-api-0.3.0.drv' failed with exit code 1; last 10 log lines:
warning: file source/setup.py may be generated; SOURCE_DATE_EPOCH may be non-deterministic
patching sources
configuring
no configure script, doing nothing
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
running build_py
error: package directory 'libgen_api' does not exist
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/x9ymy0xdx8afhl79plncliyc40kc106l-python3.8-getBook-0.1.0.drv' failed to build
Apparently, the project is pretty broken:
- It uses
APIin directory name butapiinsetup.py - It uses
bs4instead ofbeautifulsoup4
The following seems to make it build:
pkgs.python3Packages.buildPythonPackage rec {
pname = "libgen-api";
version = "0.3.0";
src = pkgs.fetchFromGitHub {
owner = "harrison-broadbent";
repo = "libgen-api";
rev = "v${version}";
sha256 = "GwomZduHpF/vqT9h0kw9Df6bG0enGPWwDlvdKIS85Ww=";
};
propagatedBuildInputs = with pkgs.python3Packages; [
lxml
requests
beautifulsoup4
];
postPatch = ''
mv libgen_{API,api}
substituteInPlace setup.py --replace "bs4" "beautifulsoup4"
'';
# There are no tests.
doCheck = false;
}
3 Likes