How can I write a Python package derivation in a flake?

I’m having difficulty packaging a Python script. Basically I wrote a python script that requires a couple of external dependencies, one in PyPi and one not. I just want to package it so I can use it on my own system, on the command line.

Here’s as far as I got in flake.nix:

{
  description = "Scripts for getting books.";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
  };
  outputs = { self, nixpkgs, ... }:

    nixpkgs.lib.buildPythonPackage rec {
      pname = "getBook";
      version = "0.1.0";

      libgenAPI = nixpkgs.lib.fetchPypi {
        pname = "libgen-api";
        sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
      };

      buildInputs = [ libgenAPI ];

      meta = with nixpkgs.lib; {
        description = "Just some python scripts for getting books.";
        license = licenses.gplv3;
        maintainers = with maintainers; [ JonathanReeve ];
      };
    };

}

What am I doing wrong here? And is there an easier way of doing this?

You can’t have the package directly returned from the outputs function, you need to wrap in package or defaultPackage attributes with correct children.

But hard to say more without knowing what error you get or how you tried to build.

Here’s what I have so far:

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }: (
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      (let
        libgenApi = stdenv.mkDerivation {
          name = "libgen-api";
          src = fetchFromGitHub {
            owner = "harrison-broadbent";
            repo = "libgen-api";
            sha = "1304a6f21df12e6ee843a9315faf4229908c3a00";
          };
        };
      in
        stdenv.mkDerivation {
        name = "getBook";
        src = ./.;
        system.python3.withPackages(ps: with ps; [ numpy libgenApi ])
      };
      );
}

The error message I’m getting is syntax error, unexpected '=', expecting ')'. And I’ve tried to put ) there instead, but it just doesn’t work.

I’m having difficulty even finding an example of how to do this anywhere. I feel like this shouldn’t be this hard.

This syntax doesn’t look right…

      let
        libgenApi = stdenv.mkDerivation {
          name = "libgen-api";
          src = fetchFromGitHub {
            owner = "harrison-broadbent";
            repo = "libgen-api";
            sha = "1304a6f21df12e6ee843a9315faf4229908c3a00";
          };
        };
      in
        stdenv.mkDerivation {
          name = "getBook";
          src = ./.;
          buildInputs = [ python3.withPackages(ps: with ps; [ numpy libgenApi ]) ];
        };
1 Like

That’s getting me “cannot coerce a function to a string.”

Here’s the flake now:

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils }: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      let
        libgenApi = stdenv.mkDerivation {
          name = "libgen-api";
          src = fetchFromGitHub {
            owner = "harrison-broadbent";
            repo = "libgen-api";
            sha = "1304a6f21df12e6ee843a9315faf4229908c3a00";
          };
        };
      in
        stdenv.mkDerivation {
          name = "getBook";
          src = ./.;
          buildInputs = [ python3.withPackages(ps: with ps; [ numpy libgenApi ]) ];
        };
  };
}

You need to parenthesize the input, otherwise python3.withPackages and the lambda are separate list elements.

@jtojnar, I don’t understand where you’re saying I should put parentheses. I’ve tried putting parentheses in a few different places, but I still get:

    21|         stdenv.mkDerivation ({
    22|           name = "getBook";
      |           ^
    23|           src = ./.;

cannot coerce a function to a string

Like this.

Though to avoid that problem I prefer:

  1. use let bindings
  2. do not use with
let myPython = python3.withPackages (ps: [ ps.numpy libgenApi ]); in
{
  # …

  buildinputs = [ myPython ];
}

Though that will probably not work for a python as you won’t have any runtime dependencies to your python with packages without doing some additional wrapping, similar will probably happen to the libgenApi.

Therefore you should prefer the python helpers like buildPythonPackage, buildPythonApplication or poetry2nix which do some of the nasty details for you.

Which module contains buildPythonPackage? It’s telling me it’s not in stdenv.

It is in the “top-level”, as most of the builders are.

Depending or your level of experience with Nix, anonymous functions, the tendency to call the latter lambdas, etc.; the above advice might range from spot-on-helpful to totally useless (or even worse, by raising more confusing questions than it answers).

So, in case it helps you, or anyone else in the future, here is an explanation of what it means.

Lambda

Firlstly, as you’re writing in Python, chances are you know what a lambda is in general, but you might not appreciate what they look like in Nix. The following are equivalent

lambda a: a + 1  # Python
       a: a + 1  # Nix

They are both anonymous functions with one parameter a which return a + 1 when they are called. The top one is in Python, the bottom one in Nix.

(In other languages these might look like

|a| a + 1
[](int a) {return a+1;}
\a -> a + 1
(lambda (a) (+ a 1))

etc., etc., etc. but they all mean the same thing (modulo polymorphism))

Hopefully this clears up why the advice talks about ‘the lambda’ while ‘lambda’ does not appear anywhere in the code.

Role of parentheses in Nix

The way you write withPackages(ps: ...) with the opening-parenthesis right up against the function, suggests that you maybe think that the parentheses play the role of the function call syntax, like they do in Python (and many other languages). Beware! this is not the case in Nix!

This is how to call a function fn with two arguments a and b, first in Python, then in Nix

fn(a, b)  # Python
fn a b    # Nix

I recommend that you change the style in which you write function calls in Nix, putting a space between the function and its argument, even if the argument is in parentheses. That should help you remember that the role of parentheses is to group things together rather than calling functions.

(A discussion of currying would probably be more confusing than helpful to the majority of the target audience at this point, so I’ll leave it out.)

Question

What is the meaning of

fn(a b)

in Nix?

Answer

  1. First, call the function a with the argument b
  2. then call the function fn with the result of the previous step

Moral of the story: the parentheses are there for explicit precedence, not as a part of the function call syntax.

To someone used to parsing C-family languages, this can be very misleading: you might have to work hard to suppress your natural parsing instincts when reading and writing Nix!

Make sure you understand these:

fn a b    # Call `fn` with two arguments
fn (a b)  # Call `fn` with one argument (the result of calling `a` with one argument)
fn a      # Call `fn` with one argument
fn(a)     # Exactly the same as the previous line!

Nix list syntax

Note that the Nix list syntax, unlike Python (and many other languages), does not use commas to separate elements. List elements are separated by whitespace (… or parentheses!). But function arguments are also separated by whitespace … so if you put a function call inside a list, who wins? Which syntax claims the spaces (… or parentheses)?

You might be surprised by the answer:

[1 2 3]    # 3 integers in a list
[fn a b]   # 3 elements in a list
[fn (a b)] # 2 elements in ...
[fn(a b)]  # 2 elements !!!!!! (Don't be fooled by your Python parsing habits!)
[fn a]     # 2 elements
[fn(a)]    # 2 elements !!!!!!
[(fn a b)] # 1 element (compare to `[fn(a b)]`, three lines up)
[(fn a)]   # 1 element, c.f. `[fn(a)]` two lines up

What’s this got to do with the original problem?

Armed with this knowledge, let’s look at your problem. When you wrote

buildInputs = [python3.withPackages(ps: with ps; [ numpy libgenApi ])]

Nix parsed it as

buildInputs = [
  python3.withPackages               # first element in the list
  (ps: with ps; [ numpy libgenApi ]) # second element in the list
]
  • element 1: named function python3.withPackages
  • element 2: anonymous function (lambda) ps: with ps; [ numpy libgenApi ]

So we should now be able to fully understand @jtojnar’s statement that

python3.withPackages and the lambda are separate list elements.

and, hopefully, now we also understand that we wrote something of the form [fn(a)] when we meant [(fn a)].

A bit more explicitly, we had something of the form

[fn(parameter: body)] # list with 2 elements: 1. named function, 2. lambda

where we need

[(fn (parameter: body))] # list with one element: the result of calling fn with a lambda as its sole argument

Conclusion

I hope that this serves to clear up a possible source of confusion in Nix, for someone.

15 Likes

Thanks for the very thorough explanation.

Ok here’s what I have so far:

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils, buildPythonPackage }: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      let
        libgenApi = buildPythonPackage {
          name = "libgen-api";
          src = fetchFromGitHub {
            owner = "harrison-broadbent";
            repo = "libgen-api";
            sha = "1304a6f21df12e6ee843a9315faf4229908c3a00";
          };
        };
      in
        buildPythonPackage {
          name = "getBook";
          src = ./.;
          buildInputs = [ (python3.withPackages (ps: with ps; [ numpy libgenApi ])) ];
        };
  };
}

But I’m still having trouble finding where buildPythonPackage is. I either get “undefined variable ‘buildPythonPackage’” or “cannot find flake ‘flake:buildPythonPackage’ in the flake registries”

pkgs.python3.pkgs.buildPythonPackage or pkgs.python3Packages.buildPythonPackage.

If you want to be more specific about the python version, then replace the 3s with 38s (or 37s or 39s).

For this sort of thing, it’s quite useful to use a Nix repl:

nix repl -I nixpkgs=fetchTarball "https://github.com/NixOS/nixpkgs/archive/15a64b2facc1b91f4361bdd101576e8886ef834b.tar.gz"

should give you a nix-repl> prompt, where you can use tab-completion to explore what is where. In this repl you will be inside pkgs already.

[REPL: Read-Eval-Print-Loop, aka interactive ‘shell’]

Ok great, so it’s not at the top level after all, but in pkgs.python3Packages.

Here’s what I have so far:

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils }: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      let
        libgenApi = pkgs.python3Packages.buildPythonPackage {
          name = "libgen-api";
          src = pkgs.python3Packages.fetchPypi "libgen-api";
        };
      in
        pkgs.python3Packages.buildPythonPackage rec {
          name = "getBook";
          src = ./.;
          buildInputs = [ (python3.withPackages (ps: with ps; [ numpy libgenApi ])) ];
        };
  };
}

but now I’m getting a bizarre error:

error: --- TypeError --------------------------------------------------------------------------- nix
at: (69:16) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix

    68|     let
    69|       result = f origArgs;
      |                ^
    70| 

value is a string while a set was expected
(use '--show-trace' to show detailed location information)

The full trace is:

jon@jon-laptop ~/C/getBook (master) [1]> nix build --show-trace
warning: Git tree '/home/jon/Code/getBook' is dirty
error: --- TypeError --------------------------------------------------------------------------- nix
at: (69:16) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix

    68|     let
    69|       result = f origArgs;
      |                ^
    70| 

value is a string while a set was expected
-------------------------------------------- show-trace --------------------------------------------
trace: while evaluating 'makeOverridable'
at: (67:24) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix

    66|   */
    67|   makeOverridable = f: origArgs:
      |                        ^
    68|     let

trace: from call site
at: (14:17) in file: /nix/store/izzqlvb6nyfy06pdpyq2yw3fd8fx44s5-source/flake.nix

    13|           name = "libgen-api";
    14|           src = pkgs.python3Packages.fetchPypi "libgen-api";
      |                 ^
    15|         };

trace: while evaluating the attribute 'src.name'
at: (14:11) in file: /nix/store/izzqlvb6nyfy06pdpyq2yw3fd8fx44s5-source/flake.nix

    13|           name = "libgen-api";
    14|           src = pkgs.python3Packages.fetchPypi "libgen-api";
      |           ^
    15|         };

trace: while evaluating 'hasSuffix'
at: (234:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/strings.nix

   233|     # Input string
   234|     content:
      |     ^
   235|     let

trace: from call site
at: (122:25) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix

   121|       pythonRemoveBinBytecodeHook
   122|     ] ++ lib.optionals (lib.hasSuffix "zip" (attrs.src.name or "")) [
      |                         ^
   123|       unzip

trace: while evaluating 'optionals'
at: (270:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/lists.nix

   269|     # List to return if condition is true
   270|     elems: if cond then elems else [];
      |     ^
   271| 

trace: from call site
at: (122:10) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix

   121|       pythonRemoveBinBytecodeHook
   122|     ] ++ lib.optionals (lib.hasSuffix "zip" (attrs.src.name or "")) [
      |          ^
   123|       unzip

trace: while evaluating 'chooseDevOutputs'
at: (493:22) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/attrsets.nix

   492|   /* Pick the outputs of packages to place in buildInputs */
   493|   chooseDevOutputs = drvs: builtins.map getDev drvs;
      |                      ^
   494| 

trace: from call site
trace: while evaluating the attribute 'nativeBuildInputs' of the derivation 'python3.8-libgen-api'
at: (111:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix

   110| 
   111|     name = namePrefix + name;
      |     ^
   112| 

trace: while evaluating the attribute 'outPath'
at: (164:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix

   163|       drvPath = assert condition; drv.drvPath;
   164|       outPath = assert condition; drv.outPath;
      |       ^
   165|     };

trace: while evaluating anonymous lambda
at: (645:24) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/lists.nix

   644|    */
   645|  unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
      |                        ^
   646| 

trace: from call site
at: (64:6) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/top-level/python-packages.nix               

    63|     modules = filter hasPythonModule drvs;
    64|   in unique ([python] ++ modules ++ concatLists (catAttrs "requiredPythonModules" modules));
      |      ^
    65| 

trace: while evaluating 'requiredPythonModules'
at: (62:27) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/top-level/python-packages.nix               

    61|   # Get list of required Python modules given a list of derivations.
    62|   requiredPythonModules = drvs: let
      |                           ^
    63|     modules = filter hasPythonModule drvs;

trace: from call site
at: (15:13) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/wrapper.nix 

    14|   env = let
    15|     paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
      |             ^
    16|     pythonPath = "${placeholder "out"}/${python.sitePackages}";

trace: while evaluating the attribute 'passAsFile' of the derivation 'python3-3.8.7-env'
at: (7:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/build-support/trivial-builders.nix          

     6|     stdenv.mkDerivation ({
     7|       name = lib.strings.sanitizeDerivationName name;
      |       ^
     8|       inherit buildCommand;

trace: while evaluating the attribute 'buildInputs' of the derivation 'python3.8-getBook'
at: (111:5) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/pkgs/development/interpreters/python/mk-python-derivation.nix

   110| 
   111|     name = namePrefix + name;
      |     ^
   112| 

trace: while evaluating the attribute 'drvPath'
at: (163:7) in file: /nix/store/yy0vp4kdpr31svz22rk7vhnj3bblghkg-source/lib/customisation.nix

   162|       outputUnspecified = true;
   163|       drvPath = assert condition; drv.drvPath;
      |       ^
   164|       outPath = assert condition; drv.outPath;


The error message says it all, even though the position is confusing:

fetchPypi requires attribute set as an argument:

pkgs.python3Packages.fetchPypi {
    pname = "libgen-api";
    version = "something";
    sha256 = "0000000000000000000000000000000000000000000000000000000000000000";
}

Another useful trick for finding where things are, is to clone (or fetch and unpack a tarball of) the nixpkgs repo. Then search that for occurrences of what you want, making sure to include a leading dot.

If you’re not already doing so, do yourself a favour and start using ripgrep. It’s in nixpkgs, so you can use Nix to install it. For instance (with a flake-enabled Nix[*])

nix shell nixpkgs#ripgrep -c rg -i '\.buildpythonpackage' path/to/your/nixpkgs/clone

shows me lots of valid, qualified, uses of buildPythonPackage, and it would do so even if I didn’t have ripgrep installed on my machine. Given that I do have ripgrep installed in my nix profile, I can omit the nix shell nixpkgs#ripgrep part, and get the same result, with just

rg -i '\.buildpythonpackage' path/to/your/nixpkgs/clone

[*] With a pre-flakes Nix, I might use

nix-shell -p --command "rg -i '\.buildpythonpack' path/to/your/nixpkgs/clone"

Edit:

I didn’t suggest searching directly in the GitHub web interface, because it gave me zero hits and made me question my sanity. This thread has both given me some peace of mind about my mental health, as well as pointing out another way of finding at you want, without the need to get a local copy of nixpkgs: https://search.nix.gsc.io/. Be sure to click on the “Advanced …” bar under the search box, and check the Ignore Case box (otherwise you’re bound to make some CamelCase error, and end up with zero hits), and then type \.builtpythonpackage in the search box.

1 Like

Hmm, I beg to differ. I think that there is plenty more than needs to be said, to make the required course of action clear to a Nix-non-expert.

1 Like

Good tips, thanks.

Now it builds, at long last. But when I enter the shell, it can’t find the Python modules I declared. (ModuleNotFoundError). What am I missing?

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils }: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      let
        libgenApi = pkgs.python3Packages.buildPythonPackage rec {
          pname = "libgen-api";
          version = "0.3.0";
          doCheck = false;
          propagatedBuildInputs = with pkgs.python3Packages; [
            setuptools
            wheel
	          beautifulsoup4
	        ];
          src = (pkgs.python3Packages.fetchPypi {
            inherit pname version;
            sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
        });};
      in
        pkgs.python3Packages.buildPythonPackage rec {
          name = "getBook";
          src = ./.;
          propagatedBuildInputs = [ (python3.withPackages (ps: with ps; [ requests libgenApi ])) ];
        };
  };
}

If you are using nix shell, that puts you into a shell with your defaultPackage on PATH. If you want to develop with the package’s dependencies, you need nix develop. For that, you will also need a devShell output. devShell = self.defaultPackage; should probably be enough.

Hm, that doesn’t seem to work, I don’t think. Unless it’s just that I’m not putting that line in the right place? Here’s what I have so far:

{
  description = "Just some python scripts to download books.";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils }: {
    devShell = self.defaultPackage;
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };
      let
        libgenApi = pkgs.python3Packages.buildPythonPackage rec {
          pname = "libgen-api";
          version = "0.3.0";
          doCheck = false;
          propagatedBuildInputs = with pkgs.python3Packages; [
            setuptools
            wheel
	          beautifulsoup4
	        ];
          src = (pkgs.python3Packages.fetchPypi {
            inherit pname version;
            sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560";
        });};
      in
        pkgs.python3Packages.buildPythonPackage rec {
          name = "getBook";
          src = ./.;
          propagatedBuildInputs = [ (python3.withPackages (ps: with ps; [ requests libgenApi ])) ];
        };
  };
}

And here’s what happens when I try to run it inside nix develop or nix shell:

[jon@jon-laptop:~/Code/getBook]$ python getBook.py 
Traceback (most recent call last):
  File "getBook.py", line 10, in <module>
    from libgen_api import LibgenSearch
ModuleNotFoundError: No module named 'libgen_api'