Where is `callPackage` defined exactly? (part 2)

I had this Stackoverflow question on how to find lambda definitions in general (saved), because I wanted to see how callPackage is implemented, and found the place alright,

$ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> pkgs = import <nixpkgs> {}

nix-repl> pkgs.callPackage           
«lambda @ /nix/store/.../nixos/lib/customisation.nix:117:31»

but lib/customization.nix:117 shows callPackagesWith:

  /* Call the package function in the file `fn' with the required
    arguments automatically.  The function is called with the
    arguments `args', but any missing arguments are obtained from
    `autoArgs'.  This function is intended to be partially
    parameterised, e.g.,

      callPackage = callPackageWith pkgs;
      pkgs = {
        libfoo = callPackage ./foo.nix { };
        libbar = callPackage ./bar.nix { };
      };

    If the `libbar' function expects an argument named `libfoo', it is
    automatically passed as an argument.  Overrides or missing
    arguments can be supplied in `args', e.g.

      libbar = callPackage ./bar.nix {
        libfoo = null;
        enableX11 = true;
      };
  */
  callPackageWith = autoArgs: fn: args:
    let
      f = if lib.isFunction fn then fn else import fn;
      auto = builtins.intersectAttrs (lib.functionArgs f) autoArgs;
    in makeOverridable f (auto // args);

I presume that lib.callPackage is defined somewhere by calling callPackageWith with all-packages.nix (as the first, autoArgs, argument) but I just can’t seem to find it. (This assumption may not even be correct.)

3 Likes

I’m not sure if there’s a trivial way to look something like this up directly, but the blame for the lines you linked led me back to * Add callPackage etc. · NixOS/nixpkgs@fd268b4 · GitHub

I looked for a newScope definition like this in all-packages but didn’t find one.

I cast a wider net and found it here:

Sure enough, there’s still a callPackage def above it, so I suspect that’s the one you’re looking for.

The definition was moved there in top-level: Introduce `buildPackages` for resolving build-time deps · NixOS/nixpkgs@bf17d6d · GitHub

1 Like

It’s defined in pkgs/top-level/splice.nix as callPackage = pkgs.newScope {};. pkgs in that file comes from pkgs/top-level/stage.nix where we have
splice = self: super: import ./splice.nix lib self (adjacentPackages != null);

So pkgs is actually self. Which makes sense because newScope is also defined in splice.nix as
newScope = extra: lib.callPackageWith (splicedPackagesWithXorg // extra);
And there we have it; we found callPackageWith!

Whew, the internals of nixpkgs make my head spin.

TL;DR look in splice.nix where
callPackage == newScope {} == lib.callPackageWith splicedPackagesWithXorg

4 Likes

Oh, @abathur beat me to it =)

1 Like

Thank you so much to both of you! Not sure how long it would’ve taken me to find this - and as I have never encountered newScope before, it means that will have to go down deeper in the rabbit hole.

You can say that again… How long did it take for you to figure this out? The couple of lines in your answer are pretty dense with info, especially now that I’ve looked at pkgs/top-level/splice.nix. I’m still stuck at

in

{
  inherit splicePackages;

  # We use `callPackage' to be able to omit function arguments that can be
  # obtained `pkgs` or `buildPackages` and their `xorg` package sets. Use
  # `newScope' for sets of packages in `pkgs' (see e.g. `gnome' below).
  callPackage = pkgs.newScope {};

  callPackages = lib.callPackagesWith splicedPackagesWithXorg;

  newScope = extra: lib.callPackageWith (splicedPackagesWithXorg // extra);

  # Haskell package sets need this because they reimplement their own
  # `newScope`.
  __splicedPackages = splicedPackages // { recurseForDerivations = false; };
}

because it seems as if the callPackage definition calls the newScope one below. (And couldn’t even figure out where stage.nix comes into the picture.)

Fun times ahead:)

It took me like 10 minutes? The trick was many calls to git grep. callPackage is used too many times for it to be a useful search. I don’t remember if callPackageWith or callPackagesWith is what led me in the right direction, but once I found splice.nix my next step was "okay, where does pkgs come from here? It’s passed in at the top of the file, okay, git grep splice.nix, which led me to stage.nix, where I saw the self: thing. That still doesn’t quite explain the whole picture, because that’s just a function, that in turn gets thrown into toFix, which I still don’t understand what that’s doing. But I did a quick git grep "newScope =", and the only place it could have possibly come from is splice.nix.

I’ve also been spending far too much time poking around nixpkgs recently for my own projects.

I like @abathur’s git blame strategy too. I’ll have to keep that in mind for next time

1 Like

Oh, yeah, so in answer to your question:

callPackage = pkgs.newScope {};

where does pkgs come from? It’s passed in at the top of the file. Where does the file get imported? In stage.nix

splice then gets used here:

Which I’m assuming works kind of the same as overlays, but that’s kind of where my search stopped.

2 Likes

Just add " =" to speed up a search, I use a lot to find definitions: git grep "callPackage =" , git grep "newScope ="

3 Likes

Adding @jeff-hykin’s great answer from Stackoverflow:

Liked this part especially:

The reason the path in the lambda doesn’t return that^ position is because nixpkgs does some absolutely insane recurisve computation. My understanding is that pkgs is replaced over and over (bootstrapped), and the attributes on pkgs (like callPackages ) are replaced multiple times.

How could this be documented properly and in a way that would make possible to grasp in finite time?@jeff-hykin mentions fixed point combinators in his answer, and I think Nix Pills #17 does a great job of explaining the basics, but still can’t wrap my head around it; I understand it on an intuitive level, but still have no solid mental model even after years of working with Nix.

Love this article about visualizing JavaScript’s event loop so maybe something along that line? (It would probably also work to answer questions such as How are Nix packages built? (Discord).)

1 Like

I’ve been digging into this question a lot the last few days.

How could this be documented properly and in a way that would make possible to grasp in finite time?

I don’t know but I’ll give it a shot below.

fetchurl is one of the really recursive things, so picking it apart has been helpful. There’s a fetchurl/default.nix but there’s actually other code for it too. And fair warning, my statements below are probably slightly inaccurate, so take “it first happens” and the like as “I’m pretty sure it first happens”

What I’ve found most helpful for the thought process is

  1. lib is pure-nix code, isolated. So mentally put that in a box.
  2. Theres a machine-specific boot-stage sequence that happens first. Basically all the impure stuff like stdenv gets created through that process, and its pretty complicated. So mentally put that in a box and mostly ignore it.
  3. After machine-specific bootstrapping we get stdenv, stdenvNoCC, and buildPackages which I think should be called foundationalPackagesYouCanTakeForGranted
  4. Then all-packages.nix is evaluated

Inside of the machine-bootstrapping the fetchurl/default.nix is loaded for the very first time, with absolutely barebone arguments. It creates the stdenv.fetchurlBoot that everyone can use. (I think its possible buildPackages.fetchurl is equivlent to stdenv.fetchurlBoot but I’m not sure)

But inside of all-packages.nix we see a fetchurl = ....
It’s what I would call a big-ole-spagetti-meatball of an expression (below), but

  # `fetchurl' downloads a file from the network.
  fetchurl = if stdenv.buildPlatform != stdenv.hostPlatform
    then buildPackages.fetchurl # No need to do special overrides twice,
    else makeOverridable (import ../build-support/fetchurl) {
      inherit lib stdenvNoCC buildPackages;
      inherit cacert;
      curl = buildPackages.curlMinimal.override (old: rec {
        # break dependency cycles
        fetchurl = stdenv.fetchurlBoot;
        zlib = buildPackages.zlib.override { fetchurl = stdenv.fetchurlBoot; };
        pkg-config = buildPackages.pkg-config.override (old: {
          pkg-config = old.pkg-config.override {
            fetchurl = stdenv.fetchurlBoot;
          };
        });
        perl = buildPackages.perl.override { fetchurl = stdenv.fetchurlBoot; };
        openssl = buildPackages.openssl.override {
          fetchurl = stdenv.fetchurlBoot;
          buildPackages = {
            coreutils = buildPackages.coreutils.override {
              fetchurl = stdenv.fetchurlBoot;
              inherit perl;
              xz = buildPackages.xz.override { fetchurl = stdenv.fetchurlBoot; };
              gmp = null;
              aclSupport = false;
              attrSupport = false;
            };
            inherit perl;
          };
          inherit perl;
        };
        libssh2 = buildPackages.libssh2.override {
          fetchurl = stdenv.fetchurlBoot;
          inherit zlib openssl;
        };
        # On darwin, libkrb5 needs bootstrap_cmds which would require
        # converting many packages to fetchurl_boot to avoid evaluation cycles.
        # So turn gssSupport off there, and on Windows.
        # On other platforms, keep the previous value.
        gssSupport =
          if stdenv.isDarwin || stdenv.hostPlatform.isWindows
            then false
            else old.gssSupport or true; # `? true` is the default
        libkrb5 = buildPackages.libkrb5.override {
          fetchurl = stdenv.fetchurlBoot;
          inherit pkg-config perl openssl;
          keyutils = buildPackages.keyutils.override { fetchurl = stdenv.fetchurlBoot; };
        };
        nghttp2 = buildPackages.nghttp2.override {
          fetchurl = stdenv.fetchurlBoot;
          inherit pkg-config;
          enableApp = false; # curl just needs libnghttp2
          enableTests = false; # avoids bringing `cunit` and `tzdata` into scope
        };
      });
    };

We can flatten it out into something sensible, like this

let
    #
    # note: stdenv, buildPackages, and lib are the only things not-defined below
    #
   
    # not the full perl, but enough args for a minimal verison
    bootstrap_perl = buildPackages.perl.override {
        fetchurl = stdenv.fetchurlBoot; 
    };
    
    # not the full xz, but enough args for a minimal verison (... etc)
    bootstrap_xz = buildPackages.xz.override {
        fetchurl = stdenv.fetchurlBoot; 
    };
    
    bootstrap_coreutils = buildPackages.coreutils.override {
        fetchurl    = stdenv.fetchurlBoot;
        perl        = bootstrap_perl;
        xz          = bootstrap_xz;
        gmp         = null;
        aclSupport  = false;
        attrSupport = false;
    };

    bootstrap_openssl = buildPackages.openssl.override {
        fetchurl = stdenv.fetchurlBoot;
        perl     = bootstrap_perl;
        buildPackages = {
            perl      = bootstrap_perl;
            coreutils = bootstrap_coreutils;
        };
    };
    
    bootstrap_keyutils = buildPackages.keyutils.override {
        fetchurl = stdenv.fetchurlBoot;
    };
    
    bootstrap_zlib = buildPackages.zlib.override {
        fetchurl = stdenv.fetchurlBoot; 
    };
    
    bootstrap_pkg-config = buildPackages.pkg-config.override (old: {
        pkg-config = old.pkg-config.override {
            fetchurl = stdenv.fetchurlBoot;
        };
    });
    
    bootstrap_libssh2 = buildPackages.libssh2.override {
        fetchurl = stdenv.fetchurlBoot;
        zlib     = bootstrap_zlib;
        openssl  = bootstrap_openssl;
    };
    
    bootstrap_libkrb5 = buildPackages.libkrb5.override {
        fetchurl    = stdenv.fetchurlBoot;
        pkg-config  = bootstrap_pkg-config;
        perl        = bootstrap_perl;
        openssl     = bootstrap_openssl;
        keyutils    = bootstrap_keyutils;
    };
    
    bootstrap_nghttp2 = buildPackages.nghttp2.override {
        fetchurl    = stdenv.fetchurlBoot;
        pkg-config  = bootstrap_pkg-config;
        enableApp   = false; # curl just needs libnghttp2
        enableTests = false; # avoids bringing `cunit` and `tzdata` into scope
    };
    
    bootstrap_curl = buildPackages.curlMinimal.override (old: rec {
        # break dependency cycles # <- original comment, not mine --Jeff
        fetchurl   = stdenv.fetchurlBoot;
        pkg-config = bootstrap_pkg-config;
        zlib       = bootstrap_zlib;
        perl       = bootstrap_perl;
        openssl    = bootstrap_openssl;
        libssh2    = bootstrap_libssh2;
        libkrb5    = bootstrap_libkrb5;
        nghttp2    = bootstrap_nghttp2;
        gssSupport =
            if stdenv.isDarwin || stdenv.hostPlatform.isWindows
                then false
                else old.gssSupport or true; # `? true` is the default
    });
    
    #
    #
    #  finally, using only buildPackages, stdenv, lib, 
    #  and the things above, we can make fetchurl
    #
    #
    fetchurl = 
        if stdenv.buildPlatform != stdenv.hostPlatform then
            buildPackages.fetchurl # No need to do special overrides twice,
        else 
            (import ../build-support/fetchurl) {
                lib            = lib;
                stdenvNoCC     = stdenvNoCC;
                buildPackages  = buildPackages;
                cacert         = buildPackages.cacert;
                curl           = bootstrap_curl;
            }
    ;
in
    fetchurl

That fetchurl^ at the very bottom, I would call fetchurl2.
stdenv.fetchurlBoot would be fetchurl1

However, and this is the fixed-point function bit, I did modifiy one thing; the cacert argument .
In the original expression its just cacert = cacert;, and I honestly don’t know if

  1. It is the full cacert package (which needs python3, and python3Packages, and all of their dependencies), or
  2. If cacert is actually equal to buildPackages.cacert at that moment

I think its actually possible for both of those to be true;

  1. we build fetchurl2 using buildPackages.cacert (because thats the cacert we have at the time)
  2. now that we have fetchurl2, we can build python3, and other stuff that needed fetchurl
  3. eventually we build the full/normal cacert using fetchurl2
  4. now that we have cacert2, we start building fetchurl3 with cacert=cacert2 as the input argument

(and so on, and so on, until fetchurl4 or whatever, going until they get arguments that never change; e.g. fixedpoint)
Even for just the cowsay package, I know perl has to go through +3 iterations like that.

There’s still a lot more to be discovered, and hopefully others will chime in. I’m a 3rd year PhD Computer Science student at a top university who loves recursion … and this stuff still crushes my brain. We really need to reduce the complexity/recursion used in Nixpkgs.

Maybe it’s too late for this question, but, how about just give it a click?

call-package

This is a feature released in Nixd 1.1.0 released

And you can even inspect more locations:

inspect-more

7 Likes