How to search nixpkgs by e.g. `buildInputs`?

Help
1

tldr: does anyone have a favorite method for finding example packages by criteria such as “packages depending on pkg-config”, or answering questions like “do most packages put cmake in buildInputs or nativeBuildInputs”?

I’m continuing to grind at building cadquery for aarch64-darwin, and I continue to get errors about clang finding c++ headers (another related issue, I think: building package with bazel and clang fails due to C++ headers not found · Issue #150655 · NixOS/nixpkgs · GitHub). I assume I’m just doing things wrong, so I keep wanting to reference other packages in nixpkgs and see how they are using x dependecy or y tool.

I keep thinking it would be really great to have a “quick and dirty” approach to finding example packages by these criteria. It seems like something that nix eval / nix repl might be able to do “correctly” (e.g. accounting for imports and other nix logic), though I imagine this could also take a prohibitively long time for evaluation. In contrast it seems like something that a perl multiline regex might be able to do easily, quickly, and incorrectly.

I figured it was at least worth reaching out to see if anyone has any recommended techniques. If you had a local checkout of nixpkgs and wanted to answer the following types of questions, how would you do so?

  • do more packages put cmake in nativeBuildInputs or buildInputs?
  • how can I find an example package that builds on aarch64-darwin and uses libcxxStdenv?
  • how can I find an example python package that uses clang’s python bindings?

Note that I don’t necessarily need answers to these exact questions, I’m just trying to find a good solution to search / filter through nixpkgs based on criteria such as a package’s dependencies / attributes / metadata.

TIA for any recommendations!

2

CMake needs to run during the build, so it should be put in nativeBuildInputs.

The Darwin stdenv uses libc++ by default, so you should be able to use default stdenv on Darwin. If you need it anyway (e.g., to use libc++ on Linux), you can use libcxxStdenv where you would normally use stdenv. For example, the FoundationDB derivation uses libcxxStdenv.

As far as I can tell, it’s not packaged yet. It might be possible to fetch it with fetchPypi and use buildPythonPackage, but I expect it’ll require some work to get it to build.

1 Like
3

Oh, sorry. I should have responded to this. :sweat_smile:

For stuff like buildInputs and nativeBuildInputs, I usually consult the manual (and the cross-compilation section in particular). Most of the time, the following rule of thumb is sufficient: if it needs to run during the build, it goes in nativeBuildInputs and if it runs on the target, it goes in buildInputs. The other stuff really should only come up when dealing with building compilers or other, uncommon situations.

For the other questions, I grepped my local nixpkgs tree to see what other packages are doing. I did the same for libclang, which turned up a comment in the Tensorflow package that it’s not packaged yet. After that, it’s looking at how Python modules are packaged to determine what it would take to use e.g., libclang from PyPI.

4

Thanks for your thoughts!

Yes, I was reading about this, which is what prompted me to search and confirm that it was the case. My approach was:

$ rg \
    --no-filename \
    --no-line-number \
    --multiline-dotall '.*\b((native)?[bB]uildInputs) = \[.* cmake.*\];' \
    --replace '$1' |
    sort |
    uniq -c
      1 buildInputs
   1551 nativeBuildInputs

I’m running this on a release-23.05 checkout; the 1 exception above was fixed here: xplorer: move cmake to nativeBuildInputs · NixOS/nixpkgs@c8929e4 · GitHub

Interesting – I still haven’t figured out this issue, but I’ve never done C or C++ development, so I’m not surprised it’s taking me a while.

I have it building: https://github.com/n8henrie/nix-cadquery/blob/3adc4c1590b46a00b57d8826548184c7789199c8/flake.nix#L22 – as mentioned, these were just examples of the types of questions I was hoping to answer by searching nixpkgs.

So it sounds like your general approach is “grep nixpkgs and manually review / refine results.” Is that a fair characterization?

5

Pretty much. Sometimes I’ll also search the nixpkgs issues on GitHub to see if someone else has experienced a similar issue.

Adding packages to buildInputs should add the appropriate include and library search paths automatically to the compilation commands, so setting up the include paths should be mostly unnecessary. Note that it’s also not usually necessary to add clang (or GCC) to your nativeBuildInputs. The stdenv takes care of making sure you have a compiler, and that it’s properly wrapped.

Also, if you do need to provide paths, you can use ${} to insert the result of a Nix expression into a string. For example, instead of using "$(nix path-info github:nixos/nixpkgs#llvmPackages_14.libcxx.dev)/include/c++/v1", you can directly reference the package as "${lib.getDev llvmPackages_14}/include/c++/v1" (assuming that llvmPackages_14 is in scope).

6

I explored writing something in nix for this purpose. Unfortunately evaluation failures make it kind of clumsy to write and ultimately I was unable to catch specific errors making it fail when I try to run across all of nixpkgs.

# filterByInput.nix
{
  depName,
  attr ? "propagatedBuildInputs",
}: let
  pkgs = import <nixpkgs> {};

  hasName = pkg: (builtins.tryEval pkg).success && (pkg ? pname || pkg ? name);
  hasDep = pkg: pkg ? "${attr}" && builtins.any (dep: hasName dep && (builtins.match (depName + ".*") (dep.pname or dep.name)) != null) pkg.${attr};

  pkgList = pkgs.lib.take 20 (builtins.attrValues pkgs);
  # pkgList = builtins.attrValues pkgs;
in {
  triedOn = builtins.map (pkg: (builtins.tryEval (pkg.pname or pkg.name or "<unnamed>")).value) pkgList;
  result = builtins.map (pkg: pkg.pname) ((builtins.filter (pkg: hasName pkg && hasDep pkg)) pkgList);
}

Obviously could use some refactoring and simplification, but it seems to work:

$ nix eval --json -f filterByInput.nix --argstr depName python --argstr attr propagatedBuildInputs result
["fabric"]
$ nix eval --json -f filterByInput.nix --argstr depName cmake --argstr attr nativeBuildInputs result
["AusweisApp2","CHOWTapeModel","ChowCentaur","ChowKick","ChowPhaser","EBTKS","empty-epsilon","LAStools","LASzip","LASzip"]

I’m not sure how to get rid of the clumsy “result” requirement for the nix eval call ($ nix eval ... result) since the -f makes it expect an attrset (and an attr to get from that attrset).

Unfortunately expanding pkgList to all of nixpkgs results in an error, and I can’t seem to tryEval my way around it:

$ nix eval --json -f filterByInput.nix --argstr depName cmake --argstr attr nativeBuildInputs result
error: anonymous function at /nix/store/hqyzlrq2kpp6w9cgvaybgx8swmc3p6za-source/pkgs/development/compilers/openjdk/darwin/19.nix:1:1 called with unexpected argument 'enableJavaFX'

       at /nix/store/hqyzlrq2kpp6w9cgvaybgx8swmc3p6za-source/lib/customisation.nix:80:16:

           79|     let
           80|       result = f origArgs;
             |                ^
           81|

I’m sure the easy way would be to just write something in bash that deals with the exceptions.
filterAttrs would probably also be an improvement.