Query all pnames in Nixpkgs with flakes

Hi there,

I’m looking to query all the pnames (not attr names or names [0], as nix-env uses pnames) available in Nixpkgs.

While I could use:

nix-env --query --available

This is very slow (~30 seconds).

I’m hoping to benefit from flake hermetic evaluation allowing nix expression caching.

I’ve tried:

nix flake show nixpkgs --legacy --json

But there seems to be no way to limit this to the current system, and it only displays attr names and names [0].

I’ve also tried:

nix eval nixpkgs\#pkgs --apply "pkgs: pkgs.lib.attrsets.mapAttrsToList (n: v: v.pname ) pkgs"

But this fails to evaluate v.pname as I believe because v is a derivation.

The reason for doing this is adding support for opam, an OCaml package manager, to use nix to install ‘external dependencies’. For context, see here: Add support for nixos depexts by gridbugs · Pull Request #5332 · ocaml/opam · GitHub

[0] Clarification on Package Names and Versions

Using nix-env’s --attr/-A option would allow use to use attr names to install packages.

IIRC flake evaluation caching doesn’t work for nested package sets (as nixpkgs is) and thus is not disabled for anything that is in legacyPackages (normal packages has to be a flat attribute set).

On a more general note, I don’t think that plan for opam will be very fruitful – installing has a very different sense on NixOS to other linux distributions. Specifically depext for libraries won’t work, as NixOS doesn’t have something akin to /usr/lib where linkers will look for libraries, so not all depext will work as expected – or rather only depext that just want to make an executable available may work. (I seem to remember nixos depext where added as a way for opam2nix to track some needed metadata in the opam repository.)

Overall the use of nix-env has some pitfalls and there is a risk of messing up an users environment when invoking it in this way. If I were to add a Nix integration to opam, I’d rather have it set up a switch specific nix-shell environment and execute build commands within that environment (which of course would be a very different approach to other depext integration, but I believe necessarily so).

Thanks for your reply @sternenseemann.

IIRC flake evaluation caching doesn’t work for nested package sets (as nixpkgs is) and thus is not disabled for anything that is in legacyPackages (normal packages has to be a flat attribute set).

It does seem like this is being cached, as the two flake commands are slow in the initial run and very fast in subsequent runs. Maybe this is because we’re only looking at the attr names. Do you know if this is documented anywhere?

On a more general note, I don’t think that plan for opam will be very fruitful – installing has a very different sense on NixOS to other linux distributions. Specifically depext for libraries won’t work, as NixOS doesn’t have something akin to /usr/lib where linkers will look for libraries, so not all depext will work as expected – or rather only depext that just want to make an executable available may work.

This is a very good point. My preference would be for creating a shell.nix (or flake.nix with a devShell) containing the depexts, as described here, which I believe should set appropriate flags for linking.

(I seem to remember nixos depext where added as a way for opam2nix to track some needed metadata in the opam repository.)

Interesting! I didn’t know this. I’ve had better luck with tweag/opam-nix for building OCaml projects in a nix derivation.

Overall the use of nix-env has some pitfalls and there is a risk of messing up an users environment when invoking it in this way. If I were to add a Nix integration to opam, I’d rather have it set up a switch specific nix-shell environment and execute build commands within that environment (which of course would be a very different approach to other depext integration, but I believe necessarily so).

Yes this makes sense, I think is the same conclusion we were coming to independently.

Previous art are the Nix integrations stack and cabal-install have. The latter one is a bit simplistic (just reusing a shell.nix manually written by the user), but they work similarly: build comands are invoked in a nix-shell.

Seems like you are right. Maybe this was only initially true or I misremembered.