Maybe let's sort and/or categorize all-packages?

1

I’m just trying to add a new top-level package and don’t know where to add it in the humongous file pkgs/top-level/all-packages.nix. This made me ask myself why the entries in this file are neither sorted nor categorized. The only message I can find is the friendly but unhelpful comment at the beginning of the file:

/* The top-level package collection of nixpkgs.
 * It is sorted by categories corresponding to the folder names
 * in the /pkgs folder. Inside the categories packages are roughly
 * sorted by alphabet, but strict sorting has been long lost due
 * to merges. Please use the full-text search of your editor. ;)
 * Hint: ### starts category names.
 */

It is unhelpful because it is untrue. The packages jump wildly between folder structures. A snippet:

  aether = callPackage ../applications/networking/aether { };

  alda = callPackage ../development/interpreters/alda { };

  align = callPackage ../tools/text/align { };

  althttpd = callPackage ../servers/althttpd { };

  among-sus = callPackage ../games/among-sus { };

There is some vague alphabetic sorting going on, but it’s not kept throughout the file. I tried to search with Issues · NixOS/nixpkgs · GitHub, but didn’t find anything helpful.

I don’t understand why we don’t split this big file up. We have a great categorisation in form of the folder structure already. Why doesn’t every folder simply contain a file e.g. tools/text/all-packages.nix which contains all packages of that subcategory, and further up the tree we simply merge these sets?

Some advantages I can see:

  • Maintainers will immediately know where to put their packages
    • Great for newcomers
    • No need to scroll and search and ponder endlessly
  • Github will be able to display all-packages.nix
  • My guess: Merge conflicts are reduced, because adding new packages is split up on many files, sorted by topic
  • We could even add the category to the meta attribute in an orderly way, in the long run allowing people to filter on that information e.g. in the NixOS Search - Loading... search

Some disadvantages:

  • It’s a tiny bit harder to search for an existing package. Rebuttal: You probably have a text editor capable of searching for the string my-package =, and you might even know the category of your package.
  • Extra evaluation when merging the sets. Are there some reliable estimates how much that would be?
  • Might take some time to convert to this new structure. Rebuttal: Work can maybe be automatized in a simple way. In any case it can be split efficiently between categories.
6 Likes
2

Nix is a relatively slow language and I wonder if such ‘multi-file large attrset merge’ will have a noticeable impact :thinking:

As for the idea itself - sounds great but I personally care if it is a basic ‘callPackage’ call or overwrite (or some other additional logic).
So may I suggest splitting it into 2 or 3 files instead, one file is expected to contain ‘dumb’ package definitions and another one should contain everything else. (And maybe the whole stdenv stuff should be split into a third file?)

Not sure if anyone agrees with such separation, but these are my two cents :slight_smile:

3

Such a separation has been done in some places, typically when there is a big automatically generated list of “basic” packages, and a manually maintained list of overwrites. For example Haskell works like that in nixpkgs. Whether it is a good idea here I’m not sure, but anyways I think it’s orthogonal to what I’m proposing. Either idea can work without the other.

1 Like
4

How would one go about testing that?

5

You may mean broader process/methodology, but in case you mean lower-level I recall seeing an env for eval stats: nix/doc/manual/src/command-ref/env-common.md at 323e5450a1a6e4eb97ba1c9aeba195187cfaff37 · NixOS/nix · GitHub

I’m not otherwise familiar with any of this, so I’m not really sure if the stats it has are meaningful in this case. Here’s the underlying source nix/eval.cc at 6f46434f3226784e809158a04a8067036f9e6291 · NixOS/nix · GitHub. I don’t see many references in the wild, but:

2 Likes
6

Tbh I think just removing ###-sections and automatically re-sorting everything on each release would go a long way. It’d be an easy first step and does not leave space for bikeshedding

3 Likes
7

I guess by separating packages and comparing evaluation with NIX_SHOW_STATS output that is mentioned by @abathur