CI will soon enforce `pkgs/by-name` for new packages

Development Nixpkgs Architecture
1

Four months ago, the first milestone towards RFC 140 was achieved by introducing the pkgs/by-name directory for Nixpkgs, allowing new packages to be declared in that directory instead of the category-based file hierarchy, removing the need for all-packages.nix in most cases. Thanks to the efforts of all the contributors and reviewers, we’ve seen the number of packages in pkgs/by-name skyrocket to over 800!

Now’s time for the next step: Enforcing pkgs/by-name for all new packages (where applicable)! This is implemented in the following PR:

https://github.com/NixOS/nixpkgs/pull/275539

Unless significant problems are brought up, the PR is scheduled to have its final review and merge party on Monday, 2024-01-15T16:00:00Z in this Jitsi, anybody is free to join! :tada:

Edit: Now over! While we couldn’t merge during the meeting due to a test failure in CI, I was able to fix it fairly easily.

The PR is now merged, and will take effect once the nixos-unstable channel updates the next time!

Good to know

  • Only packages using pkgs.callPackage need to use pkgs/by-name, so this doesn’t apply for e.g. python3Packages.callPackage, callPackages (note the s!) or package aliases.
  • Whether pkgs/by-name should be used is automatically checked by CI, so there’s no need for reviewers to think about this: If CI is green, there’s no problems with pkgs/by-name and merging won’t break it.
  • This also applies to older PRs that still introduce new packages outside pkgs/by-name: They can still be merged as long as CI is green. CI will start failing if you push new code though, in which case it would have to be moved to pkgs/by-name.
  • There’s no need to migrate existing packages to pkgs/by-name. That will be done automatically as the next and final step of RFC 140. See also the guide on manual pkgs/by-name migration.

Sponsoring

Finally, a huge shout-out to mainly Tweag but also newly Antithesis for sponsoring this work! :sparkles:

51 Likes
2

Update: After merging the PR I discovered a problem, but there’s a fix ready:

https://github.com/NixOS/nixpkgs/pull/281390

It’s going to take a bit longer until it actually takes effect, but some very nice improvements came out of this, see

https://github.com/NixOS/nixpkgs/pull/281374

3 Likes
3

What is the plan for avoiding CI failures on unrelated packages due to packages not using pkgs/by-name? Apparently, staging checks for pkgs/by-name now, but new packages not using it were merged into that branch, which leads to CI failures: https://github.com/NixOS/nixpkgs/actions/runs/7600068348/job/20697863923?pr=282473

It seems as if these two could easily be fixed manually, but the documentation says we should wait for the automated migration, which is still a draft.

1 Like
4

@gm6k Oh that’s a very interesting case. See workflows/check-by-name: Trigger on base branch changes by infinisil · Pull Request #282707 · NixOS/nixpkgs · GitHub for a fix

2 Likes
5

Stupid question: what if a package is built with pkgs.libsForQt5.callPackage or qt5.callPackage… (which is, I expect, the case of all Qt packages?) Is there a way to turn it into a pkgs.callPackage entry, or should we just use the “legacy” way?

6

@tobiasBora Check out Documentation: Qt callPackage approach incompatible with pkgs/by-name · Issue #287015 · NixOS/nixpkgs · GitHub, looks like it can work with the standard pkgs.callPackage!

3 Likes
7

I love the pkgs/by-name change. I had special rules in my editor not to highlight all-packages.nix because of its size. It might be written somewhere (the RFC I guess) or enforced in the CI check but I prefer ask here anyay.
Should library functions and wrappers go into pkgs/by-name ? A few examples I have in mind: buildEnv and the neovim wrapper.

I have a bunch of changes to the wrapper I plan to upstream and I would have more peace of mind preemptively moving to pkgs/by-name if it makes sense to avoid future PRs conflicts.

8

pkgs/by-name is only for packages, specifically values for which lib.isDerivation is true, so buildEnv wouldn’t work, but the wrapped neovim would. And CI will complain when a new package should go to pkgs/by-name, but you can also run it locally ahead of time :slight_smile:

There’s also plans to support other package sets, which would be interesting to extend to builders too

1 Like