- The evaluation times are split. The CI doesn’t need to evaluate all of (what is presently) nixpkgs, but only the packages. I’m not sure how big the advantage is, but I’m hoping that it is considerable.
As fast as I know, eval-release.nix is slower than anything else except hydra-eval of all the packages (neither touch NixOS code anyway).
Well, short of actual package builds that also do not care about NixOS code…
- The complete test pipeline for
nixpkgscan run much faster: No VMs are started and the tests should be smaller (unit tests instead of integration tests).
You can already add a runCommand test to passthru.tests, there are some but not many.
Of course, a lot of large and annoying packages need X11 or Wayland (Xvfb works, of course); and often D-Bus to test the interesting and annoying to test parts…
So people start complaining about duplication of functionality like when [RFC] nixpkgs/tests: create nixos-independent PoC tests by 7c6f434c · Pull Request #36842 · NixOS/nixpkgs · GitHub was rejected. Splitting has additional synchronisation costs and does not guarantee that people will not still complain that some functionality you need for GUI tests are kind of (and not too usably because of the overhead, but they will not mention that) present in NixOS repo and you should not duplicate.
Additionally there is Automatic UID allocation by edolstra · Pull Request #3600 · NixOS/nix · GitHub which would allow running container NixOS tests inside a Nix build (without VMs) with much better overhead, there is a further PR linked there to update 3600 to the current state of Nix development. Apparently getting a way to have highly efficient tests for simple things is not a priority even after Eelco Dolstra has already himself figured out all the things that would need to be done.
Re-splitting will probably indeed happen to show off how nicely flakes work, make some cross-cutting changes annoying to synchronise, and also make CI for impact of package changes less understandble, and then a re-merge will occur (alternatively, we will split everything and mess up the CI so badly the current situation will appear perfect in comparison, cannot exclude that of course)