Monorepos don't map to our social structure

ashkitten · April 29, 2024, 9:06pm

I broadly agree with this. Splitting the repo up also introduces more opportunity for things to get out of sync, PRs in one repo may depend on PRs in a different repo entirely, independent maintainers struggle to keep up with nixpkgs, packages receive even less scrutiny than before, and it becomes impossible to keep an eye on everything happening in this complex interconnected web of dependencies.

Packages do not exist in a vacuum, and neither do we.

amjoseph · April 29, 2024, 10:40pm

FWIW the messages mentioning pijul in this topic should be split out into a separate topic. It’s kinda sad that discourse doesn’t have reply-to-thread-with-subject-line-change, which is the way of signaling a new subthread in email discussions. Sigh yet another 1980s innovation that seems to have been forgotten. Anyways the pijul discussion is not just orthogonal to the monorepo discussion, but sort of transcends it – there is no pijul submodule or josh-for-pijul because these things are unnecessary.

The joke goes: two git users are arguing, heatedly, about monorepo vs multirepo. A pijul user drifts by, saying

the-matrix-there-is-no-spoon-0

anon20203487 · May 1, 2024, 4:39am

What exactly is needed? To me, two years is not a huge amount of time.

waffle8946 · June 25, 2024, 6:44pm

I don’t know how the conversation ended up on pijul when the OP was about eliminating the board and monorepo… but anyway, I don’t think pijul is a serious option.

On a tmpfs:

$ time pijul clone https://nest.pijul.com/pijul/pijul
Repository created at /tmp/tmp.dSyrVH5tU1/pijul
Downloading changes  [==================================================] 1053/1053 [00:00:12]
Applying changes     [==================================================] 1053/1053 [00:00:16]                                                     Downloading changes  [==================================================] 1053/1053 [00:00:00]
Completing changes... done!                                                                                                                        
real	0m19.583s
user	0m8.949s
sys	0m0.618s

$ time pijul clone pijul pijul-copy
Repository created at /tmp/tmp.dSyrVH5tU1/pijul-copy
Downloading changes  [==================================================] 1053/1053 [00:00:03]
Applying changes     [==================================================] 1053/1053 [00:00:07]                                                     Downloading changes  [==================================================] 1053/1053 [00:00:00]
Completing changes... done!                                                                                                                        
real	0m8.737s
user	0m8.447s
sys	0m0.454s

Even a local, in-tmpfs clone took nearly 9 seconds for the clone of only 1k changes. And I think I have a quite decent CPU (Ryzen 5700X). Extrapolating this to the 640k commits that nixpkgs currently has, assuming (optimistically!) linear scaling, that’d be 1.48 hours for a local clone. Network speed and even disk I/O are huge variables, but in the end, will only make those times massively worse.

Meanwhile, with git:

$ time git clone file://$PWD/nixpkgs nixpkgs-copy
Cloning into 'nixpkgs-copy'...
remote: Enumerating objects: 4930209, done.
remote: Counting objects: 100% (4930209/4930209), done.
remote: Compressing objects: 100% (949117/949117), done.
Receiving objects: 100% (4930209/4930209), 877.36 MiB | 72.31 MiB/s, done.
remote: Total 4930209 (delta 3616600), reused 4928000 (delta 3614680), pack-reused 0 (from 0)
Resolving deltas: 100% (3616600/3616600), done.
Updating files: 100% (41804/41804), done.

real	2m40.512s
user	7m50.344s
sys	0m33.091s

About 33x faster. And even nowadays, we consider the barrier to contribution quite high, as people don’t know about blobless clones, or just don’t have the network bandwidth and stability to download gigabytes of data.

anon20203487 · June 26, 2024, 1:22pm

Well, it looks like a nice use case for improvements!

Infinisil · February 13, 2025, 9:33pm

I’ve just made a quick draft proposal to split lib into a separate repo!

github.com/NixOS/nixpkgs

Move `lib` to a separate repository

opened 09:31PM - 13 Feb 25 UTC

infinisil

9.needs: community feedback significant

I've previously suggested this [on Discourse](https://discourse.nixos.org/t/mono…repos-dont-map-to-our-social-structure/44162/12) (and again [here](https://discourse.nixos.org/t/any-interest-in-separating-out-nixpkgs-lib-into-its-own-repo-flake/39374/4)), which has garnered a surprising number of likes, so let's consider this! Here's a first draft proposal to discuss ## Motivation `lib` (note: not everything, see the plan below) is arguably the most independent part of Nixpkgs, it has effectively no dependency on anything other than itself (and Nix) and virtually no PRs touch both `lib` and non-`lib` (which is also in the [lib PR guidelines](https://github.com/NixOS/nixpkgs/tree/master/lib#pr-guidelines)). - The vast majority of Nixpkgs issues and PRs are unrelated to `lib`, which makes `lib` development hard to track. Yes there's [a label](https://github.com/NixOS/nixpkgs/pulls?q=%226.topic%3A+lib%22), but e.g. people can't just watch the entire repo. - Having a separate repo allows more granular repo permissions (though for the start let's keep the status quo) - A separate repo allows avoiding the dependency on the rest of Nixpkgs for those who don't need it. - Although this point is kind of moot, because https://github.com/nix-community/nixpkgs.lib exists, which could be made official as an alternative - With a separate repo, it's impossible for `lib` to depend on Nixpkgs, encourages stabilisation of interfaces between them. ## Plan - Figure out which parts of `lib` should stay in Nixpkgs, off the top of my head at least the systems stuff and maintainers (could also consider moving that to a different repo at some point, but that's orthogonal). Refactor Nixpkgs so that the `lib` directory is ready to be moved. - Set up CI to automatically mirror the new repo to Nixpkgs' `lib` directory (Nixpkgs is very self-contained right now, and while this could be changed, it's very tricky, maybe controversial and not necessary for now if we just mirror) - We need CI to do the syncing, I've done [some testing](https://github.com/Infinisil-s-Test-Organization/subtree-source) with automated `git subtree` mirroring before - We need CI in Nixpkgs to tell people to PR to the new repo instead - Give all Nixpkgs committers write access (we could also consider limiting this to more specific maintainers, but that's orthogonal and more controversial, so let's maintain the status quo for now) - Extract the `lib`-specific CI actions to the new repo, this should include at least part of the Nixpkgs manual build, which builds docs for `lib`, but also some others - Deprecate https://github.com/nix-community/nixpkgs.lib, or alternatively consider whether it can be reused as the source of truth. Could also look at that to get inspiration for the mirroring CI - Inform all the existing `lib` PRs that they should be moved to the new repo now, or do it automatically (probably not worth it, not many `lib` PRs are actively being worked on) --- Let's get some general consensus on whether we should do this. If we do agree, somebody will have to lead this effort, which might as well be me :) Ping @roberth @hsjobeki @adisbladis @Profpatsch

nyanbinary · February 14, 2025, 1:07am

So this is something similar to what Project Ekala but on a way smaller scale, if I understand correctly?

drupol · February 14, 2025, 7:34am

I would say that this is similar to any kind of projects growing.

ddoonniixx · February 14, 2025, 8:15am

You might want to have a look at at Auxolotl’s lib. The project is not very active at the moment, but they made some good progress on things.

Somewhere the initiator @jakehamilton made a video about it and I think the ideas were very good. It’s been a while since I watched it, but it was (to me) quite impressive.

jakehamilton · February 14, 2025, 2:02pm

Aux Lib hasn’t been worked on for a few months, but it is pretty much done. It won’t be changing much if at all and is already used by the binary bootstrap and package/module set. Should be solid reference for an extracted version of the Nixpkgs lib.