The topic of splitting NixOS into sub repositories has been on the table a lot of times and I it seems like every discussion has always led to the same situation. Either you are in favor of splitting it to allow more flexible management of sub groups of packages or you are against splitting because you fear that this will lead to more overhead to keep things stable. I believe we could discuss this topic for a few more years without coming to any change which can be agreed on.
But the following question hasnāt been discussed at all and is probably the core reason for all this discussion:
is git/github the right tool for nix, nixpkgs and nixos?
In my opinion the git workflow on github is what causes all these discussions. Git/Github limits access permissions in a very bad way. But git allows us to control all packages at once. From what I understand the whole discussion ādo we split into subrepos or do we notā would not exist if we were to use something else than git.
We see git/github as granted (especially because of all of its benefits!) but just take a look at e.g. debian which has a drastically different approach. They have source package files and deb files and mulitple package building tools in various complexities. Starting to contribute to Debian feels completely different. Therefore Debian doesnāt have the struggles we have with NixOS. (of course debian therefore lacks some of the benefits we can enjoy!).
What I want to say is that it doesnāt necessarily have to be a git repo on github. Iām probably the wrong person to make a list of pros and cons for either Git or Github, but I think that would be a good thing to do. The idea is not to ged rid of git or github. I honestly believe that it is a massively bigger blessing than it is a curse!! But making a list of pros & cons would show us what other tooling might be necessary besides github to provide the lacking features.
So to change the above kinda aggressive question into a more contributing one:
What are the weaknesses of our toolchain [git/github] and how can we fix/circumvent them?
Answering this question would probably solve a lot more problems than trying to just split the git repo into multiple sub-repositories without thinking of other solutions to the actual problem.
I genuinely donāt properly understand which of the current discussion topics leads to the conclusion that the technical solution should be a split.
we canāt give enough fine grained permissions on a āpackage levelā? ā Arenāt github bots a solution to this?
āone repo = one issue tracker = the same people have to deal with all the burdenā? But isnāt there a way to separate package ārelevanceā with a metadata field and/or with issue tags?
we want to be able to test some part of the code more in depth than other parts? ā Why should it not be possible to test only parts of the whole repo?
Is it because nixpkgs advances too slow? ā I assume this is caused by technically only having one master branch, ergo only one git chain which is not flexible? Could this be solved alternatively by e.g. having a more complex merge strategy with more branches = more chains = being able to e.g. delay failing packages while working packages could already be advanced?
The whole āmetaā discussion with a Federated Republic of Nixpkgs doesnāt seem like something that needs an actual technical change. I lost you down the road but it seems more like a discussion about where the community should go in a cultural way [which can then indirectly lead to different decisions on technical changes] .
Excellent observation! Discussing this would hopefully finally lead us down a constructive path and should be a different thread. @ryantm can you split this out, too?
Iām not sure whether Iām in a tiny minority, but the main blocker for me in learning to contribute to nixpkgs has been lack of motivation in learning github, partly because itās complicated and partly because it belongs entirely to Microsoft (end therefore can be embraced and extinguished fairly rapidly).
Nix doesnāt really have a technology issue, it has a people issue. Thereās just too few trusted people in this purely volunteer organization to handle the overhead of issues and PRs. Plus peopleās engagement with nixpkgs is subject to change as personal and professional factors change.
Itās been my opinion for over a year that the NixOS Foundation should have a ācommunity liasonā person which can focus on triaging issues, reviewing PRs, and other work which volunteers wouldnāt want to do in their free time.
Even if I finish GitHub - jonringer/basinix: (WIP) Nixpkgs pull request review website which should remove most of the āwhat can be automatedā from a nixpkgs review, you still want human eyes to review PRs from a compliance perspective. This still requires someoneās time.
I genuinely donāt properly understand which of the current discussion topics leads to the conclusion that the technical solution should be a split.
I take for granted that Iām probably massively misunderstood. I donāt mind. Weāll figure it out together.
The technical reason that best assimilate my purely social/organizational motivations is: interfaces. A split would require us to improve our interfaces (document, honor, stabilize, etc.). It would make a lot things harder, too.
The social dimension to interfaces is that they separate realms of concern, and preserve each realm with a certain autonomy, that they otherwise would not have. That in turn might boost diversity in solutions (and if only diversity in speed of decision making and implementation!), and form there on we can see āprofitā on the horizon, I guess.
But I also think, since our BDFL is pretty much absent, it is very hard to meaningfully steer the (entire) ship (at once).
Many of you have seen pijul, I suppose. I havenāt tried it, but Iām waiting for the moment Iāll be able to and see how it compares to git.
The disadvantage of github, because it seems ācompleteā, in my opinion is, that few people take the chance like @jonringer and attempt to implement a greenfield solution with our actual needs meaningfully staring at them upclose.
With some kafka (any message queue) at the heart of our workflows with a couple of pretty decent interfaces, I fantazise we would already have automated the shit out of it.
Framing this in āpros/consā is useful, because in any change we might make, we should not only consider what āconsā of the current system it alleviates, but also whether it breaks any of the āprosā of the current system.
In the āprosā list:
I love how a single nixpkgs commit hash describes an entire consistent collection of all packages and associated tools
I love how for designated commit hashes (those that are part of a channel), the packages that are part of that hash have a high likelihood of being available in the binary cache
I love how a contribution to a package whose maintainer is inactive is not stuck, but can be reviewed and merged by non-maintainers
I, too, suspect bots might enable great opportunities to reduce the burden on those with merge rights, either by automating work/checks or perhaps even allowing a wider range of maintainers to trigger merges of changes that meet certain criteria. Of course we already heavily use bots and it is a huge help. It is very non-obvious what work would be useful to automate, or what criteria should allow this, though. I think we should flesh that out in separate topics around particular ideas.
I agree the ābig heapā of issues and PRās is daunting, even with our already extensive use of tags/metadata. Having a way to filter/browse/subscribe to the issues/PRās I might be interested in as a āmore casualā contributor seems useful for me. Iāve started sketching something like that in https://git.sr.ht/~raboof/nixpkgs-contributor-dashboard
Define ātoo slowā - actually I think it already advances at an impressive speed . It would always be good to speed it up, though, of course.
My gut-feeling is more branches/chains would make things more complicated while itās not obvious it would help. I think āgreen hydraā strategies have been discussed in much more detail in other threads, but in summary I think there are 2 things to do:
prevent breakage: adding more validations to PRās, more tests, etc. There are of course resource constraints there
fix breakage when it occurs: this might actually be where much more opportunity for improvement is.
this is where āpackage tiersā might help: we could decide when ālower-tierā packages break, we just mark them broken and allow nixpkgs-unstable to advance. Itās of course still good to fix those packages, but it can be done āin parallelā by those who care about them without interfering with the main project
also I think we have an opportunity for better tools for collaborating on failures: right now, when you see nixpkgs-unstable is not advancing, itās not super easy to find out why itās failing, who might already be working on this, what they found so far, etc. Introducing tools to find that information more easily might make it easier for people to contribute to fixing the breakage.
Anyway, itās much easier to dream about what could be improved than to actually work on improving it, and itās hard to say if itās really that productive, but itās hard to resistā¦ I hope you excuse my indulgence .
Iāll just chime in to say that for me personally the fact that NixOS/nixpkgs is on github was actually the deciding factor that led me to contribute to it and use it in the first place. I would never dream of trying to contribute to a project like debian or ubuntu simply because I donāt even know what tools they use or where to begin. But Iām very comfortable with git and GitHub, and this made nix far, far more approachable.
It also just seems far more modern IMHO. Whether you love it or hate it, the OSS community lives and breathes git and GitHub. And so it āfeels rightā that a progressive, up-and-coming OSS distro should operate on those tools.
I know OP isnāt actually suggesting migrating away from git/github but just wanted to throw that out there.
Exactly this. Iām not a major contributor or anything, but even though Iāve been using Linux for a long time (mostly Arch), until Nix I never contributed anything. I have zero desire to figure out how to send patches to a mailing list or something, and itās hard to even begin figuring out where to look for bugs, fixes, how to create new packages, how to apply fixes to broken packages, etc. for most Linux distros.
With Nix, when something is broken (which to be fair seems to happen more frequently than many other distros, at least on unstable) or doesnāt work how I want it to, thanks to Git and already being familiar with it I can fork nixpkgs, fix it myself, put my own fix into an overlay to immediately apply/use the fix, then easily submit a PR to nixpkgs to share the fix. Or I can find a last working point/hash and pin the system to that.
If for some crazy reason Microsoft decides to go evil with GitHub, I think GitLab would still enable basically the same functionality, ease, and transparency. Until then though I actually think nixpkgs being on GitHub is a huge win.
. It would always be good to speed it up, though, of course.