Is git/GitHub the appropriate tool for us?

The topic of splitting NixOS into sub repositories has been on the table a lot of times and I it seems like every discussion has always led to the same situation. Either you are in favor of splitting it to allow more flexible management of sub groups of packages or you are against splitting because you fear that this will lead to more overhead to keep things stable. I believe we could discuss this topic for a few more years without coming to any change which can be agreed on.

But the following question hasn’t been discussed at all and is probably the core reason for all this discussion:

is git/github the right tool for nix, nixpkgs and nixos?

In my opinion the git workflow on github is what causes all these discussions. Git/Github limits access permissions in a very bad way. But git allows us to control all packages at once. From what I understand the whole discussion “do we split into subrepos or do we not” would not exist if we were to use something else than git.

We see git/github as granted (especially because of all of its benefits!) but just take a look at e.g. debian which has a drastically different approach. They have source package files and deb files and mulitple package building tools in various complexities. Starting to contribute to Debian feels completely different. Therefore Debian doesn’t have the struggles we have with NixOS. (of course debian therefore lacks some of the benefits we can enjoy!).

What I want to say is that it doesn’t necessarily have to be a git repo on github. I’m probably the wrong person to make a list of pros and cons for either Git or Github, but I think that would be a good thing to do. The idea is not to ged rid of git or github. I honestly believe that it is a massively bigger blessing than it is a curse!! But making a list of pros & cons would show us what other tooling might be necessary besides github to provide the lacking features.

So to change the above kinda aggressive question into a more contributing one:

What are the weaknesses of our toolchain [git/github] and how can we fix/circumvent them?

Answering this question would probably solve a lot more problems than trying to just split the git repo into multiple sub-repositories without thinking of other solutions to the actual problem.

I genuinely don’t properly understand which of the current discussion topics leads to the conclusion that the technical solution should be a split.

  • we can’t give enough fine grained permissions on a “package level”? → Aren’t github bots a solution to this?
  • “one repo = one issue tracker = the same people have to deal with all the burden”? But isn’t there a way to separate package “relevance” with a metadata field and/or with issue tags?
  • we want to be able to test some part of the code more in depth than other parts? → Why should it not be possible to test only parts of the whole repo?
  • Is it because nixpkgs advances too slow? → I assume this is caused by technically only having one master branch, ergo only one git chain which is not flexible? Could this be solved alternatively by e.g. having a more complex merge strategy with more branches = more chains = being able to e.g. delay failing packages while working packages could already be advanced?

The whole “meta” discussion with a Federated Republic of Nixpkgs doesn’t seem like something that needs an actual technical change. I lost you down the road but it seems more like a discussion about where the community should go in a cultural way [which can then indirectly lead to different decisions on technical changes] .

7 Likes

Excellent observation! Discussing this would hopefully finally lead us down a constructive path and should be a different thread. @ryantm can you split this out, too?

2 Likes

I’m not sure whether I’m in a tiny minority, but the main blocker for me in learning to contribute to nixpkgs has been lack of motivation in learning github, partly because it’s complicated and partly because it belongs entirely to Microsoft (end therefore can be embraced and extinguished fairly rapidly).

1 Like

I split this into its own topic.

3 Likes

Nix doesn’t really have a technology issue, it has a people issue. There’s just too few trusted people in this purely volunteer organization to handle the overhead of issues and PRs. Plus people’s engagement with nixpkgs is subject to change as personal and professional factors change.

It’s been my opinion for over a year that the NixOS Foundation should have a “community liason” person which can focus on triaging issues, reviewing PRs, and other work which volunteers wouldn’t want to do in their free time.

Even if I finish GitHub - jonringer/basinix: (WIP) Nixpkgs pull request review website which should remove most of the “what can be automated” from a nixpkgs review, you still want human eyes to review PRs from a compliance perspective. This still requires someone’s time.

16 Likes

I genuinely don’t properly understand which of the current discussion topics leads to the conclusion that the technical solution should be a split.

I take for granted that I’m probably massively misunderstood. I don’t mind. We’ll figure it out together.

The technical reason that best assimilate my purely social/organizational motivations is: interfaces. A split would require us to improve our interfaces (document, honor, stabilize, etc.). It would make a lot things harder, too.

The social dimension to interfaces is that they separate realms of concern, and preserve each realm with a certain autonomy, that they otherwise would not have. That in turn might boost diversity in solutions (and if only diversity in speed of decision making and implementation!), and form there on we can see “profit” on the horizon, I guess.

But I also think, since our BDFL is pretty much absent, it is very hard to meaningfully steer the (entire) ship (at once).


Many of you have seen pijul, I suppose. I haven’t tried it, but I’m waiting for the moment I’ll be able to and see how it compares to git.

The disadvantage of github, because it seems “complete”, in my opinion is, that few people take the chance like @jonringer and attempt to implement a greenfield solution with our actual needs meaningfully staring at them upclose.

With some kafka (any message queue) at the heart of our workflows with a couple of pretty decent interfaces, I fantazise we would already have automated the shit out of it.

1 Like

Framing this in “pros/cons” is useful, because in any change we might make, we should not only consider what ‘cons’ of the current system it alleviates, but also whether it breaks any of the ‘pros’ of the current system.

In the ‘pros’ list:

  • I love how a single nixpkgs commit hash describes an entire consistent collection of all packages and associated tools
  • I love how for designated commit hashes (those that are part of a channel), the packages that are part of that hash have a high likelihood of being available in the binary cache
  • I love how a contribution to a package whose maintainer is inactive is not stuck, but can be reviewed and merged by non-maintainers

I, too, suspect bots might enable great opportunities to reduce the burden on those with merge rights, either by automating work/checks or perhaps even allowing a wider range of maintainers to trigger merges of changes that meet certain criteria. Of course we already heavily use bots and it is a huge help. It is very non-obvious what work would be useful to automate, or what criteria should allow this, though. I think we should flesh that out in separate topics around particular ideas.

I agree the ‘big heap’ of issues and PR’s is daunting, even with our already extensive use of tags/metadata. Having a way to filter/browse/subscribe to the issues/PR’s I might be interested in as a ‘more casual’ contributor seems useful for me. I’ve started sketching something like that in ~raboof/nixpkgs-contributor-dashboard - sourcehut git

Define ‘too slow’ - actually I think it already advances at an impressive speed :wink: . It would always be good to speed it up, though, of course.

My gut-feeling is more branches/chains would make things more complicated while it’s not obvious it would help. I think ‘green hydra’ strategies have been discussed in much more detail in other threads, but in summary I think there are 2 things to do:

  • prevent breakage: adding more validations to PR’s, more tests, etc. There are of course resource constraints there
  • fix breakage when it occurs: this might actually be where much more opportunity for improvement is.
    • this is where ‘package tiers’ might help: we could decide when ‘lower-tier’ packages break, we just mark them broken and allow nixpkgs-unstable to advance. It’s of course still good to fix those packages, but it can be done ‘in parallel’ by those who care about them without interfering with the main project
    • also I think we have an opportunity for better tools for collaborating on failures: right now, when you see nixpkgs-unstable is not advancing, it’s not super easy to find out why it’s failing, who might already be working on this, what they found so far, etc. Introducing tools to find that information more easily might make it easier for people to contribute to fixing the breakage.

Anyway, it’s much easier to dream about what could be improved than to actually work on improving it, and it’s hard to say if it’s really that productive, but it’s hard to resist… I hope you excuse my indulgence :wink: .

2 Likes

I’ll just chime in to say that for me personally the fact that NixOS/nixpkgs is on github was actually the deciding factor that led me to contribute to it and use it in the first place. I would never dream of trying to contribute to a project like debian or ubuntu simply because I don’t even know what tools they use or where to begin. But I’m very comfortable with git and GitHub, and this made nix far, far more approachable.

It also just seems far more modern IMHO. Whether you love it or hate it, the OSS community lives and breathes git and GitHub. And so it “feels right” that a progressive, up-and-coming OSS distro should operate on those tools.

I know OP isn’t actually suggesting migrating away from git/github but just wanted to throw that out there.

20 Likes

Exactly this. I’m not a major contributor or anything, but even though I’ve been using Linux for a long time (mostly Arch), until Nix I never contributed anything. I have zero desire to figure out how to send patches to a mailing list or something, and it’s hard to even begin figuring out where to look for bugs, fixes, how to create new packages, how to apply fixes to broken packages, etc. for most Linux distros.

With Nix, when something is broken (which to be fair seems to happen more frequently than many other distros, at least on unstable) or doesn’t work how I want it to, thanks to Git and already being familiar with it I can fork nixpkgs, fix it myself, put my own fix into an overlay to immediately apply/use the fix, then easily submit a PR to nixpkgs to share the fix. Or I can find a last working point/hash and pin the system to that.

If for some crazy reason Microsoft decides to go evil with GitHub, I think GitLab would still enable basically the same functionality, ease, and transparency. Until then though I actually think nixpkgs being on GitHub is a huge win.

9 Likes