Monorepos don't map to our social structure

Development
1

(edit: this was in response to notable contributors leaving the project for social reasons. i don’t know how to edit this to free it from that context, so just read the start of the conversation linked below here by Discourse if it feels too ungrounded).

i came to NixOS (first my server, then my desktop, then my phone) specifically because i wanted to escape from under certain actors: Apple, Google, NSA, MPAA. draw the map, it’s pretty easy to guess who i do or don’t trust to coinhabit a space with me from that.

i don’t like force. that’s the point, that’s why i’m here. if the sponsorship policy hasn’t stopped the bleeding, then here are more extreme ideas to consider:

  • i don’t think NixOS should have a board, because i don’t think it should elevate anyone to an absolute position of power.
  • i don’t think we should be putting all our packages into a single, authoritative repository, because the social layers of NixOS can’t be mapped onto such a structure (would we have lost marsam’s maintainership if we were pulling from his repo instead of asking him to push to ours?).

don’t mistake this as a call to delete these things immediately. they’re just ideas i think we should consider, directionally.

it’s mentioned above: nearly every maintainer is already comfortable with overlays. when i’m looking for a package, i check nixpkgs, nixpkgs PRs, and then the NUR. if i can’t find it, i package it locally, where it’s instantly available to anyone pointing at my public nix config, and the NUR pulls from my repo a few hours later without any interaction from me. only after several weeks, once i’m committed to keeping the package around, do i upstream it to nixpkgs – with a bunch of caveats, the most notable being do i care enough for the people who would benefit from me investing extra time to make the package more accessible to them.

so firstly, i encourage anyone less willing to upstream into nixpkgs to consider keeping your packages available in some form, if any of these other forms are less burdensome or more agreeable to you.

and secondly, anyone serious about architecting the components of a NixOS system with an eye to reducing social friction (NUR OS?): loop me in on those efforts. i suppose i’ll lurk in nixpkgs-architecture, at least. merge bots are one cool idea, i wonder if they dovetail into larger things.

15 Likes
2

Very much agree wrt. moving away from a monorepo. It shouldn’t be a point of pride that we have the largest repository on GitHub (IIRC?).

I don’t see how the NixOS foundation would exist without a board though. Someone needs to make decisions for the foundation.

7 Likes
3

Moving away from a monorepo also brings the problem of efficiency, trust, etc. The way I understand this is that if we don’t keep the nixpkgs in sync, the immutability nature of nix packages will use exponentially larger disk space, which is not ideal. Having a centralized repo for packages with many active maintainers means that it is less likely that someone can inject malicious code (not impossible, but harder), and it is harder to trust random flakes/overlays from around the internet. There will probably be some curated lists, but there will be much less manpower to review the individual packages. And I think flake still lacks some features that are quite important, and iirc there are long standing PRs trying to address that but are not yet ready for merging.

Technical things aside, I think the recent events do show that there are some serious issues with the current project leadership, and not just with a certain individual. Merely abandoning the sponsorship will not fix the problem… What formal decision procedure are there for things mentioned in the open letter against Eelco’s leadership? Is there a plan to fix it? How will the moderation team respond to accusations in rfcs/rfcs/0175-appeals-council.md at e83bd8fcd6b94af02489a1f2af2f911f96d1c7a5 · nrdxp/rfcs · GitHub? Is there a boundary on what the “NixOS” community is? What if it is in another forum, and the speech is not targeted at an individual?

10 Likes
4

re monorepo I’ll just link Contributor retention - #42 by Ma27
Reducing the scope of nixpkgs may be reasonable[1], but I haven’t heard satisfactory answers to how to manage a critical set of packages then.

[1] Though for every single flake I haven’t written on my own I had to read the sourcecode to know how to best include it which seems like a downgrade to me.

9 Likes
5

This is a red herring; Linux isn’t a monorepo and they don’t have this vague “problem of trust” that you allude to. And after the XZ debacle I think the whole “it’s on github so I can trust it” myth is dead too.

The real obstacle is that desynchronized development, like kerneldev uses, requires conscious effort. You have to adjust your development style to deliberately avoid things that will cause merge conflicts. Right now nixpkgs interleaves version bumps with (sometimes OCD-driven) treewide style changes, which makes anything other than “merge as quickly as you can” miserable.

It doesn’t have to be like this.

Without any changes to Nix, there is an architecture which avoids rebuilds when a dynamically linked dependency changes, so long as the symbol table of the .so file does not change (i.e. set all symbols to zero length) which covers basically any change other than a version bump. This is different from both pkgs.replaceDependency and also from guix’s grafts.

This does not require content-addressed derivations. However it does require a strict hygenic separation between the buildPlatform and the hostPlatform essentially equivalent to cross-by-default.

Both of the above changes (development style and symtab-hash dependencies) have an incentive problem. They are positive changes for nixpkgs in the long run, but since they make it easier to maintain competing forks those who manage the monorepo are disincentivized to adopt them. Also, both changes require a substantial amount of real work.

9 Likes
6

FWIW IMHO the Linux monolithic monorepo kernel is horrible. We should not replicate it.

The other issues you mention will be fixed when we make use of dynamic derivations.

2 Likes
7

we could drop python packages unused in tree, move in-tree projects (like yarn2nix) in separate repos, reduce big package sets like already done with nodePackages or question if including all of hackage is necessary.

Reducing inter package dependencies like the ones used in wrappers or like we already did for openssh, curl, etc could be an option.

Or declining RFCs that balloon the scope and only add interfaces for them and move them out of tree.

Moving literally everything into flakes doesn’t scale and doesn’t work but could be a good option for single big package sets (eg auto generated package set like for vscode extensions), modules (eg authentik) or hardwares (Jovian aka Steam Deck.

14 Likes
8

I don’t think Linux is monorepo? As far as I understand, it is a single source tree, which is split across several repositories.

So development of different subsystems happens in different forks, and then changes are pulled in bulk from forks into the main repo.

Admittedly, this “single tree, multiple repositions” approach is not at all popular among projects hosted on GitHub.

1 Like
9

Interesting! This is basically just git subtrees.
We could do that too, but we need more than one GH repo for Nixpkgs. It isn’t scaling.

1 Like
10

This effectively already happens with python and haskell; updates to these subsystems are prepared in separate branches and then pulled into staging(/-next).

3 Likes
11

If nixpkgs weren’t a monorepo, I doubt I would contribute. Dealing with Darwin breakage is already painful. If I had to do that across however many repos, especially if any of their maintainers were apathetic or hostile towards Darwin support? No way.

14 Likes
12

I think at least lib would be ideal to split into a separate repo, and once I have some time I’d like to make that a reality :smiley:

35 Likes
13

Nixpkgs being a monorepo has some serious advantages, like the ability of finding absolutely anything with a simple grep. The repo is also not that big of a monorepo, so you don’t need special tool to clone and check it out.

That said, several components could definitely split out and use separate issue trackers, NixOS for once, without major disadvanteges. However, I wouldn’t recomment that unless we get flakes stabilised, because the current composability of Nix repositories is pretty bad.

26 Likes
14

The solutions to social problems aren’t technical, they’re social. I don’t think splitting up the monorepo is the answer there (though maybe its a good idea for other reasons), because people just avoiding each other by not having to interact often won’t solve this problem.

11 Likes
15

Very much disagree. These two go hand in hand. So much of what we do is technological solutions to social problems.

2 Likes
16

There is nothing in Flakes that helps with this, it’s an orthogonal problem. Overlays are kind of working, but they don’t compose that well. The module system would work, but is too slow on that scale.

7 Likes
17

I said composing but I was thinking more of the hassle that is managing multiple sources of Nix packages with channels or using fetchTarball; not composing packages sets.
AFAIK, flakes should solve this by proving a way of specifying inputs to your configuration and commands to easily pin and update them.

1 Like
18

Also neatly solved by any other pinning tool like npins, niv or gridlock, there’s nothing exclusive about Flakes that would make it a better solution for pinning (in fact it’s a worse solution in at least one way) :slight_smile:

In fact, I’m very interested in using such tooling for individual package sources as well. Flakes definitely wouldn’t be a good fit there, because we don’t want a flake.nix file for every package (at least not in Nixpkgs) and it couldn’t support non-builtin fetchers.

6 Likes
19

Yes, the problem is that they are all third party tools.
I would really appreciate an official solution in the Nix CLI.

5 Likes
20

I think it’s not even that, it’s one standard method of composing Nix project that we really need; an interface between Nix projects. Flakes aim to be that standard.

6 Likes