Low-effort and high-impact tasks

Development
3

This would basically drop their property of being experimental.

Though flakes are still not there yet. The commands are still not there yet.

Instead of having a non-empty default for the experimental features, we should make sure to stabilize them, and get 3.0 released.

@edolstra said in his talk during SoN that he expects 3.0 before or with 22.11. I do not see that though.

And also: nixos-generate-config needs to be redesigned to be able to generate flake based configs.

In my opinion: Get nix 3.0 ready and give us at least one nixos release to adjust the tooling.

This is not low-effort high-impact.

8 Likes
4

Make nix package searches (CLI and web) specify if there is a nixos module for the package, and recommend using it if there is. New users often miss the module vs. package distinction, and this would clear up a lot of bumps in the road for them.

It’s tedious to connect up the data, but not actually difficult.

14 Likes
5

A fresh documentation pass on the nixpkgs.lib functions, and the various generic builders and such (e.g. makeRustPlatform). Ideally finding a way to write doc comments for them so we don’t need this every few months.

I think this is relatively low effort, if a bit tedious. The pass doesn’t have to be complete. But there are so many undocumented functions, and even for those that are documented I often have to dig into the bowels of nixpkgs to find out what arguments they take. Half of mkDerivation’s features seem undocumented (like the noCC attribute)!

Also, help get Preface | nixpkgs over the line. Then this rework won’t be lost to the unwieldy nixpkgs manual.

7 Likes
6

This question is not easy to answer (for me at least) but I will try my best.

Most things that annoy me in my regular (day- and night-time work) that are low-effort are already fixed (by myself or somebody else) in nixpkgs/hydra or in nix via downstream patches. This is why to me personally, only medium- to large-effort tasks remain, although these can be broken down into smaller chunks.

While I’m happy to have the knowledge to fix these smaller issues myself, I am also aware that not everybody has this knowledge, especially newcomers which can be very frustrating. The onboarding of new users has gotten a lot better over the years, but a lot issues (especially when trying to understand the basics) remain (see also the next paragraph on that).

Two major points I’m going to mention but not discuss any further are a high dissatisfaction with the instability of cli-interfaces and the lack of documentation in general. I got around that issue by suffering through reading code and trial/error and I know this situation improved a lot and I love seeing that work is being done by you. Documentation is the most important thing (especially for new users) imo, but also for more advanced users. Just having complete lists of things like builtins and advanced derivation attributes would be a great help.

Last-minute addition (but potentially not relevant for this thread): I also think it’s weird to newcomers who want to learn about NixOS and go to https://nixos.org (which clearly indicates a operating system in its domain) to have no information on the distribution whatsoever. The only places the distro is mentioned is the announcements, the manual and the download area afaict. That seems like not too much effort to change while potentially being high-impact to new users. Also compare the website to Arch which clearly states what Arch is in the first paragraph and Debian which mentions the existence of an operating system prominently on the landing page.

But you pinged me because I am on the team list as the 22.05 RM so I’ll try to focus on topics that were hard to deal with when I did the release. It’s 3 points essentially, most of these are hard to estimate the effort of.

Responsibilities

It’s not clear at all who is responsible for what. When I had a subsystem or something, I usually knew who to DM because I just learned these names over the years. To somebody who is more absent in the community, it’s hard to know who to ask and who can make authoritative decisions. As an example, I had to work a lot with the Staging team during the release. Most responses and decisions were made by @vcunat. There’s nothing wrong with that and I’m glad he helped me throughout the process, but there is technically no reason to trust him with these topics (other than that I know he does a lot of staging things) and there is also no reason to not make decisions myself when somebody asks something in the Staging Matrix room. If he and I got into a disagreement in the channel, who would be the one to make a final decision?

This is also an annoying point when leading a team yourself. I was the Release Manager and I had to make every decision that didn’t directly impact a subsystem myself. While I know this is one of the tasks of the job and I just decided a lot of things, sometimes it felt like somebody higher up should make a single decision. Only issue being that there is nobody higher up, there’s nobody I could ask. The only one was @jonringer who was able to answer me some questions but only because he did releases before. He didn’t have any official saying in anything. Again, I’m really glad he helped me (even in a phase where he had limited time), but it just seems like there should be a more formal structure.

This is something where I can see the new board bootstrapping a structure of responsibilities. I don’t expect them to choose responsible people for each and every area of the ecosystem, but forming a top-level structure where sub-structures can build themselves would be a great start. Just finding a small team who manage the subsystem teams on nixpkgs for example would be awesome. These responsibilities should however not come without anything in return. Financial compensation is probably not possible or fesible and the ability to have authority is not enough for some people so I really don’t know what to do with this.

Communication

Goes hand in hand with my previous issue. How do I communicate with teams? Do I just DM Graham because I know he has admin rights in the infrastructure or do I use the channel where responses may be slower? There is no clear way on how teams communicate (which is probably also caused by my next point) and since there is no feeling of responsibility, questions in Matrix rooms often stay unanswered.

While the communication in one direction (me having a question about a subsystem) does often not work, the other direction is difficult as well. There is no proper way to convey current information about changes to users of your subsystem, other than writing a full Discourse post which often feels too heavy as something the weight of a tweet would have been good enough.

Bus factor

Some very important areas of the ecosystem have a horrible bus factor (1 or 2 if we’re being lucky). This doesn’t help with the previous point as there may be nobody answering my $SUBSYSTEM question just because the only person who maintains it is on vacation. That is horrible to new users who may get the feeling that areas are unmaintained and left to rot. This also means that having a user drop out of the community may cause a subsystem rotting for a while. As there is no higher authority who oversees the teams, there is no pressure to re-form a team in that event as there is no leadership anymore.

This is probably not too much effort to fix, just reminding people that there are teams/subsystems/areas that are completely unmaintained and ungoverned often helps. Whether a area is actively maintained or just in maintenance mode is up to the respective team, then.

Now that I write down these points, only the first one seems relevant and the other ones are something that feels like it cannot be controlled without a lot of effort. I’m still going to post this because I think my experiences as RM in the context of this thread are worth writing down. Also, as most community members have never been RMs, these viewpoints may not be clear to everyone. If you want to chat about the topic of release management, feel free to reach out to me in the Release Management room.

14 Likes
7

What are the smallest changes , requiring the least amount of work , that would produce the largest positive impact on the Nix ecosystem and community?

Simplifications and fixes to papercuts.

Double down on declarative configuration. Phase out the nix installer in favor of installing via NixOS, home-manager and nix-darwin. Make the latter two official NixOS projects. Integrate them into the nixpkgs repo. This makes installation and “best practices” so much easier and it greatly reduces the number of installation variations that need to be considered by nix and nixpkgs contributors.

(Also, nice to have, but not low hanging: eventually integrate nix-darwin into NixOS, allowing shared modules)

Regarding papercuts, here’s one: having to specify runHook which can be made unnecessary.

Probably the UX label can be considered paper cuts. Same for the DX label.

6 Likes
8

This reminds me of The papercut thread - post your small annoyances/confusions here

2 Likes
9

That thread really makes me sad, because all of those posts should be GitHub issues.

For instance, there is no meaningful way to go back and tell what has been fixed, short of asking individual commenters to strike through and link to the respective pull request or delete their post altogether.

What I am trying to do here is figure out which of those issues to prioritize, ideally those which are not just paper cuts. Dealing with the paper cuts alone would a full time job on its own.

3 Likes
10

should discourse development category should be discouraged / removed?

11

I don’t think so. Think of this as the newish github discussions tab. This is a place to discuss what should be done, while issues are more concrete packets of work that have mostly been thought out. Having a place for hypotheticals and bigger architectural/organizational ideas is useful.

Things that crop up here should absolutely be converted into tickets though, ideally with an associated epic/tracking issue, and linked to this thread. I expect that @fricklerhandwerk is thinking of doing just that :wink:

4 Likes
12

That thread is explicitly not meant to replace filed issues - in fact, the reason for its existence is explained in the opening post. In reality, for small papercuts, the choice isn’t “papercut thread vs. GitHub issue”, but “papercut thread vs. never being recorded anywhere at all”. These papercuts were never going to be GitHub issues to begin with (and the thread has already repeatedly proven useful).

To your question: I’ve been working on a cheatsheet of utility functions that groups them by purpose, rather than alphabetically or by subsystem, a HTML export of which currently exists here (there are known errors in it though).

This cheatsheet has already repeatedly proven useful not just to myself but also to others in the community, for quickly discovering the correct utility function to use for a given problem. Something like this should probably exist in the NixOS documentation proper - but historically this sort of thing has been blocked on a desire to keep a strict separation in the docs between Nix, NixOS and nixpkgs, meaning that such a ‘combined view’ has no ‘correct’ place to exist.

3 Likes
13

I’m curious how much of the packages in nixpkgs have zero dependants. If we could isolate these “leaf” packages then that could feed a discussion on breaking up the monorepo. I’m not saying we should break up nixpkgs, but I don’t know that we shouldn’t.

14

Issues and PR
Sitting down and cleaning up old issues and PR. There are a lot of them lingering, and yes the bot can mark them stale, but we should really go back and do some cleanup. It may not be “low effort” but it is also mostly because it is toil.

And it may not seem high impact, but the reason noone open issues is, quite simply, that they think it is useless because they get forgotten.

Cross Compile in nixpkgs
Even without making it work, I think a report on the state of cross compiling with nix in nixpkgs would help a lot. There is a section in the manual, mostly about the dependency stage and all. But it is far from obvious on how to go from a non cross compilable to one that do cross compile. I really think that having an example package for cross compiling something that bootstrap itself would help. Even better if we get a good guide. It is high impact due to the amount of ARM being pushed around for servers these days, while most “free” and “accessible” CI are on x86. Being able to produce output for both in a build system would probably be nice for marketing. It would be a killer feature for multiple organisations i know.

I spent 3 months trying to get erlang to cross compile and i still could not get the bootstrapping to work correctly.

Write down a guide to package deprecation in nixpkgs
When we deprecate a package in nixpkgs, what are the steps ? Which files should be changed ? What to add to “throw” when the old package is in config ?

I have deprecated a few over the years and every time i learned of a new thing to change thanks to the amazing reviewers. Having a guide to point people at would help a lot.

Language ecosystem nix-shell example
For every language ecosystem doc in the nixpkgs documentation, let’s add an example basic “good practice” nix-shell derivation, so that people can simply get started with their languages. I know @domenkozar mentioned in his Summer of Nix code to try to automatically detect it. But we can start by supporting it easily in the docs.

7 Likes
15

Maybe proposing generic and documented tooling to nixos modules to ensure they don’t leak secrets to the store. Several attempts already exist:
https://github.com/NixOS/nixpkgs/pull/78640
https://github.com/NixOS/nixpkgs/pull/174978
a more ambitious one:
https://github.com/NixOS/nixpkgs/pull/179471

I think these PRs are in limbo because there are not many reviewers of this kind of large-looking change, and because there cannot be a perfect solution (Introduce a type for options that contain the path to a secret file by symphorien · Pull Request #174978 · NixOS/nixpkgs · GitHub)

but that would be a large improvement still, instead of having a situation where half the modules just copy the secret to the store, and the other half reimplements a shell script that bakes the secret into the configuration file in ExecStartPre (and an significant proportion does it with sed, which leak the secret to /proc)

10 Likes
16

I very much support this idea!

Beyond “install with environment.systemPackage” and “install with a dedicated option” there is also

  • “don’t install at all and use nix-shell instead”.
  • “install with python3.withPackages”
  • "install with
weechat.override {
  configure = { availablePlugins, ... }: {
    scripts = with pkgs.weechatScripts; [
      the script name
    ];

(this one is documented in the manual, but how many programs with plugins don’t have a dedicated section in the manual?).

In other words there is a need for a pretty generic meta.usageHint. So I suppose this should go through a RFC and that’s where I fear it stops being low effort :upside_down_face:

8 Likes
17

Should it, though? I feel like there is value in trying to implement this in practice first (with the understanding that it’s experimental), figuring out what works and what doesn’t so that the problem space can be mapped out fully, and only then establishing a formal policy for it via RFC.

1 Like
18

This needs an RFC as it requires coordinated work between the search page, the nix CLI and nixpkgs.

1 Like
19

I was under the impression that the purpose of the RFC process was to seek community consensus, eg. for standardization of practices in nixpkgs or community matters, where it’s important to hear all voices before definitely settling on something.

Is it really meant to be used for anything that involves coordination between more than one team/group? As that seems like an impractical amount of organizational overhead to me - lots of things fall into that category, and it would significantly slow things down if all of that needs to go through an RFC.

1 Like
20

I have never really seen the RFC process like that.

For me RFCs, PEP, EEP, and all those from the various ecosystems should be used to standardize certain stuff before dissent can occur.

You can not try around different implementations and throw yet another PR to each of nix, nixpkgs and the search website to sync their functionality.

What happens when you use a nix from iteration n to search a nixpkgs from iteration n+3? This scenario is very likely to happen as iteration n will eventually part of a release, and unstable channel will move along.

What happens when you use a nix from iteration n to search a nixpkgs from iteration n-3? This scenario is likely to happen when someone using a nix from unstable or master tries to search on a stable release of nixpkgs for whatever reason.

There is a lot of reasons to not even introduce any dissent!

21

PS: My general assumption about “low-effort high-impact” is, that any such task that might exist and is know to someone would already have been implemented or it is hold back as it was a breaking change/would make future changes harder (eg. releasing flakes today). Or the only person that knows about it has no clue about the ecosystem and conclude it being “high-effort” and keep it for themselves. Therefore, I consider questions that leave the effort open or ask “what would you do if you had unlimited ressources” much more valuable, as those might reach the people that estimated the effort wrong.

Of course: the middle ground is covered by neither of these questions.

1 Like
22

This is not how effective standardization works in practice - it’s how you get “design by committee”-style standards that fit real-world applications poorly. It’s very rare for a problem domain to be fully understood right from the start, and often the nuances and peculiarities of a problem are only discovered in the process of solving it and gathering user feedback.

That means there’s roughly two realistic options for standardizing things:

  1. Standardize upfront, discover it doesn’t work, have everything diverge in practice because the standard doesn’t actually solve their problem, and then invent a new standard that solves these issues. In this process, those diverging from the standard will likely have to expend a lot of effort to defend their divergence from the broken standard, because it is viewed as “official”. This likely creates a lot of friction and negative feelings towards others in the process.
  2. Experiment first, to figure out what works in practice, then over time converge the implementations into a standard that covers all the actual needs that people have. This is a bit messy implementation-wise at the start, but eventually can meet all needs.

One way or another, you’re going to be standardizing after experimentation, that’s just fundamental to exploring a problem domain. The only difference is whether you also introduce an additional painful and high-conflict standardization process before experimentation, which is what happens when you try to standardize right away.

There’s a reason that pretty much every effective standards process in the wild has an experimentation period baked into the process, where implementations are allowed to diverge or become incompatible over time, up until the spec is finalized.

1 Like