Doc: Unified (automatic) reference manual

Announcements
1

Sharing this here; Just in case somebody needs it. :slight_smile:

It is still a prototype, but might be a valuable resource for some.

Should be pretty complete by now, but may have some minor bugs, that i need to work on.

16 Likes
2

Thank you. Is this generated from source?

3

Yes. It is autogenerated from nixpkgs unstable and the latest nix release. (for builtins)

1 Like
4

@fricklerhandwerk what do you think of adopting this for the default reference documentation?

5

For the Nixpkgs library stuff, please go wild and upstream it. I know the barrier to contributing to the Nixpkgs manual is still quite high and it’s hard, very hard for many, to iterate on it. But IMO this should be the single source of truth, where the setup should be documented so thoroughly that anyone can incrementally extend and improve it. Such pilot projects as @hsjobeki‘a are a great way to show a possible direction though.

Nix builtins are already being documented in the Nix manual. As @infinisil and @roberth pointed out, Nixpkgs users should probably not even be encouraged to use them, as that will make dealing with long-term compatibility in Nix itself a bit easier. I strongly urge to keep that separation of concerns and bridge the navigation gap through better cross-referencing instead of lumping things together that shouldn’t be seen as one.

We, the docs team, are continuously working on improving discoverability and clarifying the ecosystem’s architecture for beginners on nix.dev, and I’m convinced this is not only enough, but actually helps keeping the scope of individuals’ work in check. If I learned anything in the past two years is that attempts big changes are very likely to fail. @7c6f434c wrote on multiple occasions that this is a feature, and I now tend to agree.

2 Likes
6

I still think that’s a very opinonated question.
Reference documentation doesn’t encourage anyone for using a function in nixpkgs. It is only displaying (objectively) available functions and how to correctly call them. This doesn’t offer any guides / tutorials or further contributing guidelines, that you’d need for contributing to nixpkgs.

Not having them documented all at one place is a totally different question. And there is also a big and growing opensource ecosystem around nix they might have different rules on how they deal with builtins.

So correct me that this is something opionated to nixpkgs and shouldn’t affect objectively correct and complete reference documentation for the nix ecosystem.

Yes we should make it a small change. Would it be an option to just let the manual be as it is, and put some efforts into this new kind of documentation instead. The manual sucks ever since and swallows all our efforts without major improvements (i feel like that).

A possible roadmap: (The first three steps must happen anyways if the rfc is accepted)

  • Refactoring our comments (according to rfc145)

  • Update the current manual rendering (for backwards compatibility)

  • Native nix (c++) implementations for doc-comments.

  • Stop invest major times into the manual. Instead focus on the new comment format (see the RFC) In parallel, until is better than the current manual.

  • We can also easily create a special output for nixpkgs where only functions added from nixpkgs exist.

  • Focus on writing good doc-comments and documentation.

7

This is amazing, I’ve wanted something like this for a long time. Though it seems there’s a lot of overlap with noogle, I like the browsable UI of this approach more.

I’ve found a few issues and added them to the repo.

I guess this auto-generated reference could add a note on every alias of a builtin that the lib.* alias should be preferred over the builitins.* alias? Seems simple enough. I do appreciate that this shows what functions are aliases of builtins, though, so it would be a little sad to see that feature being removed.

2 Likes
8

Maybe it’s possible to find a path to integrate that work with the documentation team’s plans? The degree to which we can make room for new contributors is also what makes this community healthy or not. It’s better to have something that works today than building a perfect cathedral alone.

3 Likes
9

@hsjobeki what you’re working on with RFC 145 and all its implications is a very big, heroic endeavor, and I have a lot of respect for that. So the following is my way of trying to be constructive.

Ideally that would be the case, but in practice I don’t find that to be true: Things that are not documented (or badly documented, or hard to find) effectively don’t exist for all but the few brave souls that dig into the source code. Conversely, things being simply put side by side can leave an impression they are somehow equivalent and it doesn’t matter what you do with them. We can use these signals to our advantage and nudge people to do things that make their lives and our long-term maintenance easier.

In any case, we probably only disagree on superficial aspects. Let’s set them aside until more of your work materialises upstream.

The “Nix ecosystem” is not well-defined, and my gut feel is that it’s futile to pretend it is or to try forcing it.

I deem it more fruitful to instead strengthen separation of concerns, because it promises to

  1. Make it easier for new users to understand the system (i.e. build a correct mental model of it) and hence use it more effectively, in a smaller amount of time
  2. Make maintenance more tractable. When problems and their solution mechanisms are not tightly scoped, we tend to get into system-wide architectural debates that cost time but don’t turn out concrete improvements.
tl;dr: Please go wild on the Nixpkgs manual

I don’t feel like I have any authority over the Nixpkgs manual, as I barely work on it and don’t know it very well, so my opinions on that subject are not well-informed and largely a matter of taste. If I were maintaining it, I wouldn’t want to duplicate documentation that’s in the responsibility of Nix maintainers; it’s a source of more work and confusion for users. Noting which library functions are aliases to builtins (or could be, since they’re often conditional on the Nix version) would be really helpful @iFreilicht, but not a concern for this rendering feature - that just has to be written into each function’s docs.

But again, I’m not the one to tell anyone what to do. Feel free to propose changes as you see fit and get buy-in from whoever maintains that code!

However I do maintain nix.dev and the Nix manual, and here I have a clear vision:

  • In nix.dev don’t lump together things that live in different components.

    The goal is to communicate very clearly the distinction between each of the major ecosystem components, primarily to ease onboarding and help people decide what to focus on and where to go when they want to learn more.

    Also nix.dev should mostly serve as guidance, i.e. be a thin wrapper around actual documentation. Anything else simply doesn’t scale. We the documentation team, and I personally, are already constantly being drawn into more responsibilities. If we don’t set boundaries we won’t finish anything and won’t be able to help anyone.

    Therefore I’d say: No, put function documentation in the reference manual for the component where they are defined. We’ll gladly provide pointers to that from nix.dev.
    around

  • In the Nix manual make sure that Nix, its constituent parts, and their maturity and support status are represented correctly. It should be the single source of truth on everything that is specific to Nix.

    Therefore I’d frown upon builtins documentation being worked on somewhere downstream. Please just make PRs to Nix. I know it’s not fun or easy most of the time, but it’s getting better literally by the day.

    Somewhat related: We won’t get far with the RFC 145 doc comments syntax for builtins, as those are implemented in C++. We’re lacking the infrastructure to produce automated builtins interface documentation, and I’d appreciate help in that area a lot.

This is always an option and you’re free to put effort into whatever you like.

tl;dr: I still recommend incremental changes to upstream instead

I don’t think more new kinds of anything is beneficial to the broader ecosystem though, because:

  • The bigger the new kind of thing grows,
  • the larger the diff to upstream,
  • the more upstream appears to be deficient or unmaintained,
  • the more (especially new) users perceive the ecosystem to be an uncoordinated mess,
  • the less likely any of that ever gets fixed.

I’ve seen so many examples of that in the past that I lost track. All of them ran out of steam eventually, leaving the ecosystem in a state I consider worse than before: with more half-finished ideas scattered all over the place.

Upstream is the only thing that will survive all other efforts and still have the largest impact because that’s what new people start out with and take for granted. Yes, it’s operating under constraints that everyone I know rightfully considers very annoying at best, with many hard-to-predict barriers to contribution. But that’s the commons. I have no better answers than “make smaller changes, they’re easier to produce and to accept”. Let’s fix things step by step.

I fully agree on the first part, assuming we’re still talking about the Nixpkgs manual. I don’t think there is all that much effort being put into improvements (especially in comparison to other things being worked on), but the effort needed to get it into a shape I’d like would indeed be substantial.

But why not instead help to make it stop sucking? It’s mostly not fun, it’s not short-term rewarding, but this is where small changes will help immediately and add up over time. It’s a process that will quite predictably lead to good results, if ever so slowly.

Up to recently it wasn’t even easy to find what could be done concretely, but for a few months now we at least have a first stab at an overview.

I don’t know anyone who invests major time into the Nixpkgs manual. (Which doesn’t necessarily mean much, but I should know, right?) Doing things in parallel until they’re better than the original and can be used as a drop-in replacement is exactly the kind of far-reaching change that I don’t expect to come to fruition within a time span I’m comfortable planning for.

To get back to constructive feedback on what you’re trying to achieve: You’re tackling a real, far-reaching problem and I think you’re doing great. Expect the process to take a Long Time™ to come to conclusion. Showing a proof of concept is excellent. Don’t get discouraged if people don’t agree with today’s version of your vision. That’s too much to ask and you’ll need buy-in for the whole package to get it in. Again, making the package smaller will make it more likely.

Therefore I really recommend you to focus on getting incremental changes merged into Nixpkgs that don’t depend on the big outcomes such as the RFC to land or the Nix language evaluator to implement it. That will help you build up mindshare and increase the bus factor you’ll need to get the big chunks of upstream work that are still to come merged. Find your people, don’t rely on me or anyone already entrenched in something to approve or help in substantial ways. As @zimbatm implies, that would only slow down both sides.

I’d love to see you taking ownership of library documentation in Nixpkgs, and ideally share it with a few others. Where you really need approval for some reason: I definitely prefer changes the full extent and consequences of which I can gasp in a few minutes, and likely this is true for many others who are busy enough with their own area of responsibility.

2 Likes
10

I think it could make sense to take the ownership of doc-comments and their rendering. Might visit one of the next documentation team meets to discuss that.

I‘d love to share ownership with someone that is familiar and can drive the changes we need for the whole manual rendering.

11

Maybe a bit out of topic, but could this be integrated with search.nixos.org? Having a universal search that can provide results for all the projects could be super helpful. Especially if it explains the difference between all of them.

3 Likes