My first idea was: Why not use Markdown? Everyone can use it!
But after checking out AsciiDoc, there are good reasons to use that over Markdown for documentation.
I think one of the most important things about documentation tooling is to make it trivial for people to fix documentation issues they come across. Requiring users to learn a new format is non-trivial.
For this reason alone my opinion is that Markdown is the way to go because everyone knows it and almost everyone uses it for documentation. It’s not about the number of features and ways to generate different highlights if those don’t meaningfully improve the quality of the documentation.
Most of the docs issues will always be about how to structure them, how the text in them is phrased etc. and that should outweigh everything else.
I’ve started writing some Nix language documentation in Markdown (nix-1p) and will add more over time as I integrate these into the Nixery documentation.
@tazjin I agree, but there’s no consensus on that.
As we’re using docbook at the moment, being the mighty god of semantic power, asciidoc seems a good compromise as it at least has good tooling while preserving most of the power.
Saying that I’m interested to provided good tooling outside Nix/Nixpkgs/NixOS · GitHub ecosystem and hope that one day there’s going to be recognition that it’s a better path going forward.
It’s good for short snippets but not for e.g. writing a manual.
I disagree. For example, most documentation at Google is written in Markdown (with a few minor extensions such as username highlighting) and we’ve got thousands of people contributing to it (also see this slide deck).
The format is perfectly fine for 99% of documentation needs, and for the 1% it’s easy enough to “shell out” to HTML. What’s much more important is that the documentation lives directly alongside the code it’s documenting, is easy to find and easy to edit. Docbook is none of those things, and I personally don’t see any Asciidoc features that have a significant enough advantage over Markdown to make it worthwhile.
Markdown is rather limited, you’re forced to “shell out” to HTML for anything not included in the spec, and it’s got annoying limitations like the inability to write an inline code block that contains a backtick (without shelling out to HTML at least, and writing the following is especially gross: <code>`<code>; it required using ` for the backtick). And if you want to actually generate anything other than HTML, this means you need to not only consume Markdown syntax but also be able to understand HTML syntax too.
Personally, I’m a fan of reStructuredText. It’s actually extensible in a standardized fashion and supports more stuff out of the box than Markdown. By default you need two backticks for inline code (e.g. ``foo``), where a single backtick introduces “interpreted” text, but we can configure the default role for interpreted text as literal, thus recovering the use of single-backticks. And yes, reStructuredText can encode an inline literal as an inline literal; the double-backtick syntax disables escapes, but interpreted text doesn’t, so I can write :literal:`foo\` bar` to get foo ` bar.
It’s also cherry-picked from the things I said. The major issues I currently see for Nix documentations are these:
Existing documentation is outdated
Existing documentation is hard to find (Google searches often lead me to the XML source of some DocBook page in nixpkgs, of which I can find the rendered version by quoting an exact string - but this is already too advanced for someone casually browsing)
Not enough people are contributing to documentation (a “drive-by” fix in the manual is not an easy feat)
The structure of the existing documentation is idiosyncratic because it’s neither consistently organised (it grew organically, which is fine up to a point) nor following a principled layout (such as mapping to file paths)
None of these are addressed by whatever syntax we have for writing backticks in inline code literals, but some of them are adversely affected by unfamiliar/niche tooling.
Unfamiliar tooling is a problem, but it’s not as bad as insufficient tooling. If we go with Markdown we’re really asking people to use Markdown plus a specific subset of HTML that we define somewhere with specific naming conventions for things.
Just off the top of my head, three features of our current manuals that Markdown does not support are:
Callouts, such as the blue “Note:” blocks used everywhere in our manual (example).
Cross-references. Markdown doesn’t even have any way of defining an anchor at all except by falling back to HTML. And even if we settle on a specific naming convention with HTML <a> tags to define anchors, everything breaks the moment we decide to split up the manual into a separate page for each section.
Our Markdown tooling at work uses hoextdown which supports all of those, but without modifying the base language (i.e. all standard Markdown is still valid).
Extensions are sort of inevitable for all of these formats at a certain project size, be it due to an actual need or just for convenience (e.g. we might want to be able to quickly link to the source of an attribute in the package set).
Definition lists, such as used in function documentation
Fun fact #1: When I wrote the tool that generates most of this function documentation I experimented with several other output formats, but decided not to pick the DocBook fight too much at that point. I still think that the format we ended up with isn’t the most readable.
Fun fact #2: If you look through some of the generated functions that didn’t have its documentation manually edited (such as most things in lib.strings, lib.lists) you will often see that people are actually writing Markdown syntax in the doc comments - it’s deeply engrained in many programmer brains!
What do you think of splitting the discussion between the types of documentation? Otherwise we’ll end-up in the same stalemate we had in previous documentation format discussion.
Maybe by splitting the discussion per documentation type, specific requirements might become clearer. We could even use different formats.
You may well ask: where do things like changelogs, contribution policies, and other information about the project fit into this scheme? The answer is that they do not - because they are, strictly speaking, project documentation rather than documentation of the software itself ,
They can simply be kept in appropriately-named sections alongside the other material - as long as they are not mixed up in it.
I think you’re onto something @tazjin. For technical documentation, the chances of me updating the documentation is much higher if it lives next to the code instead of somewhere in the nixos/doc/ folder. Did you publish the output anywhere to see if Markdown has all the features that we need?
For project documentation specifically we already have some files like the CONTRIBUTING.md file which are forced to be into Markdown due to GitHub. It also doesn’t need to be as precise as the reference documentation so it might be acceptable to loose some features. And it’s nice to get a preview for free directly in the PR.
GitHub can render reStructuredText files as well (use the extension .rst). Looking at github/markup it looks like it can also handle Textile, RDoc, Org mode, WikiCreole, MediaWiki, AsciiDoc, and perlpod.
It’s quite barebones right now and I’ve done nothing extra to deal with any specific formatting.
NixOS modules were the obvious low-hanging fruit because they’re in a well structured tree, it makes sense to have per-module documentation and several things follow easily (e.g. pick a doc comment from the top as the module description, pick a README.md in the module folder instead if it exists, …).
This doesn’t make as much sense for the package tree (not every package needs a documentation page), but I’ve got some ideas for how to experiment with switching docs in that tree on/off for stuff like the language-specific toolchains.
(Example of a module with a doc comment is my journaldriver module, there are some others around, too).
Regarding shelling out to HTML: That won’t work, because our docs have many targets, including man pages, epubs and pdfs. (brought to my attention by @grahamc on IRC)