Documentation format

[danbst]

Did anything concrete about moving from Docbook come about?

1. There are 3 candidates to move from Docbook: markdown, asciidoc and RST. The last one has least supporters, but some do know it.
2. There are actually 2 objections to Docbook which we want to address: rarely anybody knows it and GitHub doesn’t render XML nicely in Github web UI.
   a) For first one there exists https://docbook.rocks/, but it doesn’t mention all the gazillion of special tags used in Nix/pkgs/OS manuals (different terminals outputs, filepath, crosssection targets, …).
   b) For second one @edolstra objected that people who write docs don’t need GitHub rendering on Github website. Well we should convince him this is not true and we often end up making small doc changes in GH webpage and we want rendered preview right at hand.
   c) Eelco also pointed out that Emacs XML mode powers him to write docs fluently. But what are the options if we use something different from emacs?
3. Asciidoc(tor) is the most obvious choice to switch from Docbook:
   • it is designed from start as XML-less docbook
   • it has consistency check out of box
   • it has GitHub rendering, it has editor support
   • it is machine-processable (using Ruby)
4. I was convinced that Markdown may be of same rendering power as Asciidoc, but I don’t know the MD story about linters/checks/machine-processability. cc @tazjin
5. We also have cases when two formats (Docbook and Asciidoc/markdown) are mixed in single text blob! It is in description section of Nixos options: https://github.com/NixOS/nixpkgs/blob/08f68313a47a2093dc0f408a706b2c125bc59c95/nixos/modules/services/misc/zoneminder.nix#L73-L80. I think also that library function documentation (in function comments) is already markdown.

[peterhoeg]

There are 3 candidates to move from Docbook: markdown, asciidoc and RST.

Is it too late to throw org into the mix?

[danbst]

no, not too late

I think that as soon as someone produces a full-blown RFC/PR with clearly defined migration process and (autmated) migration is actually done for substantial part of documentation, it doesn’t matter what new format will be.

[peterhoeg]

There’s a WIP PR now with all the docbook auto-converted to org. So that’s one item down…

documentation: docbook to org [WIP]

[prusnak]

I think Markdown is the best option because it is the most popular. The only drawback is the awkward table syntax, but it’s something one can live with and I guess the Nix documentation does not have a lot of them. There are also nice and modern renderers such as mdBook.

Any other option is making contributions harder.

Sidenote: this forum also supports markdown by default.

[bhipple]

I’m also a fan of Markdown: in a head-to-head comparison, it’s inferior to more structured XML-based formats, but everyone knows it, it’s dead simple to edit, it renders nicely on GitHub, it renders nicely in git diff and is therefore easy to code review, it renders OK in raw vim/emacs for developers who haven’t setup fancier tools, and plenty of post-processing tools exist for taking Markdown and producing fancier HTML docs.

Aside from Google using markdown everywhere, all of Amazon’s public AWS documentation is in it [1], and most open source projects write docs in it – for example, all of the Rust docs [2]. Rust has great documentation, so if it’s good enough for them I’d say it’s good enough for us.

Every company I’ve ever worked for has used Markdown exclusively, not because it’s good but because it’s fast, efficient, and frictionless to edit – and at the end of the day what you care about in docs is the content, not having the most sophisticated layout features.

Another consideration is that the NixPkgs manual already has quite a bit of Markdown in it, so there’s prior art here [3]! Likewise, our RFCs are written in Markdown [4], as are comments on this forum.

All that said most of the modern doc formats are based on Markdown and relatively similar to it, so I think we’d be fine with anything that isn’t XML. I’d shy away from org-mode, though, because while it’s a fine format and I personally use it extensively for my notes, it fails the “frictionless” check above: it’s a less familiar format for many users and (more or less) requires using a well-configured emacs to use productively.

FWIW Sphinx and ReadTheDocs are really nice tools and generally work best on rST, though they support Markdown too. ReadTheDocs even has free builds and hosting of Sphinx docs triggered from webhooks on commits to open source projects [5]. This is probably the “best” option weighing both quality and efficiency – if someone’s willing to do the work to set it up.

[1] https://github.com/awsdocs/amazon-ec2-user-guide/tree/master/doc_source
[2] https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html
[3] https://github.com/NixOS/nixpkgs/tree/master/doc/languages-frameworks
[4] https://github.com/NixOS/rfcs
[5] https://readthedocs.org

[vyp]

Yeah markdown has a great advantage that it is already widely used. Meaning more people (as in including non-developers) would have a lower barrier of entry to contributing to documentation, so hopefully less people turn away from nix from being too hard to use/understand.

danbst, with regards to machine-processability, one of the reasons I think markdown has seen success in many websites (such as github, discourse etc) is the commonmark project. Before commonmark, markdown rendering was always inconsistent depending on the website due to the original 2004 markdown ‘spec’ leaving many ambiguities to be dealt with by the parsing implementer. These days for example github and commonmark have switched to commonmark or commonmark-based markdown:

1. https://github.blog/2017-03-14-a-formal-spec-for-github-markdown/
2. https://github.github.com/gfm/
3. https://meta.discourse.org/t/discourse-commonmark-migration-plans-confetti-ball-balloon/64234

So using a commonmark markdown implementation (and there’s probably one for your language) means compatible machine processability with github’s markdown rendering. (And in general just a very well thought out specification.)

[dpausp]

Markdown should be supported somehow, even if it’s not the primary markup language used for the documentation. Does the wiki support Markdown?

FWIW Sphinx and ReadTheDocs are really nice tools and generally work best on rST, though they support Markdown too. ReadTheDocs even has free builds and hosting of Sphinx docs triggered from webhooks on commits to open source projects [5]. This is probably the “best” option weighing both quality and efficiency – if someone’s willing to do the work to set it up.

I know Sphinx + ReadTheDocs and want to do it.

–

@zimbatm I think that the posts about markup language should be moved to a new topic.

[qyliss]

I’m skeptical of Markdown and other almost-plain-text markup languages.
I think that they encourage writers to care about how things look,
rather than what they mean, and make it a bit too easy to add
formatting. An example of this is overuse of code formatting, which you
see all the time on GitHub ("nixpkgs", " markdown", etc.). When you
have this several times within a paragraph, readability suffers
enormously.

When you have a format like Markdown that tries to look like plain text,
you often end up having to either choose to make the plain text readable
at the expense of the rendered output, or make the rendered output
correct and readable at the expense of the plain text. They are all
almost impossible to parse correctly (although for Markdown at least
there are of libraries that do a decent job), and they generally allow
inline HTML, which will mean rendering to any non-HTML output (roff
(used by man(1), PDF, GNU info, etc.) will always be second class. Rust
has nice documentation if you read it in a web browser, sure, but they
also have no way to read it in a terminal or a text editor. I don’t
think this is something we should seek to emulate.

Markdown is also not nearly as easy to write as it might appear to be at
first. The list syntax is objectively extremely confusing – I’d say about
half of the lists I come across on GitHub are incorrectly formatted
(split up into multiple lists, not nested where they’re supposed to be,
etc.) because the syntax makes no sense. Pop quiz: how many spaces do
you have to indent by to get a non-fenced code block to be part of
the same list item as the paragraph above it? If you guessed any number
less than eight, you would have just broken your formatting.

On the contrary, a well-defined language like DocBook or even HTML makes
you think about formatting and semantic meaning, and it’s difficult to
do it wrong. I think the quality of our documentation would decrease if
we switched to a format that wants to pretend to be plain text.

If renders of DocBook would be useful and are not provided by GitHub, I
think that’s an argument against GitHub more than anything else – with
a free web editor we could (presumably fairly easily) implement DocBook
rendering just like any other supported documentation format.

[lilyball]

The only drawback is the awkward table syntax

There are more drawbacks than that, which have already been documented in Documentation improvements

[mudrii]

AsciiDoc is a very good option even I do not have much experience with https://asciidoctor.org/docs/asciidoc-vs-markdown/

Markdown is most used format and can help add new contributors faster and easy to edit.

[domenkozar]

I think before we jump to conclusions which format is best, it’s important to think about what the outcomes should be.

The goals would be:

• extremely simple to do small improvements
• quick feedback loop
• no maintenance of the tooling from our side (we already have a big wagon to push by reinventing devops)

From the top of my mind, here are the desired properties of a good documentation tooling:

1. live preview of the documentation as I’m changing it
2. good editor integration (live preview, syntax highlighting)
3. extremely simple way to render documentation into multiple pages with a search
4. good conversion story from docbook
5. active community supporting the tooling infrastructure
6. good support of references, especially cross-project

The reason I chose asciidoc over markdown was since (4) is basically impossible as markdown support very basic markup and because of (5) - everyone at writethedocs conference said to avoid markdown. There’s also not a good story of (6) for markdown, which is needed for linking between nixos and nix documentation.

So we’re using https://antora.org/ for generating documentation across projects, which uses https://asciidoctor.org/ under the hood to render asciidoc.

[zimbatm]

I was starting to advocate to use Markdown, with a few extensions of our own, because everybody knows Markdown. But then realized that AsciiDoc is already very close to Markdown and has all the extensions that we need.

Today there was a thread on DocBook on HN and the consensus seem to be that AsciiDoc is the replacement. https://news.ycombinator.com/item?id=21436245

[domenkozar]

There’s now a draft RFC for improving the situation: https://github.com/NixOS/rfcs/pull/64