mdbook has a significantly larger build closure than mmdoc:
$ nix path-info -Sh $(nix-build -A mmdoc.inputDerivation)
this derivation will be built:
/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv
building '/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv'...
/nix/store/xxd04wgymyslbmin9j27zgln76pj7056-mmdoc-0.7.0 370.4M
$ nix path-info -Sh $(nix-build -A mdbook.inputDerivation)
this derivation will be built:
/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv
these 2 paths will be fetched (16.67 MiB download, 18.59 MiB unpacked):
/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz
/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source
copying path '/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source' from 'https://cache.nixos.org'...
building '/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv'...
/nix/store/xwhqgz1dwm59vx9s7ix83nai3g29h5ca-mdbook-0.4.9 1.6G
We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual.
The manual is reproducible, isn’t it? An idea to avoid rebuilds due to pandoc could be to make it fixed output.
That’s mostly off-topic for the discussion, but in general I don’t think we have truly good solutions for writing “larger than a comment” docs, as markdown format is limited. My long-term bet is to wait until asciidoctor publishes a spec and grows independent, embeddable implementations. As a language, asciidoctor is meaningfully better than markdown and I use it for smaller projects. As a toolchain and ecosystem, it needs to grow beyond “single primary ruby implementation”.
asciidoc-py is the original implementation from 2002, with asciidoctor appearing in 2013, and Eclipse Austen is an in-progress implementation for running on the JVM (although I don’t actually see any code in their gitlab instance repo)
What are the meaningful benefits to AsciiDoc over markdown?
From this overview from the asciidoctor documentation I’m not seeing much beyond anchors and cross references are properly supported.
Admonitions, sidebars, and includes might be nice.
Also they hammer home implementations and extensions are a bit random but I don’t think any of this is a concern for our documentation.
The negatives of switching would be converting all existing md documentation to asciidoc and limiting ourselves to 2 (or 3) implementations when markdown has a wide selection, much more usage, much better understanding across the maintainer-base
Either way once we first simplify the documentation story here any kind of converting the format should also be easier
Am I correct in thinking the only option that doesn’t have rustc in the build closure is mmdoc? Sphinx has started depending on it via Python’s cryptography package, IIUC:
we’ve tried to extend mmdoc with syntax extensions needed to render the current set of partially converted option docs, and we do not have the patience to deal with doing string processing in C. the experience using python and mistune 2 has been rather smooth though, and mistune itself seems to be pretty fast as well. at this point it seems that porting mmdoc to use python+mistune instead may be an excellent option since that would increase the system closure size by nothing at all, and would probably be more maintanable for the future as well.
For nix and nixpkgs manuals, the build closure size is not as important. For the NixOS manual is it is, because we build the manual as part of realizing the system (nixos-rebuild). If the user adds their own NixOS module options, they show up in the manual. Currently the manual is built by default on all NixOS systems. If we chose to optionally build it, then the closure size would be less important.
mmdoc has Definition Lists. We should be careful about adding a bunch of extensions on top of CommonMark, since that moves us spiritually farther away from the documentation format RFC, and also makes it less likely we can easily port the manual to a different backend.
agreed, but we should also not decrease the quality of documentation output. so far we’ve been trying to introduce only the minimal set of extensions necessary to keep current docs rendering the same (where the manpage notably makes more distinctions than the html manual). adding an extension for everything docbook offers won’t do, but a few (on top of what we already have) should be okay.
I did some of that recently for Nix documentation, and if you have a faible for writing string processing standard libraries first - sure, go ahead and upstream it somewhere while you’re at it. But other than that, it’s quite unexciting (some would say painful).
that would require a markdown parser written in nix. which, like, no
you’re of course free to attempt it, but we’d rather not waste our time like that (especially since the end result would require recursive nix or IFD to be useful)
we’ve started work on a(nother) dedicated renderer, nixos-render-docs, since none of the existing options seemed amenable to both use in nixos manual build closures and adding the extensions that define nixpkgs-flavored markdown. this tool should totally not be considered the be-all-end-all (if it gets merged, which hasn’t happened yet!), but we’re reasonably confident that it’ll be extensible enough to add e.g. add the great html rendering of mmdoc.