I love the ideas and strategies proposed in RFC 72. (e.g. move to CommonMark for documentation).
I noticed that the nix unstable manual uses mdBook.
Was there consensus to move nixpkgs’s documentation to use mdBook as part of Conversion to CommonMark Docs · GitHub or after it?
There’s a script which generates the manual.
The workflow is found here.
why not just generate the manual as part of a derivation?
Because satisfying both docbook and mdbook requires some usage of pandoc, and pandoc is a massive build closure.
Was there consensus to move nixpkgs’s documentation to use mdBook as part of Conversion to CommonMark Docs · GitHub or after it?
Docbook introduced quite a bit of pain. Personally, I had a lot of pain with merge conflicts and release notes, as xml isn’t the most friendly manually edit. Not to mention docbook has some 300+ different element and attribute types; so it quite daunting to know how to leverage it. Also, docbook isn’t very popular, so it’s quite uncommon for people to have previous experience with it (myself included). However, markdown is almost universally used, much easier to edit, and quite light on ideas and concepts which need to be learned, it can be learned essentially in a few minutes.
Probably the final nail in the coffin was this PR #68020. After that point, I think the entire community was convinced that docbook should be moved away from. RFC 72 was just the facilitation of that transition.
Thanks for the link and added context.
I’m curious about the feasibility of using mdBook for consistency at some point.
So invoking pandoc to generate a derivation wasn’t a good fit? I’m probably not following what is meant here.
Docbook sounds nice for documentation that has stricter output requirements and full time experts at the wheel.
The ubiquity of markdown throughout GitHub and contributors likely exposure makes CommonMark a much better fit in my mind!
So invoking pandoc to generate a derivation wasn’t a good fit? I’m probably not following what is meant here.
We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual. Also, for consumers of isoImage.includeSystemBuildDependencies and similar “copy all build dependencies” utilities, it would require shipping ghc and quite a few other dependencies.
There are three different manuals we need to generate, Nix, NixOS and Nixpkgs.
CommonMark files -> mdbook -> HTML
CommonMark files -> ??? -> Man pages
CommonMark files -> Pandoc -> Docbook -> HTML
Docbook files -----^
Nix files -> nixdoc ----------^
CommonMark files -> Pandoc (at authoring time) -> Docbook files (in repo) -> Docbook -> HTML
??? -> Man pages
The reason that NixOS and Nixpkgs are not the same is because the NixOS manual is built whenever we build a new generation of NixOS, so any tools used by the NixOS manual end up in the NixOS build closure. We’d prefer to avoid having Pandoc in there. Ideally, we’d find a tool that has a very small footprint that runs very quickly to avoid bloating NixOS.
nixpkgs manual in CommonMark using sphinx+myst and jupyter-book by FRidh · Pull Request #105036 · NixOS/nixpkgs · GitHub @FRidh made this demo. I don’t know much about it but it is probably fine for nixpkgs and maybe too heavy for NixOS.
https://nix.dev/ also uses this.
GitHub - ryantm/mmdoc: Minimal Markdown Documentation is my work-in-progress tool for making CommonMark to HTML converter. doc: add nixpkgs manual split into multiple pages by ryantm · Pull Request #108063 · NixOS/nixpkgs · GitHub is the PR that integrates that for the nixpkgs manual. Current demo rendering of the nixpkgs manual is here Preface | nixpkgs
Probably suitable for nixpkgs and NixOS, but might not have as many features as people want for the nixpkgs manual and doesn’t have as much resources or development time behind it as the other options.
This is a good choice for nixpkgs considering it is already in use for nix. I suspect the Rust toolchain may be too heavy for NixOS, but I haven’t put in the work to verify this.
You mean bootstrapping the rust toolchain takes too long? I’d imagine a decently written program in rust to run quite quickly
No, I mean a rust tool chain requirement would impose more disk space usage than is desirable.
I haven’t assesed the speed of mdbook either.
Edit: removed “and slow” from my previous post.
This summary is a great introduction to how the Nix community does documentation!
I do not fully understand how packages are built within NixOS, but I do have experience compiling Rust programs for various targets. One of the nice things is that you do not need the Rust toolchain
installed to use a binary written in Rust.
For my benefit, does that not apply here because mdBook’s derives from the Rust toolchain?
You can also specify a “minimal” Rust toolchain which provides the bare minimum.
I also won’t have anywhere near as deep an understanding as you @ryantm but just shoving lsd (a rust tool) into a quick buildLayeredImage and the only dependencies for running are as follows:
/nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10
λ du --total --max-depth=0 \
/nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1 \
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50 \
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2 \
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10
3520 /nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1
32804 /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50
504 /nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2
1608 /nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10
38436 total
Am I missing something here?
You’re looking at the outputs, not the dependencies necessary to build lsd.
Try looking at:
$ nix-instantiate -A lsd
/nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv
$ nix-store -qR /nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv
We try very hard to limit the number and size of binary blobs we depend on for reproducibility and trust reasons.
That enlightens me further to nix packaging design. I better understand the tradeoffs.
As a lazy developer I often forsake trust and reproducibility for a blob that works. That said, I’m often deploying blobs in an already trusted context 
.
mdbook has a significantly larger build closure than mmdoc:
$ nix path-info -Sh $(nix-build -A mmdoc.inputDerivation)
this derivation will be built:
/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv
building '/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv'...
/nix/store/xxd04wgymyslbmin9j27zgln76pj7056-mmdoc-0.7.0 370.4M
$ nix path-info -Sh $(nix-build -A mdbook.inputDerivation)
this derivation will be built:
/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv
these 2 paths will be fetched (16.67 MiB download, 18.59 MiB unpacked):
/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz
/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source
copying path '/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source' from 'https://cache.nixos.org'...
building '/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv'...
/nix/store/xwhqgz1dwm59vx9s7ix83nai3g29h5ca-mdbook-0.4.9 1.6G
We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual.
The manual is reproducible, isn’t it? An idea to avoid rebuilds due to pandoc could be to make it fixed output.
λ nix path-info /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 -Sh
/nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 1.4G
Might be able to build mdbook with mrustc
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 273.7M
This is what I got:
{ lib, stdenv, fetchFromGitHub, rustPlatform, mrustc, CoreServices }:
let
buildRustPackage = rustPlatform.buildRustPackage.override {
rustc = mrustc;
};
in
buildRustPackage rec {
pname = "mdbook";
version = "0.4.12";
# original stuff here ...
}
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 273.7M
λ nix path-info $(nix-build -A mrustc.inputDerivation) -shr
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10 1.6M
/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh 7.3K
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2 254.7K
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50 29.8M
/nix/store/jr35z7n8jbv9q89my50vhyndqd3y541i-attr-2.5.1 78.7K
/nix/store/krc4xirbvjnff8m62snqdbayg46z5l5b-acl-2.3.1 108.9K
/nix/store/xyn0240zrpprnspg3n0fi8c8aw5bq0mr-coreutils-8.32 1.8M
/nix/store/1nq62klcc9n2jv2ixaf77makkzdcghrh-findutils-4.8.0 1.7M
/nix/store/31pkw5yi08fj4l0glzvpf1cp4ywkxh86-gawk-5.1.0 2.1M
/nix/store/gya6j8d4mj03ija5f2xp393yz79nm5qx-ed-1.17 128.7K
/nix/store/347zp4r9a7gm5gk0gwijqw294nnyypcs-patch-2.7.6 226.5K
/nix/store/59jmzisg8fkm9c125fw384dqq1np602l-move-docs.sh 744.0
/nix/store/7fv9v6mnlkb4ddf9kz1snknbvbfbcbx0-gcc-10.3.0-lib 5.7M
/nix/store/c5giadpm71mvla6202mhz32x9criabna-zlib-1.2.11 121.4K
/nix/store/a4mmjm3bblxwp8h53bcfx3dly80ib0ba-binutils-2.35.1 30.8M
/nix/store/df49bj95bxw0qrqlnawr89d8s8w4kbna-expand-response-params 17.0K
/nix/store/mfz26azl9561jgd5n73nkszzp6qhsaal-glibc-2.33-50-bin 2.7M
/nix/store/nlf419m8sicv4n35asap5npgn7ym4z1r-linux-headers-5.14 5.6M
/nix/store/f1r94qx7a8m8iz7fxvlq655gbawql7xf-glibc-2.33-50-dev 2.1M
/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8 1.5M
/nix/store/5d5j1z9bg01wqxahihy2x5n22ykc32w8-binutils-wrapper-2.35.1 44.7K
/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh 1.5K
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 152.0
/nix/store/a2w3l1m75908yd05a0h40vnrzvxfd0gd-gcc-10.3.0 173.6M
/nix/store/a3v9gckl2l893yn817ssfdkw1hb27ki9-xz-5.2.5 478.3K
/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh 4.1K
/nix/store/c8n9kcdddp9np665xz6ri61b383nxvz8-move-systemd-user-units.sh 880.0
/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh 1.0K
/nix/store/csxpmhlzyhdkjir2clakkfp66yyflfvg-bzip2-1.0.6.0.2 79.2K
/nix/store/dy4ylp9439la4lq35ah2mj80fi87pk4w-gnused-4.8 725.1K
/nix/store/f2x98vk07px8916b9xid7jq6ky86sfmi-xz-5.2.5-bin 169.6K
/nix/store/fj3ywsx22xvjd4mly4323ikjcavyv91v-patchelf-0.13 198.7K
/nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh 904.0
/nix/store/gakfgapj20lv13vkcz6c38j8i9vz4ypi-bzip2-1.0.6.0.2-bin 68.4K
/nix/store/h54dzwd7rdh2jlcv91424csl6d0ccgjy-strip.sh 1.7K
/nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh 664.0
/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh 1008.0
/nix/store/m54bmrhj6fqz8nds5zcj97w9s9bckc9v-compress-man-pages.sh 1.1K
/nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh 1.5K
/nix/store/nwg8in201f7y6vdm787v3j84jjrn0ayw-gnutar-1.34 2.8M
/nix/store/qiapps0f4wfp87nm1nxklzp65vqyl872-pcre-8.44 510.3K
/nix/store/xxgddhdi57bbgd1yxza44plq6krjmiz1-gnugrep-3.6 819.4K
/nix/store/s5hkav7whndbfz0szshpb46h4idqdq9a-gcc-wrapper-10.3.0 46.8K
/nix/store/wlwcf1nw2b21m4gghj70hbg1v7x53ld8-reproducible-builds.sh 552.0
/nix/store/xgp0bgw4rpnbc3vr2qdsdbixp3zy4v1l-gnumake-4.3 1.4M
/nix/store/xwkxkx4bk005q35hsdhqbkbdv7g28cz5-diffutils-3.8 1.5M
/nix/store/ygzg6wzhgxf51ianb4zjvrzq4ilx9jd7-gzip-1.10 147.8K
/nix/store/qcq1y0nfxv8za7w6c682s93gk87r2xy1-stdenv-linux 41.7K
/nix/store/vfrkvba8m0kwg86042dr33ya5jx85mgp-source 4.8M
/nix/store/zzvn0d7p8nc3yq2n5y1i2sx4wpphf87l-zlib-1.2.11-dev 111.8K
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 2.2K
# ^ 51
# VS the original - squashed with wc since it's very long:
λ nix path-info $(nix-build -A mdbook.inputDerivation) -shr | wc -l
128
Either way I don’t mind what we use. I’m glad minimal closure sizes is a concern. 
That’s mostly off-topic for the discussion, but in general I don’t think we have truly good solutions for writing “larger than a comment” docs, as markdown format is limited. My long-term bet is to wait until asciidoctor publishes a spec and grows independent, embeddable implementations. As a language, asciidoctor is meaningfully better than markdown and I use it for smaller projects. As a toolchain and ecosystem, it needs to grow beyond “single primary ruby implementation”.
AsciiDoc is undergoing standardization
asciidoc-py is the original implementation from 2002, with asciidoctor appearing in 2013, and Eclipse Austen is an in-progress implementation for running on the JVM (although I don’t actually see any code in their gitlab instance repo)
What are the meaningful benefits to AsciiDoc over markdown?
From this overview from the asciidoctor documentation I’m not seeing much beyond anchors and cross references are properly supported.
Admonitions, sidebars, and includes might be nice.
Also they hammer home implementations and extensions are a bit random but I don’t think any of this is a concern for our documentation.
The negatives of switching would be converting all existing md documentation to asciidoc and limiting ourselves to 2 (or 3) implementations when markdown has a wide selection, much more usage, much better understanding across the maintainer-base
Either way once we first simplify the documentation story here any kind of converting the format should also be easier
The nix/nixpkgs/NixOS procect decided explicitly in favor of commonmark, so asciidoc won’t be considered.
Am I correct in thinking the only option that doesn’t have rustc in the build closure is mmdoc? Sphinx has started depending on it via Python’s cryptography package, IIUC:
> nix --extra-experimental-features nix-command why-depends -f '<nixpkgs>' --derivation sphinx rustc
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
/nix/store/diqa4kqpiv4zijzic8jplkyi2lrd8fpp-python3.9-sphinx-4.2.0.drv
└───/: …nx-4.2.0","","")],[("/nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv",…
→ /nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv
└───/: ….0.0.drv",["out"]),("/nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv",["…
→ /nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv
└───/: …hook.drv",["out"]),("/nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.dr…
→ /nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.drv
└───/: …hy-3.4.8","","")],[("/nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv",["out"]),("/n…
→ /nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv
we’ve tried to extend mmdoc with syntax extensions needed to render the current set of partially converted option docs, and we do not have the patience to deal with doing string processing in C. the experience using python and mistune 2 has been rather smooth though, and mistune itself seems to be pretty fast as well. at this point it seems that porting mmdoc to use python+mistune instead may be an excellent option since that would increase the system closure size by nothing at all, and would probably be more maintanable for the future as well.
I’m too young to know this pain but love this line and laughed aloud reading it.
That is good to know python and mistune 2 seem to be a good fit.