Any consensus on documentation generation tool for nixpkgs manual?

I love the ideas and strategies proposed in RFC 72. (e.g. move to CommonMark for documentation).

I noticed that the nix unstable manual uses mdBook.

Was there consensus to move nixpkgs’s documentation to use mdBook as part of Conversion to CommonMark Docs · GitHub or after it?

1 Like

There’s a script which generates the manual.

The workflow is found here.

why not just generate the manual as part of a derivation?

Because satisfying both docbook and mdbook requires some usage of pandoc, and pandoc is a massive build closure.

Was there consensus to move nixpkgs’s documentation to use mdBook as part of Conversion to CommonMark Docs · GitHub or after it?

Docbook introduced quite a bit of pain. Personally, I had a lot of pain with merge conflicts and release notes, as xml isn’t the most friendly manually edit. Not to mention docbook has some 300+ different element and attribute types; so it quite daunting to know how to leverage it. Also, docbook isn’t very popular, so it’s quite uncommon for people to have previous experience with it (myself included). However, markdown is almost universally used, much easier to edit, and quite light on ideas and concepts which need to be learned, it can be learned essentially in a few minutes.

Probably the final nail in the coffin was this PR #68020. After that point, I think the entire community was convinced that docbook should be moved away from. RFC 72 was just the facilitation of that transition.

3 Likes

Thanks for the link and added context.
I’m curious about the feasibility of using mdBook for consistency at some point.

So invoking pandoc to generate a derivation wasn’t a good fit? I’m probably not following what is meant here.

Docbook sounds nice for documentation that has stricter output requirements and full time experts at the wheel.
The ubiquity of markdown throughout GitHub and contributors likely exposure makes CommonMark a much better fit in my mind!

So invoking pandoc to generate a derivation wasn’t a good fit? I’m probably not following what is meant here.

We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual. Also, for consumers of isoImage.includeSystemBuildDependencies and similar “copy all build dependencies” utilities, it would require shipping ghc and quite a few other dependencies.

1 Like

There are three different manuals we need to generate, Nix, NixOS and Nixpkgs.

Current situation

Nix

CommonMark files -> mdbook -> HTML

CommonMark files -> ??? -> Man pages

Nixpkgs

CommonMark files -> Pandoc -> Docbook -> HTML
           Docbook files -----^
Nix files -> nixdoc ----------^

NixOS

CommonMark files -> Pandoc (at authoring time) -> Docbook files (in repo) -> Docbook -> HTML

??? -> Man pages

Considerations

The reason that NixOS and Nixpkgs are not the same is because the NixOS manual is built whenever we build a new generation of NixOS, so any tools used by the NixOS manual end up in the NixOS build closure. We’d prefer to avoid having Pandoc in there. Ideally, we’d find a tool that has a very small footprint that runs very quickly to avoid bloating NixOS.

Leading contenders for replacing Pandoc/Docbook

Sphinx

nixpkgs manual in CommonMark using sphinx+myst and jupyter-book by FRidh · Pull Request #105036 · NixOS/nixpkgs · GitHub @FRidh made this demo. I don’t know much about it but it is probably fine for nixpkgs and maybe too heavy for NixOS.

https://nix.dev/ also uses this.

mmdoc

GitHub - ryantm/mmdoc: Minimal Markdown Documentation is my work-in-progress tool for making CommonMark to HTML converter. doc: add CommonMark nixpkgs manual by ryantm · Pull Request #108063 · NixOS/nixpkgs · GitHub is the PR that integrates that for the nixpkgs manual. Current demo rendering of the nixpkgs manual is here Preface | nixpkgs

Probably suitable for nixpkgs and NixOS, but might not have as many features as people want for the nixpkgs manual and doesn’t have as much resources or development time behind it as the other options.

mdbook

This is a good choice for nixpkgs considering it is already in use for nix. I suspect the Rust toolchain may be too heavy for NixOS, but I haven’t put in the work to verify this.

3 Likes

You mean bootstrapping the rust toolchain takes too long? I’d imagine a decently written program in rust to run quite quickly

No, I mean a rust tool chain requirement would impose more disk space usage than is desirable.

I haven’t assesed the speed of mdbook either.

Edit: removed “and slow” from my previous post.

2 Likes

This summary is a great introduction to how the Nix community does documentation!

I do not fully understand how packages are built within NixOS, but I do have experience compiling Rust programs for various targets. One of the nice things is that you do not need the Rust toolchain
installed to use a binary written in Rust.

For my benefit, does that not apply here because mdBook's derives from the Rust toolchain?

You can also specify a “minimal” Rust toolchain which provides the bare minimum.

I also won’t have anywhere near as deep an understanding as you @ryantm but just shoving lsd (a rust tool) into a quick buildLayeredImage and the only dependencies for running are as follows:

/nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10


λ du --total --max-depth=0 \
  /nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1 \
  /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50 \
  /nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2 \
  /nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10
3520	/nix/store/kscs5v2qjxy7294na17bc258hb4v8s40-lsd-0.20.1
32804	/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50
504	    /nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2
1608	/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10
38436	total

Am I missing something here?

You’re looking at the outputs, not the dependencies necessary to build lsd.

Try looking at:

$ nix-instantiate -A lsd
/nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv

$ nix-store -qR /nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv

We try very hard to limit the number and size of binary blobs we depend on for reproducibility and trust reasons.

5 Likes

That enlightens me further to nix packaging design. I better understand the tradeoffs.
As a lazy developer I often forsake trust and reproducibility for a blob that works. That said, I’m often deploying blobs in an already trusted context :sweat_smile: .

2 Likes

mdbook has a significantly larger build closure than mmdoc:

$ nix path-info -Sh $(nix-build -A mmdoc.inputDerivation)
this derivation will be built:
  /nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv
building '/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv'...
/nix/store/xxd04wgymyslbmin9j27zgln76pj7056-mmdoc-0.7.0	 370.4M
$ nix path-info -Sh $(nix-build -A mdbook.inputDerivation)
this derivation will be built:
  /nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv
these 2 paths will be fetched (16.67 MiB download, 18.59 MiB unpacked):
  /nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz
  /nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source
copying path '/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source' from 'https://cache.nixos.org'...
building '/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv'...
/nix/store/xwhqgz1dwm59vx9s7ix83nai3g29h5ca-mdbook-0.4.9	   1.6G
3 Likes

We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual.

The manual is reproducible, isn’t it? An idea to avoid rebuilds due to pandoc could be to make it fixed output.

λ nix path-info /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 -Sh
/nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0	   1.4G

Might be able to build mdbook with mrustc

λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9	 273.7M

This is what I got:

{ lib, stdenv, fetchFromGitHub, rustPlatform, mrustc, CoreServices }:

let
  buildRustPackage = rustPlatform.buildRustPackage.override {
    rustc = mrustc;
  };
in
buildRustPackage rec {
  pname = "mdbook";
  version = "0.4.12";

  # original stuff here ...
}
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9	 273.7M

λ nix path-info $(nix-build -A mrustc.inputDerivation) -shr
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10                    1.6M
/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh                    7.3K
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2                        254.7K
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50                         29.8M
/nix/store/jr35z7n8jbv9q89my50vhyndqd3y541i-attr-2.5.1                            78.7K
/nix/store/krc4xirbvjnff8m62snqdbayg46z5l5b-acl-2.3.1                            108.9K
/nix/store/xyn0240zrpprnspg3n0fi8c8aw5bq0mr-coreutils-8.32                         1.8M
/nix/store/1nq62klcc9n2jv2ixaf77makkzdcghrh-findutils-4.8.0                        1.7M
/nix/store/31pkw5yi08fj4l0glzvpf1cp4ywkxh86-gawk-5.1.0                             2.1M
/nix/store/gya6j8d4mj03ija5f2xp393yz79nm5qx-ed-1.17                              128.7K
/nix/store/347zp4r9a7gm5gk0gwijqw294nnyypcs-patch-2.7.6                          226.5K
/nix/store/59jmzisg8fkm9c125fw384dqq1np602l-move-docs.sh                          744.0
/nix/store/7fv9v6mnlkb4ddf9kz1snknbvbfbcbx0-gcc-10.3.0-lib                         5.7M
/nix/store/c5giadpm71mvla6202mhz32x9criabna-zlib-1.2.11                          121.4K
/nix/store/a4mmjm3bblxwp8h53bcfx3dly80ib0ba-binutils-2.35.1                       30.8M
/nix/store/df49bj95bxw0qrqlnawr89d8s8w4kbna-expand-response-params                17.0K
/nix/store/mfz26azl9561jgd5n73nkszzp6qhsaal-glibc-2.33-50-bin                      2.7M
/nix/store/nlf419m8sicv4n35asap5npgn7ym4z1r-linux-headers-5.14                     5.6M
/nix/store/f1r94qx7a8m8iz7fxvlq655gbawql7xf-glibc-2.33-50-dev                      2.1M
/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8                            1.5M
/nix/store/5d5j1z9bg01wqxahihy2x5n22ykc32w8-binutils-wrapper-2.35.1               44.7K
/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh                        1.5K
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh                    152.0
/nix/store/a2w3l1m75908yd05a0h40vnrzvxfd0gd-gcc-10.3.0                           173.6M
/nix/store/a3v9gckl2l893yn817ssfdkw1hb27ki9-xz-5.2.5                             478.3K
/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh                      4.1K
/nix/store/c8n9kcdddp9np665xz6ri61b383nxvz8-move-systemd-user-units.sh            880.0
/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh                 1.0K
/nix/store/csxpmhlzyhdkjir2clakkfp66yyflfvg-bzip2-1.0.6.0.2                       79.2K
/nix/store/dy4ylp9439la4lq35ah2mj80fi87pk4w-gnused-4.8                           725.1K
/nix/store/f2x98vk07px8916b9xid7jq6ky86sfmi-xz-5.2.5-bin                         169.6K
/nix/store/fj3ywsx22xvjd4mly4323ikjcavyv91v-patchelf-0.13                        198.7K
/nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh                         904.0
/nix/store/gakfgapj20lv13vkcz6c38j8i9vz4ypi-bzip2-1.0.6.0.2-bin                   68.4K
/nix/store/h54dzwd7rdh2jlcv91424csl6d0ccgjy-strip.sh                               1.7K
/nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh                          664.0
/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh            1008.0
/nix/store/m54bmrhj6fqz8nds5zcj97w9s9bckc9v-compress-man-pages.sh                  1.1K
/nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh     1.5K
/nix/store/nwg8in201f7y6vdm787v3j84jjrn0ayw-gnutar-1.34                            2.8M
/nix/store/qiapps0f4wfp87nm1nxklzp65vqyl872-pcre-8.44                            510.3K
/nix/store/xxgddhdi57bbgd1yxza44plq6krjmiz1-gnugrep-3.6                          819.4K
/nix/store/s5hkav7whndbfz0szshpb46h4idqdq9a-gcc-wrapper-10.3.0                    46.8K
/nix/store/wlwcf1nw2b21m4gghj70hbg1v7x53ld8-reproducible-builds.sh                552.0
/nix/store/xgp0bgw4rpnbc3vr2qdsdbixp3zy4v1l-gnumake-4.3                            1.4M
/nix/store/xwkxkx4bk005q35hsdhqbkbdv7g28cz5-diffutils-3.8                          1.5M
/nix/store/ygzg6wzhgxf51ianb4zjvrzq4ilx9jd7-gzip-1.10                            147.8K
/nix/store/qcq1y0nfxv8za7w6c682s93gk87r2xy1-stdenv-linux                          41.7K
/nix/store/vfrkvba8m0kwg86042dr33ya5jx85mgp-source                                 4.8M
/nix/store/zzvn0d7p8nc3yq2n5y1i2sx4wpphf87l-zlib-1.2.11-dev                      111.8K
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9                             2.2K

# ^ 51

# VS the original - squashed with wc since it's very long:
λ nix path-info $(nix-build -A mdbook.inputDerivation) -shr | wc -l
128

Either way I don’t mind what we use. I’m glad minimal closure sizes is a concern. :+1:

That’s mostly off-topic for the discussion, but in general I don’t think we have truly good solutions for writing “larger than a comment” docs, as markdown format is limited. My long-term bet is to wait until asciidoctor publishes a spec and grows independent, embeddable implementations. As a language, asciidoctor is meaningfully better than markdown and I use it for smaller projects. As a toolchain and ecosystem, it needs to grow beyond “single primary ruby implementation”.

AsciiDoc is undergoing standardization

asciidoc-py is the original implementation from 2002, with asciidoctor appearing in 2013, and Eclipse Austen is an in-progress implementation for running on the JVM (although I don’t actually see any code in their gitlab instance repo)


What are the meaningful benefits to AsciiDoc over markdown?

From this overview from the asciidoctor documentation I’m not seeing much beyond anchors and cross references are properly supported.
Admonitions, sidebars, and includes might be nice.
Also they hammer home implementations and extensions are a bit random but I don’t think any of this is a concern for our documentation.

https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-markdown/

The negatives of switching would be converting all existing md documentation to asciidoc and limiting ourselves to 2 (or 3) implementations when markdown has a wide selection, much more usage, much better understanding across the maintainer-base


Either way once we first simplify the documentation story here any kind of converting the format should also be easier

The nix/nixpkgs/NixOS procect decided explicitly in favor of commonmark, so asciidoc won’t be considered.

3 Likes

Am I correct in thinking the only option that doesn’t have rustc in the build closure is mmdoc? Sphinx has started depending on it via Python’s cryptography package, IIUC:

> nix --extra-experimental-features nix-command why-depends -f '<nixpkgs>' --derivation sphinx rustc
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
/nix/store/diqa4kqpiv4zijzic8jplkyi2lrd8fpp-python3.9-sphinx-4.2.0.drv
└───/: …nx-4.2.0","","")],[("/nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv",…
    → /nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv
    └───/: ….0.0.drv",["out"]),("/nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv",["…
        → /nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv
        └───/: …hook.drv",["out"]),("/nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.dr…
            → /nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.drv
            └───/: …hy-3.4.8","","")],[("/nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv",["out"]),("/n…
                → /nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv
2 Likes
Hosted by Flying Circus.