Any consensus on documentation generation tool for nixpkgs manual?

You’re looking at the outputs, not the dependencies necessary to build lsd.

Try looking at:

$ nix-instantiate -A lsd
/nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv

$ nix-store -qR /nix/store/yv9b1bvinc1mjsr26v4nd4msw0dlchiv-lsd-0.20.1.drv

We try very hard to limit the number and size of binary blobs we depend on for reproducibility and trust reasons.

6 Likes

That enlightens me further to nix packaging design. I better understand the tradeoffs.
As a lazy developer I often forsake trust and reproducibility for a blob that works. That said, I’m often deploying blobs in an already trusted context :sweat_smile: .

2 Likes

mdbook has a significantly larger build closure than mmdoc:

$ nix path-info -Sh $(nix-build -A mmdoc.inputDerivation)
this derivation will be built:
  /nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv
building '/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv'...
/nix/store/xxd04wgymyslbmin9j27zgln76pj7056-mmdoc-0.7.0	 370.4M
$ nix path-info -Sh $(nix-build -A mdbook.inputDerivation)
this derivation will be built:
  /nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv
these 2 paths will be fetched (16.67 MiB download, 18.59 MiB unpacked):
  /nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz
  /nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source
copying path '/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source' from 'https://cache.nixos.org'...
building '/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv'...
/nix/store/xwhqgz1dwm59vx9s7ix83nai3g29h5ca-mdbook-0.4.9	   1.6G
3 Likes

We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual.

The manual is reproducible, isn’t it? An idea to avoid rebuilds due to pandoc could be to make it fixed output.

λ nix path-info /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 -Sh
/nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0	   1.4G

Might be able to build mdbook with mrustc

λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9	 273.7M

This is what I got:

{ lib, stdenv, fetchFromGitHub, rustPlatform, mrustc, CoreServices }:

let
  buildRustPackage = rustPlatform.buildRustPackage.override {
    rustc = mrustc;
  };
in
buildRustPackage rec {
  pname = "mdbook";
  version = "0.4.12";

  # original stuff here ...
}
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9	 273.7M

λ nix path-info $(nix-build -A mrustc.inputDerivation) -shr
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10                    1.6M
/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh                    7.3K
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2                        254.7K
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50                         29.8M
/nix/store/jr35z7n8jbv9q89my50vhyndqd3y541i-attr-2.5.1                            78.7K
/nix/store/krc4xirbvjnff8m62snqdbayg46z5l5b-acl-2.3.1                            108.9K
/nix/store/xyn0240zrpprnspg3n0fi8c8aw5bq0mr-coreutils-8.32                         1.8M
/nix/store/1nq62klcc9n2jv2ixaf77makkzdcghrh-findutils-4.8.0                        1.7M
/nix/store/31pkw5yi08fj4l0glzvpf1cp4ywkxh86-gawk-5.1.0                             2.1M
/nix/store/gya6j8d4mj03ija5f2xp393yz79nm5qx-ed-1.17                              128.7K
/nix/store/347zp4r9a7gm5gk0gwijqw294nnyypcs-patch-2.7.6                          226.5K
/nix/store/59jmzisg8fkm9c125fw384dqq1np602l-move-docs.sh                          744.0
/nix/store/7fv9v6mnlkb4ddf9kz1snknbvbfbcbx0-gcc-10.3.0-lib                         5.7M
/nix/store/c5giadpm71mvla6202mhz32x9criabna-zlib-1.2.11                          121.4K
/nix/store/a4mmjm3bblxwp8h53bcfx3dly80ib0ba-binutils-2.35.1                       30.8M
/nix/store/df49bj95bxw0qrqlnawr89d8s8w4kbna-expand-response-params                17.0K
/nix/store/mfz26azl9561jgd5n73nkszzp6qhsaal-glibc-2.33-50-bin                      2.7M
/nix/store/nlf419m8sicv4n35asap5npgn7ym4z1r-linux-headers-5.14                     5.6M
/nix/store/f1r94qx7a8m8iz7fxvlq655gbawql7xf-glibc-2.33-50-dev                      2.1M
/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8                            1.5M
/nix/store/5d5j1z9bg01wqxahihy2x5n22ykc32w8-binutils-wrapper-2.35.1               44.7K
/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh                        1.5K
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh                    152.0
/nix/store/a2w3l1m75908yd05a0h40vnrzvxfd0gd-gcc-10.3.0                           173.6M
/nix/store/a3v9gckl2l893yn817ssfdkw1hb27ki9-xz-5.2.5                             478.3K
/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh                      4.1K
/nix/store/c8n9kcdddp9np665xz6ri61b383nxvz8-move-systemd-user-units.sh            880.0
/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh                 1.0K
/nix/store/csxpmhlzyhdkjir2clakkfp66yyflfvg-bzip2-1.0.6.0.2                       79.2K
/nix/store/dy4ylp9439la4lq35ah2mj80fi87pk4w-gnused-4.8                           725.1K
/nix/store/f2x98vk07px8916b9xid7jq6ky86sfmi-xz-5.2.5-bin                         169.6K
/nix/store/fj3ywsx22xvjd4mly4323ikjcavyv91v-patchelf-0.13                        198.7K
/nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh                         904.0
/nix/store/gakfgapj20lv13vkcz6c38j8i9vz4ypi-bzip2-1.0.6.0.2-bin                   68.4K
/nix/store/h54dzwd7rdh2jlcv91424csl6d0ccgjy-strip.sh                               1.7K
/nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh                          664.0
/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh            1008.0
/nix/store/m54bmrhj6fqz8nds5zcj97w9s9bckc9v-compress-man-pages.sh                  1.1K
/nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh     1.5K
/nix/store/nwg8in201f7y6vdm787v3j84jjrn0ayw-gnutar-1.34                            2.8M
/nix/store/qiapps0f4wfp87nm1nxklzp65vqyl872-pcre-8.44                            510.3K
/nix/store/xxgddhdi57bbgd1yxza44plq6krjmiz1-gnugrep-3.6                          819.4K
/nix/store/s5hkav7whndbfz0szshpb46h4idqdq9a-gcc-wrapper-10.3.0                    46.8K
/nix/store/wlwcf1nw2b21m4gghj70hbg1v7x53ld8-reproducible-builds.sh                552.0
/nix/store/xgp0bgw4rpnbc3vr2qdsdbixp3zy4v1l-gnumake-4.3                            1.4M
/nix/store/xwkxkx4bk005q35hsdhqbkbdv7g28cz5-diffutils-3.8                          1.5M
/nix/store/ygzg6wzhgxf51ianb4zjvrzq4ilx9jd7-gzip-1.10                            147.8K
/nix/store/qcq1y0nfxv8za7w6c682s93gk87r2xy1-stdenv-linux                          41.7K
/nix/store/vfrkvba8m0kwg86042dr33ya5jx85mgp-source                                 4.8M
/nix/store/zzvn0d7p8nc3yq2n5y1i2sx4wpphf87l-zlib-1.2.11-dev                      111.8K
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9                             2.2K

# ^ 51

# VS the original - squashed with wc since it's very long:
λ nix path-info $(nix-build -A mdbook.inputDerivation) -shr | wc -l
128

Either way I don’t mind what we use. I’m glad minimal closure sizes is a concern. :+1:

That’s mostly off-topic for the discussion, but in general I don’t think we have truly good solutions for writing “larger than a comment” docs, as markdown format is limited. My long-term bet is to wait until asciidoctor publishes a spec and grows independent, embeddable implementations. As a language, asciidoctor is meaningfully better than markdown and I use it for smaller projects. As a toolchain and ecosystem, it needs to grow beyond “single primary ruby implementation”.

AsciiDoc is undergoing standardization

asciidoc-py is the original implementation from 2002, with asciidoctor appearing in 2013, and Eclipse Austen is an in-progress implementation for running on the JVM (although I don’t actually see any code in their gitlab instance repo)


What are the meaningful benefits to AsciiDoc over markdown?

From this overview from the asciidoctor documentation I’m not seeing much beyond anchors and cross references are properly supported.
Admonitions, sidebars, and includes might be nice.
Also they hammer home implementations and extensions are a bit random but I don’t think any of this is a concern for our documentation.

https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-markdown/

The negatives of switching would be converting all existing md documentation to asciidoc and limiting ourselves to 2 (or 3) implementations when markdown has a wide selection, much more usage, much better understanding across the maintainer-base


Either way once we first simplify the documentation story here any kind of converting the format should also be easier

The nix/nixpkgs/NixOS procect decided explicitly in favor of commonmark, so asciidoc won’t be considered.

4 Likes

Am I correct in thinking the only option that doesn’t have rustc in the build closure is mmdoc? Sphinx has started depending on it via Python’s cryptography package, IIUC:

> nix --extra-experimental-features nix-command why-depends -f '<nixpkgs>' --derivation sphinx rustc
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
/nix/store/diqa4kqpiv4zijzic8jplkyi2lrd8fpp-python3.9-sphinx-4.2.0.drv
└───/: …nx-4.2.0","","")],[("/nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv",…
    → /nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv
    └───/: ….0.0.drv",["out"]),("/nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv",["…
        → /nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv
        └───/: …hook.drv",["out"]),("/nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.dr…
            → /nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.drv
            └───/: …hy-3.4.8","","")],[("/nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv",["out"]),("/n…
                → /nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv
2 Likes

we’ve tried to extend mmdoc with syntax extensions needed to render the current set of partially converted option docs, and we do not have the patience to deal with doing string processing in C. the experience using python and mistune 2 has been rather smooth though, and mistune itself seems to be pretty fast as well. at this point it seems that porting mmdoc to use python+mistune instead may be an excellent option since that would increase the system closure size by nothing at all, and would probably be more maintanable for the future as well.

2 Likes

I’m too young to know this pain but love this line and laughed aloud reading it.

That is good to know python and mistune 2 seem to be a good fit.

Which syntax feature were you trying to add?

inline roles, eg {manpage}`nix.conf(5)`.

can we have a .info for opening on emacs as well? guix has that and it so lovely being able to use emacs for guix documentation

1 Like

@ryantm why is closure size an issue for nixpkgs doc but not for nix doc?

I like about mmdoc is that it generates single and multi page documentation.
I think a few useful things are not part of mmdoc:

footnotes

footnotes[^examplefn]

(partial) file inclusion

like in mdbook: mdBook-specific features - mdBook Documentation

mermaid diagrams

example: https://github.com/nixOS/nix-book#a-vision-for-the-journey-into-nix-nixpkgs-and-nixos

Definition List

term
: definition
(good for glossary)

Strikethrough

The world is flat.
to be able to write down mistakes without confusing tired readers

Task List

  • parse markdown
  • add task lists

and maybe less important:
Subscript: H~2~O
Superscript: X^2^

[^examplefn]: footnotes are helpful
to add additional information
and references.
pandoc implements two styles Pandoc - Pandoc User’s Guide

For nix and nixpkgs manuals, the build closure size is not as important. For the NixOS manual is it is, because we build the manual as part of realizing the system (nixos-rebuild). If the user adds their own NixOS module options, they show up in the manual. Currently the manual is built by default on all NixOS systems. If we chose to optionally build it, then the closure size would be less important.

mmdoc has Definition Lists. We should be careful about adding a bunch of extensions on top of CommonMark, since that moves us spiritually farther away from the documentation format RFC, and also makes it less likely we can easily port the manual to a different backend.

1 Like

agreed, but we should also not decrease the quality of documentation output. so far we’ve been trying to introduce only the minimal set of extensions necessary to keep current docs rendering the same (where the manpage notably makes more distinctions than the html manual). adding an extension for everything docbook offers won’t do, but a few (on top of what we already have) should be okay.

how about … string processing in Nix

at least yarn.lock can be parsed with nix

ryantm started a markdown compiler in C in
Find a way to reference NixOS options in CommonMark · Issue #127813 · NixOS/nixpkgs · GitHub

I did some of that recently for Nix documentation, and if you have a faible for writing string processing standard libraries first - sure, go ahead and upstream it somewhere while you’re at it. But other than that, it’s quite unexciting (some would say painful).

that would require a markdown parser written in nix. which, like, no

you’re of course free to attempt it, but we’d rather not waste our time like that (especially since the end result would require recursive nix or IFD to be useful)

2 Likes
Hosted by Flying Circus.