That enlightens me further to nix packaging design. I better understand the tradeoffs.
As a lazy developer I often forsake trust and reproducibility for a blob that works. That said, I’m often deploying blobs in an already trusted context 
.
2 Likes
mdbook has a significantly larger build closure than mmdoc:
$ nix path-info -Sh $(nix-build -A mmdoc.inputDerivation)
this derivation will be built:
/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv
building '/nix/store/4ahj4hqz6g8s6125969zwyrm3g7x3ljl-mmdoc-0.7.0.drv'...
/nix/store/xxd04wgymyslbmin9j27zgln76pj7056-mmdoc-0.7.0 370.4M
$ nix path-info -Sh $(nix-build -A mdbook.inputDerivation)
this derivation will be built:
/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv
these 2 paths will be fetched (16.67 MiB download, 18.59 MiB unpacked):
/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz
/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source
copying path '/nix/store/fn82kfxpbywwx674fkrqjaykdyh0gvrz-mdbook-0.4.9-vendor.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/ys2j5l132r6x0k7h05pwb8v63vhhpbxf-source' from 'https://cache.nixos.org'...
building '/nix/store/fjxx3zvws3rzqd57381bbn15gv6hyjnd-mdbook-0.4.9.drv'...
/nix/store/xwhqgz1dwm59vx9s7ix83nai3g29h5ca-mdbook-0.4.9 1.6G
3 Likes
We run the generate script outside-of-nix because if pandoc was part of the build closure for the manual, then any change to a one of the dependencies of pandoc, would also cause a change in the manual.
The manual is reproducible, isn’t it? An idea to avoid rebuilds due to pandoc could be to make it fixed output.
λ nix path-info /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 -Sh
/nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0 1.4G
Might be able to build mdbook with mrustc
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 273.7M
This is what I got:
{ lib, stdenv, fetchFromGitHub, rustPlatform, mrustc, CoreServices }:
let
buildRustPackage = rustPlatform.buildRustPackage.override {
rustc = mrustc;
};
in
buildRustPackage rec {
pname = "mdbook";
version = "0.4.12";
# original stuff here ...
}
λ nix path-info $(nix-build -A mrustc.inputDerivation) -Sh
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 273.7M
λ nix path-info $(nix-build -A mrustc.inputDerivation) -shr
/nix/store/11xpmmwy95396nkhih3qc3814lqhqb8f-libunistring-0.9.10 1.6M
/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh 7.3K
/nix/store/1mpxs3109cjrbhmi3q1vmvc0djz102pl-libidn2-2.3.2 254.7K
/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50 29.8M
/nix/store/jr35z7n8jbv9q89my50vhyndqd3y541i-attr-2.5.1 78.7K
/nix/store/krc4xirbvjnff8m62snqdbayg46z5l5b-acl-2.3.1 108.9K
/nix/store/xyn0240zrpprnspg3n0fi8c8aw5bq0mr-coreutils-8.32 1.8M
/nix/store/1nq62klcc9n2jv2ixaf77makkzdcghrh-findutils-4.8.0 1.7M
/nix/store/31pkw5yi08fj4l0glzvpf1cp4ywkxh86-gawk-5.1.0 2.1M
/nix/store/gya6j8d4mj03ija5f2xp393yz79nm5qx-ed-1.17 128.7K
/nix/store/347zp4r9a7gm5gk0gwijqw294nnyypcs-patch-2.7.6 226.5K
/nix/store/59jmzisg8fkm9c125fw384dqq1np602l-move-docs.sh 744.0
/nix/store/7fv9v6mnlkb4ddf9kz1snknbvbfbcbx0-gcc-10.3.0-lib 5.7M
/nix/store/c5giadpm71mvla6202mhz32x9criabna-zlib-1.2.11 121.4K
/nix/store/a4mmjm3bblxwp8h53bcfx3dly80ib0ba-binutils-2.35.1 30.8M
/nix/store/df49bj95bxw0qrqlnawr89d8s8w4kbna-expand-response-params 17.0K
/nix/store/mfz26azl9561jgd5n73nkszzp6qhsaal-glibc-2.33-50-bin 2.7M
/nix/store/nlf419m8sicv4n35asap5npgn7ym4z1r-linux-headers-5.14 5.6M
/nix/store/f1r94qx7a8m8iz7fxvlq655gbawql7xf-glibc-2.33-50-dev 2.1M
/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8 1.5M
/nix/store/5d5j1z9bg01wqxahihy2x5n22ykc32w8-binutils-wrapper-2.35.1 44.7K
/nix/store/8zxndz5ag0p6s526c2xyllhk1nrn4c3i-audit-tmpdir.sh 1.5K
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh 152.0
/nix/store/a2w3l1m75908yd05a0h40vnrzvxfd0gd-gcc-10.3.0 173.6M
/nix/store/a3v9gckl2l893yn817ssfdkw1hb27ki9-xz-5.2.5 478.3K
/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh 4.1K
/nix/store/c8n9kcdddp9np665xz6ri61b383nxvz8-move-systemd-user-units.sh 880.0
/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh 1.0K
/nix/store/csxpmhlzyhdkjir2clakkfp66yyflfvg-bzip2-1.0.6.0.2 79.2K
/nix/store/dy4ylp9439la4lq35ah2mj80fi87pk4w-gnused-4.8 725.1K
/nix/store/f2x98vk07px8916b9xid7jq6ky86sfmi-xz-5.2.5-bin 169.6K
/nix/store/fj3ywsx22xvjd4mly4323ikjcavyv91v-patchelf-0.13 198.7K
/nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh 904.0
/nix/store/gakfgapj20lv13vkcz6c38j8i9vz4ypi-bzip2-1.0.6.0.2-bin 68.4K
/nix/store/h54dzwd7rdh2jlcv91424csl6d0ccgjy-strip.sh 1.7K
/nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh 664.0
/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh 1008.0
/nix/store/m54bmrhj6fqz8nds5zcj97w9s9bckc9v-compress-man-pages.sh 1.1K
/nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh 1.5K
/nix/store/nwg8in201f7y6vdm787v3j84jjrn0ayw-gnutar-1.34 2.8M
/nix/store/qiapps0f4wfp87nm1nxklzp65vqyl872-pcre-8.44 510.3K
/nix/store/xxgddhdi57bbgd1yxza44plq6krjmiz1-gnugrep-3.6 819.4K
/nix/store/s5hkav7whndbfz0szshpb46h4idqdq9a-gcc-wrapper-10.3.0 46.8K
/nix/store/wlwcf1nw2b21m4gghj70hbg1v7x53ld8-reproducible-builds.sh 552.0
/nix/store/xgp0bgw4rpnbc3vr2qdsdbixp3zy4v1l-gnumake-4.3 1.4M
/nix/store/xwkxkx4bk005q35hsdhqbkbdv7g28cz5-diffutils-3.8 1.5M
/nix/store/ygzg6wzhgxf51ianb4zjvrzq4ilx9jd7-gzip-1.10 147.8K
/nix/store/qcq1y0nfxv8za7w6c682s93gk87r2xy1-stdenv-linux 41.7K
/nix/store/vfrkvba8m0kwg86042dr33ya5jx85mgp-source 4.8M
/nix/store/zzvn0d7p8nc3yq2n5y1i2sx4wpphf87l-zlib-1.2.11-dev 111.8K
/nix/store/svxn5i1vlws89pm7250dd8jsygnz9wnp-mrustc-0.9 2.2K
# ^ 51
# VS the original - squashed with wc since it's very long:
λ nix path-info $(nix-build -A mdbook.inputDerivation) -shr | wc -l
128
Either way I don’t mind what we use. I’m glad minimal closure sizes is a concern. 
That’s mostly off-topic for the discussion, but in general I don’t think we have truly good solutions for writing “larger than a comment” docs, as markdown format is limited. My long-term bet is to wait until asciidoctor publishes a spec and grows independent, embeddable implementations. As a language, asciidoctor is meaningfully better than markdown and I use it for smaller projects. As a toolchain and ecosystem, it needs to grow beyond “single primary ruby implementation”.
AsciiDoc is undergoing standardization
asciidoc-py is the original implementation from 2002, with asciidoctor appearing in 2013, and Eclipse Austen is an in-progress implementation for running on the JVM (although I don’t actually see any code in their gitlab instance repo)
What are the meaningful benefits to AsciiDoc over markdown?
From this overview from the asciidoctor documentation I’m not seeing much beyond anchors and cross references are properly supported.
Admonitions, sidebars, and includes might be nice.
Also they hammer home implementations and extensions are a bit random but I don’t think any of this is a concern for our documentation.
The negatives of switching would be converting all existing md documentation to asciidoc and limiting ourselves to 2 (or 3) implementations when markdown has a wide selection, much more usage, much better understanding across the maintainer-base
Either way once we first simplify the documentation story here any kind of converting the format should also be easier
The nix/nixpkgs/NixOS procect decided explicitly in favor of commonmark, so asciidoc won’t be considered.
4 Likes
Am I correct in thinking the only option that doesn’t have rustc in the build closure is mmdoc? Sphinx has started depending on it via Python’s cryptography package, IIUC:
> nix --extra-experimental-features nix-command why-depends -f '<nixpkgs>' --derivation sphinx rustc
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
/nix/store/diqa4kqpiv4zijzic8jplkyi2lrd8fpp-python3.9-sphinx-4.2.0.drv
└───/: …nx-4.2.0","","")],[("/nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv",…
→ /nix/store/29bmcfbr6jj2vwk354lyzia9c43pkb39-python3.9-requests-2.26.0.drv
└───/: ….0.0.drv",["out"]),("/nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv",["…
→ /nix/store/xd62gglgr1ggmr6x0ypqis7b04af3l5a-python3.9-trustme-0.9.0.drv
└───/: …hook.drv",["out"]),("/nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.dr…
→ /nix/store/h4lp7jnb3p0cy3ac1npd7q3hni4yp9m3-python3.9-cryptography-3.4.8.drv
└───/: …hy-3.4.8","","")],[("/nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv",["out"]),("/n…
→ /nix/store/1lmz6z65av2mfc10w3y9cd9jcc0jib6l-rustc-1.56.1.drv
2 Likes
we’ve tried to extend mmdoc with syntax extensions needed to render the current set of partially converted option docs, and we do not have the patience to deal with doing string processing in C. the experience using python and mistune 2 has been rather smooth though, and mistune itself seems to be pretty fast as well. at this point it seems that porting mmdoc to use python+mistune instead may be an excellent option since that would increase the system closure size by nothing at all, and would probably be more maintanable for the future as well.
2 Likes
I’m too young to know this pain but love this line and laughed aloud reading it.
That is good to know python and mistune 2 seem to be a good fit.
1 Like
Which syntax feature were you trying to add?
inline roles, eg {manpage}`nix.conf(5)`.
can we have a .info for opening on emacs as well? guix has that and it so lovely being able to use emacs for guix documentation
1 Like
@ryantm why is closure size an issue for nixpkgs doc but not for nix doc?
I like about mmdoc is that it generates single and multi page documentation.
I think a few useful things are not part of mmdoc:
footnotes
footnotes[^examplefn]
(partial) file inclusion
like in mdbook: mdBook-specific features - mdBook Documentation
mermaid diagrams
example: GitHub - NixOS/nix-book: Nix documentation – centralized community online learning resource for Nix
Definition List
term
: definition
(good for glossary)
Strikethrough
The world is flat.
to be able to write down mistakes without confusing tired readers
Task List
- parse markdown
- add task lists
and maybe less important:
Subscript: H~2~O
Superscript: X^2^
[^examplefn]: footnotes are helpful
to add additional information
and references.
pandoc implements two styles Pandoc - Pandoc User’s Guide
For nix and nixpkgs manuals, the build closure size is not as important. For the NixOS manual is it is, because we build the manual as part of realizing the system (nixos-rebuild). If the user adds their own NixOS module options, they show up in the manual. Currently the manual is built by default on all NixOS systems. If we chose to optionally build it, then the closure size would be less important.
mmdoc has Definition Lists. We should be careful about adding a bunch of extensions on top of CommonMark, since that moves us spiritually farther away from the documentation format RFC, and also makes it less likely we can easily port the manual to a different backend.
1 Like
agreed, but we should also not decrease the quality of documentation output. so far we’ve been trying to introduce only the minimal set of extensions necessary to keep current docs rendering the same (where the manpage notably makes more distinctions than the html manual). adding an extension for everything docbook offers won’t do, but a few (on top of what we already have) should be okay.
how about … string processing in Nix
at least yarn.lock can be parsed with nix
ryantm started a markdown compiler in C in
Find a way to reference NixOS options in CommonMark · Issue #127813 · NixOS/nixpkgs · GitHub
I did some of that recently for Nix documentation, and if you have a faible for writing string processing standard libraries first - sure, go ahead and upstream it somewhere while you’re at it. But other than that, it’s quite unexciting (some would say painful).
that would require a markdown parser written in nix. which, like, no
you’re of course free to attempt it, but we’d rather not waste our time like that (especially since the end result would require recursive nix or IFD to be useful)
3 Likes
we’ve started work on a(nother) dedicated renderer, nixos-render-docs, since none of the existing options seemed amenable to both use in nixos manual build closures and adding the extensions that define nixpkgs-flavored markdown. this tool should totally not be considered the be-all-end-all (if it gets merged, which hasn’t happened yet!), but we’re reasonably confident that it’ll be extensible enough to add e.g. add the great html rendering of mmdoc.
4 Likes