Why basic builders are so damn complex?

[gytis-ivaskevicius]

In nixpkgs everything depends on mkDerivation including fundamental builders like pkgs.fetch* or pkgs.writers.* (or most of trivial-builders.nix which probably should be mostly deprecated in favor of pkgs.writers.*)

As far as I am aware none of these require any fancy features like cross-compilation support so I’d like to hear yall’s thoughts on replacing all that with a bare derivation which is as minimal as possible (executes cpp binary or fully featured little interpreter like jannet/lua/mruby?)

Expected result:

• Fewer build dependencies to download. Only glibc and tiny binary
• Faster nix evaluation (I have not benchmarked it, I expect rewrite of pkgs.fetchFromGithub to result in fairly large improvement)
• Lower nix memory usage (I have not benchmarked it)

Example:
Derivation: pkgs.writers.writeBash "abc" "abc"; results in store with a bunch of little scripts/patches and these directories:

dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 6baxv9mqxxssvgpxlipbcr0f6vbj2h52-stdenv-linux/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 474cxb3zyanpva8hsqnna60dcr20jvkv-source/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N k4bsma2cavm0b6blwip0fzdz3nmkvlh1-source/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N xkci0s3glinkrb3prpz0ah6b8fg51504-source/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N yj1wxm9hh8610iyzqnz75kvs6xl8j3my-source/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 5mm8iw0fdifw8jg7dphz70ly7087pv0j-patchelf-0.15.0/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N nwfyy893ql9ld0y249a5miwjb2kp15y8-findutils-4.9.0/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 7ls5xhx6kqpjgpg67kdd4pmbkhna4b6c-gcc-12.2.0-lib/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 05cd846a6pbhdijq28ga5piyk6d5mzvb-xgcc-12.2.0-libgcc/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N rrz2qan6b7h9b5y061dl09wclnkhja72-gcc-12.2.0-libgcc/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 0frsfhzwyjck7hkyskmqyb81wr77dhkq-gmp-with-cxx-6.2.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 9brwnhbqn9j6z1442hsq8vdsiv4dym9v-gnumake-4.4.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 125vby0dzdc1pxclxkqszppk43vwjs1v-acl-2.3.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N l8g5py3i39sq8afzi9vfmpw5igbqs84r-coreutils-9.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N m3rp9a2xp83a52xq4nqxnix05dl2xl1n-attr-2.5.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N qcr77drzki883k68niq9r41z3sa4rmqs-libunistring-1.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N rw2r8jqvbsmpq5kmgwvkv9pd833k9h3z-gawk-5.2.1/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 4wwylhblws9na4ghmvsia38kimxl43g4-gnugrep-3.11/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 4w0m22sx8yw9s1kw35f72cfak2qww8kh-gzip-1.12/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 37a5krk4a1a8vhl93q2bm9nbv8hymyii-zlib-1.2.13/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 1hd16q6hfk0sm9zqviy9g1iqcd445pyz-ed-1.19/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N h6rxqqi4m0vipgvv9czpmwvifdjawb14-bash-5.2-p15/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 0jf1raasph01s2cirhc5gh1hfw11cq9p-xz-5.4.3/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 1z5bqgk820x7xgn560kpk8znwh4z1irh-xz-5.4.3-bin/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N g1ajd0l91n1ryyw779wlfhf73imbh4cf-gnutar-1.34/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N x33pcmpsiimxhip52mwxbb5y77dhmb21-glibc-2.37-8/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N s6cj2h4rgrdqwm1ij3lb71flgnpwym8x-libidn2-2.3.4/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 77v35v9ww1bqpzywgrg88w98zhhajiw6-pcre2-10.42/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N h6y063nwq4bnzqmqsyq4aaygfyalkrxr-file-5.44/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N lmvrwmdbb5a36m7mvhhkg275r8b8l5hl-patch-2.7.6/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N ag4q7b7cqqjzhi7yd3y841b5kpmy7lvk-bzip2-1.0.8/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N gsz5ca5ccqhdj31xv4z0pmdykrk3c81w-bzip2-1.0.8-bin/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N 37gzj9sjgswm99bmgmr8r36ip19frfig-diffutils-3.9/
dr-xr-xr-x     - gytis 1970-01-01 03:00  -N figk1iqjicv30sa9qnvbbzdb81bzsh1c-gnused-4.9/

After removing *source directory takes up 340Mb

[qyliss]

There was a time when mkDerivation use in Nixpkgs was not universal, but over time everything was converted to mkDerivation. I don’t know the rationale, because it was before my time, but somebody like @Ericson2314 would probably remember.

One issue with not using mkDerivation would be that standard interfaces like overrideAttrs would no longer be available, which may be unwelcome.

@gytis-ivaskevicius

As far as I am aware none of these require any fancy features like cross-compilation support

I frequently use runCommand with nativeBuildInputs, and expect it to work correctly when cross compiling. Any writer function that runs a compiler (writeHaskell, writeRust, …) also needs cross compilation support, although those functions IME are not widely used, so whether they may not currently do cross compilation correctly. writeShellScript needs to be cross aware because it needs a build shell to perform the syntax checking, and so would any other script writer with syntax checking. Whether mkDerivation is used to have automatically working cross support, or whether you manually reimplement the cross stuff in derivation calls doesn’t really matter.

@gytis-ivaskevicius

Fewer build dependencies to download. Only glibc and tiny binary

This only applies if you’re only ever building these trivial derivations, which sounds unusual to me. You’d also still need coreutils, and for the runCommand family we should probably want to keep all of the stdenvNoCC inputs.

[gytis-ivaskevicius]

You are right, I did not consider cases where script needs to be compiled. I meant writing scripts with interpreters since that seems to be pretty easy

writeShellScript needs to be cross aware because it needs a build shell to perform the syntax checking

It is just host dependencies and you grab the shell from target dependencies (if I recall the naming correctly)

whether you manually reimplement the cross stuff in derivation calls doesn’t really matter

As mentioned before a goal would be to improve performance all around by replacing existing implementation with a much simpler one. I expect rewriting fetchFromGithub would improve nixpkgs evaluation performance quite a bit

This only applies if you’re only ever building these trivial derivations, which sounds unusual to me.

I often need just devshell and nix run some scripts

You’d also still need coreutils, and for the runCommand family we should probably want to keep all of the stdenvNoCC inputs.

I’d like as few deps as possible and as fast nix evaluation as possible. For build themselves may not even depend on coreutils if more featured runtime was to be used. Ideally, I’d see 1mb static binary

[emilytrau]

In minimal-bootstrap we don’t have access to mkDerivation and instead made our own minimal derivationWithMeta. If this is something others are interested in using it could be moved outside minimal-bootstrap https://github.com/NixOS/nixpkgs/blob/2218e80db0008eddb83741bab67db2dfe900b357/pkgs/os-specific/linux/minimal-bootstrap/utils.nix#L9