Just curious whether this is by accident or for a reason. Chapter 11. Fetchers in the Nixpkgs manual mentions
fetchurlfetchzipfetchpatchand has sections for
fetchsvnfetchgitfossilfetchcvsfetchhgfromGitHubfromGitLabfromGitilesfromBitbucketfromSavannahfromRepoOrCzfromSourcehutbut fetchTarball is only documented in 15.5. Built-in Functions in the Nix manual. This seems odd, but then again, there are probably a lot of other fetchers that I’ve never heard of.
The reason is that builtins.fetchTarball, just like builtins.fetchGit and builtins.fetchurl, is not a fetcher but a built-in function (primop). (Yes, it is confusing that Nix imports many builtins into scope automatically.)
Fetchers are actually just functions that produce fixed-output derivations. They utilize various builders (build scripts) to achieve that. For example, fetchurl fetcher is defined in pkgs/build-support/fetchurl/default.nix and its builder is next to it.
The builtins (e.g. builtins.fetchurl) are defined in Nix source code. One difference is that the built-in functions cause download during evaluation, which is bad for all sorts of reasons. Compare
nix-repl> builtins.fetchTarball { url = https://github.com/NixOS/nixpkgs/archive/nixos-20.09.tar.gz; sha256 = ""; }
[…starts downloading immediately, blocking your script, eventually returning a store path]
with
nix-repl> fetchzip { url = https://github.com/NixOS/nixpkgs/archive/nixos-20.09.tar.gz; sha256 = ""; }
«derivation /nix/store/x4r0sqgzkjm36lil5i4dkifgs5ipwkzr-source.drv»
The latter returns a derivation almost immediately, delaying the fetching (and extraction) to only when you build the derivation.
I agree that a brief mention of the fetching builtins (maybe just a link to the Nix manual) would make sense in the Nixpkgs manual.
Just for completeness, builtins.fetchTarball serves the same purpose as fetchzip – download an archive and then unpack it into store. Despite the name and docs, both support tarballs and zip files. (Nix uses libarchive, and fetchzip depends on unzip and has gnutar available from stdenv.)
To make the fetchurl even more confusing, there are few more objects with the same name:
fetchurl depends on stdenv, which in turn uses fetchurl to obtain sources of its various dependencies. For that reason, there is also a simpler fetchurlBoot function that is used to bootstrap stdenv.
fetchurlBoot function invokes fetchurl.nix file from Nix, which creates a fixed output derivation directly without a stdenv. To be able to do that, it uses a special value for the builder attribute – builtin:fetchurl (no relation to the primops). When nix sees it during build, it uses custom code instead of executing the builder script as it would normally.
What’s more confusing is that fetchurl also takes tarballs (e.g., see example in the Nix manual).
Anyway, appreciate the write-up, and it seems to be most comprehensive summary of fetchers. Thank you for taking the time!
Related threads:
Yeah, the difference is what is produced. fetchurl will just save the tarball as is into the store, whereas fetchzip/fetchTarball will unpack it and store its contents. Perhaps a more precise name would be fetchAndExtractArchive.
You can use unextracted archives as src in stdenv.mkDerivation thanks to generic builder’s unpackPhase. In fact, fetchzip also uses unpackFile internally (although, manually, not through phases, since fetchurl replaces the builder for one that does not run genericBuild).