Deeply Understanding the Nixpkgs repo with Nix-trace

I’m hoping this will be helpful for creating future core-contributors.
Even though I’m not new to nix in the slightest, understanding the bootstrapping sequence has been challenging to say the least.

So, is there some way to:
- build cowsay
- force a rebuild of cowsay’s depenedencies recursively (its okay if it takes 48 hours)
- have the console print out a list of every .nix file that was accessed (in order) while that happens

I’m fine if I need to hack stuff, like build a custom nix binary, in order to get this printout. I also won’t be bothered by any weird ordering thats caused/not-caused by lazy evaluation. I’m mostly interested in what code is used and what isn’t; like which packages are needed for something like fetchFromGithub to work, etc.

Attempts:

  • I’ve looked at lib/debug.nix, didn’t see anything obvious to do the trick.
  • Didn’t see any kind of debug=vomit type of option inside of nix/nix repl/nix-env help menus.
  • As a last resort I’ve considered using the OS to try and detect file-reads and, while possible, it also wasn’t straightforward and it would be nice to do this in a not-os-specific way since bootstrapping is different for each OS.
4 Likes

Try something like:

nix build -vvvvv --substituters '' nixpkgs#cowsay

6 Likes

Edit: using that command, I present nixpkgs-skeleton; a repo which contains only absolutely-essential files for building cowsay (308 files, 152 directories :sweat_smile:). Hopefully people will find it useful to explore!

And for those who want a quick look at the output, here’s the begining of the file-access for MacOS M1
(nixpkgs commit cfb43ad7b941d9c3606fb35d91228da7ebddbfc5)

./flake.nix
./lib/default.nix
./lib/fixed-points.nix
./lib/attrsets.nix
./lib/systems/default.nix
./lib/systems/flake-systems.nix
./default.nix
./lib/minver.nix
./pkgs/top-level/impure.nix
./pkgs/top-level/default.nix
./lib/trivial.nix
./lib/lists.nix
./pkgs/stdenv/booter.nix
./pkgs/stdenv/default.nix
./lib/systems/inspect.nix
./lib/systems/architectures.nix
./lib/systems/parse.nix
./lib/strings.nix
./lib/types.nix
./lib/systems/platforms.nix
./lib/modules.nix
./pkgs/top-level/config.nix
./lib/options.nix
./pkgs/stdenv/darwin/default.nix
./pkgs/top-level/stage.nix
./pkgs/stdenv/adapters.nix
./pkgs/build-support/trivial-builders.nix
./pkgs/top-level/splice.nix
./pkgs/top-level/all-packages.nix
./pkgs/top-level/aliases.nix
./pkgs/stdenv/generic/default.nix
./lib/customisation.nix
./lib/versions.nix
./pkgs/tools/misc/cowsay/default.nix
./pkgs/servers/x11/xorg/default.nix
./pkgs/servers/x11/xorg/overrides.nix
./pkgs/stdenv/generic/make-derivation.nix
./lib/asserts.nix
./pkgs/stdenv/generic/check-meta.nix
./lib/licenses.nix
./lib/meta.nix
./lib/systems/doubles.nix
./pkgs/top-level/darwin-packages.nix
./pkgs/os-specific/darwin/impure-cmds/default.nix
./pkgs/os-specific/darwin/apple-source-releases/default.nix
./pkgs/os-specific/darwin/apple-source-releases/developer-tools-11.3.1.nix
./pkgs/os-specific/darwin/apple-source-releases/macos-11.0.1.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/default.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/apple_sdk.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/frameworks.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/private-frameworks.nix
./pkgs/development/interpreters/perl/default.nix
./pkgs/development/libraries/libxcrypt/default.nix
./pkgs/shells/bash/5.nix
./pkgs/shells/bash/bash-5.2-patches.nix
./pkgs/tools/compression/xz/default.nix
./pkgs/development/libraries/gnu-config/default.nix
./pkgs/build-support/cc-wrapper/default.nix
./pkgs/development/compilers/llvm/11/default.nix
./pkgs/tools/misc/coreutils/default.nix
./pkgs/tools/text/gnugrep/default.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/libSystem.nix
./pkgs/build-support/bintools-wrapper/default.nix
./pkgs/build-support/fetchurl/default.nix
./lib/deprecated.nix
./lib/fetchers.nix
./pkgs/build-support/fetchurl/mirrors.nix
./pkgs/os-specific/darwin/apple-sdk-11.0/libobjc.nix
./pkgs/os-specific/darwin/signing-utils/default.nix
./pkgs/development/libraries/gmp/6.x.nix
./pkgs/development/tools/misc/gnum4/default.nix
./pkgs/development/tools/misc/autoconf/default.nix
./pkgs/development/libraries/zlib/default.nix
./pkgs/development/tools/misc/texinfo/6.8.nix
./pkgs/development/tools/misc/texinfo/common.nix
./pkgs/development/tools/parsing/bison/default.nix
./pkgs/development/libraries/gettext/default.nix
./pkgs/os-specific/darwin/apple-source-releases/libiconv/default.nix
./pkgs/development/tools/misc/automake/automake-1.16.x.nix
./pkgs/development/tools/misc/libtool/libtool2.nix
./pkgs/tools/misc/file/default.nix
./pkgs/development/tools/misc/help2man/default.nix
./pkgs/top-level/perl-packages.nix
./pkgs/development/perl-modules/generic/default.nix
./pkgs/os-specific/darwin/sigtool/default.nix
./pkgs/development/libraries/openssl/default.nix
./pkgs/build-support/pkg-config-wrapper/default.nix
./pkgs/development/tools/misc/pkg-config/default.nix
./pkgs/build-support/fetchgithub/default.nix
./pkgs/build-support/fetchzip/default.nix
./pkgs/tools/archivers/unzip/default.nix
./pkgs/tools/compression/bzip2/default.nix
./pkgs/build-support/expand-response-params/default.nix
./pkgs/development/libraries/pcre/default.nix
./pkgs/development/libraries/libxml2/default.nix
./pkgs/development/interpreters/python/default.nix
./pkgs/development/interpreters/python/cpython/default.nix
./pkgs/development/libraries/expat/default.nix
./pkgs/development/libraries/libffi/default.nix
./pkgs/build-support/nuke-references/default.nix
./pkgs/development/interpreters/python/setup-hook.nix
./pkgs/development/libraries/ncurses/default.nix
./pkgs/development/tools/misc/autoconf-archive/default.nix
./pkgs/os-specific/linux/bluez/default.nix
./pkgs/build-support/fetchpatch/default.nix
./pkgs/data/misc/mailcap/default.nix
./pkgs/development/interpreters/tcl/8.6.nix
./pkgs/development/interpreters/tcl/generic.nix
./pkgs/development/libraries/tix/default.nix
./pkgs/development/interpreters/tcl/mk-tcl-derivation.nix
./pkgs/development/libraries/tk/8.6.nix
./pkgs/development/libraries/tk/generic.nix
./pkgs/tools/networking/curl/default.nix
./pkgs/development/libraries/libssh2/default.nix
./pkgs/development/libraries/kerberos/krb5.nix
./pkgs/os-specific/darwin/apple-source-releases/bootstrap_cmds/default.nix
./pkgs/development/tools/parsing/flex/default.nix
./pkgs/development/libraries/nghttp2/default.nix
./pkgs/development/libraries/c-ares/default.nix
./pkgs/development/libraries/libev/default.nix
./pkgs/build-support/install-shell-files/default.nix
./pkgs/tools/compression/brotli/default.nix
./pkgs/development/tools/build-managers/cmake/default.nix
./pkgs/top-level/unixtools.nix
./pkgs/os-specific/darwin/apple-source-releases/adv_cmds/boot.nix
./pkgs/os-specific/darwin/apple-source-releases/bsdmake/default.nix
./pkgs/os-specific/darwin/apple-source-releases/dyld/default.nix
./pkgs/os-specific/darwin/apple-source-releases/ICU/default.nix
./pkgs/development/compilers/llvm/11/clang/default.nix
./pkgs/os-specific/darwin/binutils/default.nix
./pkgs/development/tools/misc/binutils/default.nix
./pkgs/os-specific/darwin/cctools/port.nix
./pkgs/development/compilers/llvm/11/libcxx/default.nix
./pkgs/development/compilers/llvm/11/libcxxabi/default.nix
./pkgs/development/compilers/llvm/11/llvm/default.nix
./pkgs/tools/text/patchutils/0.3.3.nix
./pkgs/tools/text/patchutils/generic.nix
./pkgs/tools/compression/pbzx/default.nix
./pkgs/tools/compression/xar/default.nix
./pkgs/os-specific/darwin/rewrite-tbd/default.nix
./pkgs/development/libraries/libyaml/default.nix
./pkgs/development/tools/misc/autoconf/2.69.nix
./pkgs/os-specific/darwin/libtapi/default.nix
./pkgs/development/tools/misc/dejagnu/default.nix
./pkgs/tools/misc/expect/default.nix
./pkgs/build-support/substitute/substitute-all.nix
./pkgs/development/compilers/llvm/11/compiler-rt/default.nix
./pkgs/development/tools/xcbuild/wrapper.nix
./pkgs/development/tools/xcbuild/sdks.nix
./lib/generators.nix
./pkgs/development/tools/xcbuild/platforms.nix
./pkgs/tools/archivers/cpio/default.nix
./pkgs/development/tools/build-managers/gnumake/default.nix
./pkgs/tools/misc/findutils/default.nix
./pkgs/applications/editors/ed/default.nix
./pkgs/tools/compression/lzip/default.nix
./pkgs/tools/text/diffutils/default.nix
./pkgs/tools/archivers/gnutar/default.nix
./pkgs/tools/compression/gzip/default.nix
./pkgs/tools/text/gnused/default.nix
./pkgs/tools/text/gawk/default.nix
./pkgs/tools/text/gnupatch/default.nix
./pkgs/development/libraries/libidn2/default.nix
./pkgs/development/libraries/libunistring/default.nix
./pkgs/tools/compression/zstd/default.nix
./pkgs/stdenv/generic/common-path.nix
3 Likes

This is a great little project! Thanks for making and sharing! I checked out a few of the files and they look unchanged relative to cfb43ad7b941d9c3606fb35d91228da7ebddbfc5. Is it true that you didn’t change any of the files that were needed for nixpkgs-skeleton to build cowsay?

I’m not exactly sure about the question. But hopefully this will answer it;

  • I forked nixpkgs
  • Checked out that commit you mentioned
  • Built cowsay using that command
  • Recorded all the accessed files
  • Deleted all the not-mentioned files
  • Rebuilt to make sure it still worked
  • Pushed that change

I never did get around to repeating the process for Linux, mostly cause I spent a while trying to make sure nix wasn’t caching anything (which I never really confirmed)

It’d be nice to have a Github action that automates this for Linux and Mac. Sadly my Github action skills aren’t up for that task yet