How do you find reverse dependencies?

When I change a package I would like to build and test a few it’s reverse dependencies to see if there are any obvious API breakages.

For small package changes rebuild-amount.sh --print is good enough.

But larger packages like texinfo are part of bootstrap and thus it triggers rebuilds for most packages on linux. rebuild-amount.sh output is not useful. Good first approximation would be to find all derivations that mention texinfo in it’s package inputs.

I came up with the following hack to find candidates:

# cat revdeps.nix
# use as:
#    import ./revdeps.nix "texinfo" pkgs lib
revdep: pkgs: lib:
let isDrv = v: (builtins.tryEval v).success && lib.isDerivation v;
    isDrvName = name: d: isDrv d && lib.strings.hasInfix name d.name;
    getList = a: s: if builtins.hasAttr a s then builtins.getAttr a s else [];
    matchedPackages = lib.filterAttrs (n: v: isDrv v
                                             && builtins.any (isDrvName revdep)
                                                             (getList "buildInputs" v
                                                              ++ getList "nativeBuildInputs" v))
                                      pkgs;
in
  builtins.attrNames matchedPackages

Example usage (seems to work):

$ nix repl .
Welcome to Nix version 2.4pre20210908_3c56f62. Type :? for help.

Loading '.'...
Added 14984 variables.

nix-repl> import ./revdeps.nix "texinfo" pkgs lib
[ "R" "allegro" "allegro4" "allegro5" "asdf" "asdf_2_26" "asdf_3_1" "asymptote" "bashInteractive" "bashInteractive_4" "bashInteractive_5" "bc" "bintools-unwrapped" "binutils-unwrapped" "blitz" "cgdb" "cgui" "chrony" "clfswm" "cmakeCurses" "cmakeWithGui" "cre2" "dgsh" "docbook2x" "e2fsprogs" "ecl" "emacsMacport" "eukleides" "ffmpeg" "ffmpeg-full" "ffmpeg_2" "ffmpeg_2_8" "ffmpeg_3" "ffmpeg_3_4" "ffmpeg_4" "flex_2_5_35" "foxtrotgps" "freetalk" "fswatch" "fwknop" "gama" "gauche" "gaucheBootstrap" "gcc-unwrapped" "gccForLibs" "gcl" "gcl_2_6_13_pre" "gdb" "gdb-multitarget" "gengetopt" "git" "git-doc" "gitFull" "gitSVN" "glibcInfo" "gnupg" "gnupg22" "gnuplot" "gnuplot_aquaterm" "gnuplot_qt" "gnutls-kdh" "gpgme" "gpm" "gpm-ncurses" "groff" "grub" "guile-fibers" "guile-gcrypt" "guile-git" "guile-gnome" "guile-json" "guile-lib" "guile-ssh" "guile-xcb" "guileGnome" "guile_lib" "gxmessage" "heimdal" "heimdalFull" "idutils" "iksemel" "indent" "irods" "irods-icommands" "kvm" "ledger" "lepton-eda" "libgccjit" "libheimdal" "liblouis" "libmikmod" "libredwg" "librep" "libtasn1" "lilypond" "lilypond-unstable" "lzip" "macchanger" "mailutils" "marst" "maxima" "maxima-ecl" "mercury" "mitscheme" "mitschemeX11" "monotone" "mpfi" "msmtp" "mu" "myserver" "nano" "ne" "netmask" "notmuch" "ocrad" "octave" "octave-jit" "octaveFull" "pinfo" "poke" "polipo" "ponysay" "pspp" "qemu" "qemu_full" "qemu_kvm" "qemu_test" "qemu_xen" "qemu_xen-light" "qemu_xen_4_10" "qemu_xen_4_10-light" "quickjs" "r3rs" "r4rs" "r5rs" "recode" "rubber" "sawfish" "sbcl" "sbcl_2_0_8" "sbcl_2_0_9" "sbcl_2_1_1" "sbcl_2_1_2" "sbcl_2_1_8" "sdcc" "singular" "slibGuile" "solfege" "speechd" "tahoe-lafs" "tahoelafs" "tangogps" "tinc_pre" "tinycc" "udunits" "vc4-newlib" "wdiff" "wget2" "wsjtx" "xboard" "xen" "xen-light" "xen-slim" "xen_4_10" "xen_4_10-light" "xen_4_10-slim" "xnee" "xprintidle-ng" "xzgv" ]

That is almost directly usable for nix build -f . <pkgs>.

I think it lacks support for non-toplevel packages like elpa and friends. Do we have something slightly more generic and probably more maintained that covers similar use case? What do you use for similar purposes? Worth adding to maintainers/scripts/?

Thank you!

7 Likes

I don’t know of a current way to do this, and I would find this very useful!

If I understand correctly, your code looks at nativeBuildInputs and buildInputs, but it won’t see interpolated references (e.g. cp ${texinfo}/bin/* .) or references in other attrs (e.g. propagatedBuildInputs). Maybe a more robust way to do this, would be to check the entire text of each derivation for matches.

1 Like

Assuming you’re making changes to nixpkgs itself.

You can do nixpkgs-review wip and this will attempt to rebuild all packages which were changed by your changes.

You can also see all downstream dependencies by doing nix-store --query --referrers, however, this will only show you the packages which exist on your nix store and refer to the package; there’s the possibility of a package not being reflected in this (although it’s almost instant to do)

2 Likes

I think what @trofi is looking for is a way to query the “direct” referrers. For small changes building all transitive referrers is easy, but for huge staging patches it would be handy to be able to look at just the “direct” referrers.

3 Likes

ah yes, I would also be interested if you find out a reasonable way :wink:

1 Like

If I understand correctly, your code looks at nativeBuildInputs and buildInputs, but it won’t see interpolated references (e.g. cp ${texinfo}/bin/* . ) or references in other attrs (e.g. propagatedBuildInputs). Maybe a more robust way to do this, would be to check the entire text of each derivation for matches.

That’s a great point! texinfo was a simple depend that did not seem to use such tricks. linuxHeaders is one of those. The hack reports no direct users, but we have quite a few ${linuxHeaders} injections.

Here is the version based on string scanning of .drv files:

# use as:
#    import ./arevdeps.nix linuxHeaders pkgs lib
revdepAttr: pkgs: lib:
let isDrv = v: (builtins.tryEval v).success && lib.isDerivation v;
    # skip broken and unsupported packages on this system in a very crude way:
    safeReadFile = df: let c = builtins.tryEval (builtins.readFile df); in if c.success then c.value else "";
    fastHasEntry = i: s: s != builtins.replaceStrings [i] ["<FOUND-HERE>"] s;
    sInDrv = s: d: fastHasEntry s (safeReadFile d.drvPath);
    rdepInDrv = rdep: d: builtins.any (s: sInDrv s d)
                                      (builtins.map (o: rdep.${o}.outPath) rdep.outputs);
    matchedPackages = lib.filterAttrs (n: d: isDrv d && rdepInDrv revdepAttr d)
                                      pkgs;
in builtins.attrNames matchedPackages
nix-repl> import ./arevdeps.nix linuxHeaders pkgs lib
[ "apparmor-bin-utils" "apparmor-parser" "apparmor-utils" "audacity" "audacity-gtk2" "audacity-gtk3" "autofs5" "ceph" "ceph-dev" "dbus-broker" "duperemove" "gfxtablet" "glibcLocales" "glibc_memusage" "ike" "input-utils" "jujuutils" "klibc" "libapparmor" "libceph" "librseq" "libudev" "linuxHeaders" "linuxptp" "musl" "mythtv" "oprofile" "sc-controller" "steamcontroller" "systemd" "systemdMinimal" "trafficserver" "uclibc" "udev" ]

It’s far from complete (for example it misses glibc due to different parameters being passed to linuxHeaders on bootstrap), but is a good starting point for initial tests. Slightly less involved example:

nix-repl> import ./arevdeps.nix ell pkgs lib
[ "bluez" "bluez5" "bluezFull" "ell" "iwd" "ofono" ]
4 Likes

Here’s how OfBorg does it:

It calculates the drv outpath of almost every package in all-packages before and after a change is made and diffs the two.

It is very slow though.

This is what I copied for nixpkgs-update. I was thinking of moving to something simpler like some of the examples above because a majority of time updating is spent calculating outpaths. Thanks for the ideas!

2 Likes

This has two caveats:

  • drv’s reference the build closure, so there’s going to be many more dependencies which may not use the package at runtime.
  • This “only” references derivations which are at the top-level or have the “_recurseForDerivations = true;” flag on a package set. So it’s not a complete list of all possible packages, but probably as close as you can get realistically.
1 Like