How to compute build time closure without Nix?

Help
1

I’m trying to compute the build time closure of a derivation, outside of Nix. Slightly misleading title, because I think I’ve done most of the work, just ended up stuck on some details :wink:

I naively assumed parsing a derivation and then recursively gathering all inputDrvs and inputSrcs would suffice, but that doesn’t seem to hold - at least not for my nixos-system derivation. Using this strategy I can discover 7752 out of 7790 paths compared to nix-store -qR of my system derivation - so there are 38 missing.

The difference seems to stem from the store path for modules.xml (nixpkgs src) which is generated using builtins.toFile. Importantly, this file contains XML includes, which explains the dependency:

$ nix why-depends /nix/store/yb71ygc8dpxnkp64vdar1wazjf5wjbn7-modules.xml /nix/store/7l34sk54sjdlh7xcivrmj8gs312xfirg-pict-rs.xml
/nix/store/yb71ygc8dpxnkp64vdar1wazjf5wjbn7-modules.xml
╚═══/: …/>.<xi:include href="/nix/store/7l34sk54sjdlh7xcivrmj8gs312xfirg-pict-rs.xml" />.<xi:include hre…
    => /nix/store/7l34sk54sjdlh7xcivrmj8gs312xfirg-pict-rs.xml

If I query the nix-store for requisites of modules.xml I find all 38 missing paths:

$ nix-store -qR /nix/store/yb71ygc8dpxnkp64vdar1wazjf5wjbn7-modules.xml
/nix/store/07fi8dm3inmsbxhnydzjl0jshw8346hn-trezord.xml
/nix/store/0g8yl6sbi4knqfi8n48f06v2vyx31770-yggdrasil.xml
/nix/store/0nxlsrqhghky7v0shfsld98mlcfbvqrx-borgbackup.xml
...

Reading Eelco’s PhD thesis to figure out whether this is intended leads me to believe it is (because by definition all recursive references are part of a closure and they don’t necessarily have to be in any derivation inputSrcs), but I remain puzzled by the fact that this is the only occurrence (in my system closure) of a dependency not reachable by only parsing derivations - it requires outside knowledge of the dependency (e.g. at evaluation time, which is how I believe Nix handles it).

Does this make sense?

Am I missing anything that does imply modules.xml should be built from a derivation instead (which would solve my problem at least for my current system derivation)?

What’s the easiest way to figure out these missing dependencies? I managed to come up with either scanning all inputs for unknown store paths (slow) or querying the Nix store database (which keeps a table of references), just to get the last handful of paths…

1 Like
2

Those are checked into the nixos repository. They should be listed as inputSrcs, as they are added at evaluation time , and are not a result of a derivation.

3

I’m happy to hear that (as it would solve my issue), but of which derivation though? The problem is that modules.xml itself lacks a derivation (builtins.toFile produces a plain store path).

The last derivation involved is:

$ nix show-derivation /nix/store/419qkq48sbv1i5ng248z8b2yahkxj48k-generated-docbook.drv
{
  "/nix/store/419qkq48sbv1i5ng248z8b2yahkxj48k-generated-docbook.drv": {
    ...
    "inputSrcs": [
      ...
      "/nix/store/yb71ygc8dpxnkp64vdar1wazjf5wjbn7-modules.xml"
    ],

Which only lists modules.xml which is just a store path. It refers to those XML includes but there’s no derivation with a direct dependency.

ie. the dependency looks like generated-docbook.drv → modules.xml → borgbackup.xml - but only the first dependency is based on inputSrcs.

Related: is there a reason why the doc builders rely on builtins.toFile instead of pkgs.writeText? The latter would produce a derivation and thus include the XML includes in the inputSrcs of the resulting derivation.