But, this discovery isn’t verified to be true (I haven’t built without this lightdm greeter to verify that the inkscape/capitaine-cursors substree is no longer required. Also, the investigation path that I took was a little heuristic (educated guessing/grepping).
Is there a way for me to use the nix tooling (or 3rd-party tooling) to answer the question “which lines of which derivations require this dependency” almost like a stacktrace-version of why-depends such that I would see something like:
$ nix why-depends .\#nixosConfigurations.framework-laptop.config.system.build.toplevel .\#nixosConfigurations.framework-laptop.pkgs.inkscape
/nix/store/ljrqyh1pd7nd2r8zfvm91ibm9k97nwdk-source/modules/flake.nix:179 # the line which defines nixosConfigurations.framework-laptop
/nix/store/ljrqyh1pd7nd2r8zfvm91ibm9k97nwdk-source/modules/flake.nix:184 # modules array entry which pulls in ./modules/desktop.nix
/nix/store/ljrqyh1pd7nd2r8zfvm91ibm9k97nwdk-source/modules/desktop.nix:37 # the line which has `lightdm.greeters.enso.enable = true;`
Alternative to getting a clean stack trace like this, would it be possible to at least know which derivation (in this case lightdm-enso-os-greeter = callPackage ../applications/display-managers/lightdm-enso-os-greeter { };) pulls in this dependency?
It seems that why-depends glosses over parts of the tree between system-path.drv and capitaine-cursors-4.drv in such a way as to elide the relationship between system-path.drv and config.lightdm.greeters.enso. I want to see this relationship.
The problem here is that why-depends cannot know how the dep came into system-path.drv as it works on drv level. Drv files are dumb data that aren’t even meant to be introspectable because they’re a direct result of an evaluation of Nix expressions. They’re just an on-disk representation of the DAG that forms a closure. There is no info about how one drv became a direct dependency of another drv inside the drv files; none whatsoever.
Even at the Nix expressions level it isn’t trivial to figure out which option caused a package to become part of the closure because configuration is, again, just data that is read from some file and then processed further. This data is a little more introspectable but not at a high level (such as options-level) because options are a system that we have built on top of the primitives of the Nix expression language.
Your idea with the stack trace could work in a way though: You could try overlaying inkscape = throw ...; and eval. That should produce a stack trace which should look about the way you outlined, just in the limited means Nix is able to relatively easily do this on it’s low (concrete) level. Pretty certain the stack trace would include the file where the option puts a package into systemPackages that depends on inkscape.
Whether you’d be able to read that out of that stack trace if you didn’t know the culprit beforehand is another question…
@atemu, please forgive my delayed response (this isn’t a very high priority for me - more of an exploration, for now). My response to yours is inline, below.
The problem here is that why-depends cannot know how the dep came into system-path.drv as it works on drv level. Drv files are dumb data that aren’t even meant to be introspectable because they’re a direct result of an evaluation of Nix expressions. They’re just an on-disk representation of the DAG that forms a closure. There is no info about how one drv became a direct dependency of another drv inside the drv files; none whatsoever.
That makes sense. I had said that why-depends seemingly elides over the relationships I’m seeking. But, it’s the derivation (the result of Nix’s evaluation of all modules/system configuration, apparently) that’s eliding.
Even at the Nix expressions level it isn’t trivial to figure out which option caused a package to become part of the closure because configuration is, again, just data that is read from some file and then processed further. This data is a little more introspectable but not at a high level (such as options-level) because options are a system that we have built on top of the primitives of the Nix expression language.
So, are you saying that this tooling might fit into a nixos-depends kind of tool (one which understands NixOS modules)? Because, the evaluator definitely knows how to pull in dependencies based on configuration. So, the information is definitely available somewhere. I’m not clear on the implementation of NixOS modules - I think they are merged and recursively evaluated to only a simple attribute set (not a derivation). If NixOS modules were to change such that each evaluated to its own derivation then I guess system-path.drv would have a relationship with the foo-module.drv which specifies foo module’s dependencies (and then why-depends would reveal the relationships I’m interested in). Does this sound right to you? And do you see any problems with pursuing this change?
Your idea with the stack trace could work in a way though: You could try overlaying inkscape = throw ...; and eval. That should produce a stack trace which should look about the way you outlined, just in the limited means Nix is able to relatively easily do this on it’s low (concrete) level. Pretty certain the stack trace would include the file where the option puts a package into systemPackages that depends on inkscape.
Whether you’d be able to read that out of that stack trace if you didn’t know the culprit beforehand is another question…
While this could work in an ad-hoc situation like the one I’m describing, I’m seeking a more general solution. In the general case inkscape may have been provided from any one of my inputs (or the inputs of my inputs…) which would have made it more difficult to find - even then I would have needed to do specific manual work to overlay only that derivation. This doesn’t scale well for cases when I have many applications like inkscape whose progeny I’m trying to ascertain. I was “lucky” that I tracked down the b in nix why-depends a b. Ideally, there would be tooling that I could use to narrow down the exact line in a configuration (or the tree of such lines) that’s introducing a dependency b when building some attribute a.
I think that could theoretically be done using a SAT-solver; finding the set of non-default options that, when turned off, would not have some derivation as part of the closure. That wouldn’t be practical of course but possible. More efficient options might be possible too.
The evaluator knows about relationships between derivations but not modules. That’s what I meant with options being a higher level construct; their concern is data, not derivations.
Out of that set of data, a DAG of derivations is constructed but this data set has no relationship to the files from which the set of data is constructed.
That’s an interesting thought but not realistic because that’s not how the module system works currently. See above explanation. Modules do not an can not map to derivations.
So, outside of a “throw overlay” do you have any ideas for how the default tooling/modules might be altered so as to increase the “granularity” of the dependency tree?
I ran into a solution to this by accident. The NixOS module system keeps track of the provenance of every option because of the way it merges things together after the fact.
The following expression will give you a list of packages in environment.systemPackages along with the location where they were added. It doesn’t go as far as doing a stack trace back, but it’ll give you the nixos module that did it, which should be sufficient to manually find it out.
/nix/store/jdz9hdq6qvavkkyj693j9770z33f0064-source is the location of the current nixpkgs commit, it can be correlated by looking at the flake inputs’ outPath.
@mrene Thanks for sharing this! I need a minute to grok the expression and the systemPackages attribute that you’re taking advantage of. I think this could be a good base to help answer the question “which NixOS module bar did I install in order to require foo dependency”? So, this should solve my original question and I’ll circle back when I understand exactly what your code does.
However, I see a need for eval-time introspection that yields something like a line-based stacktrace for a given dependency. I don’t see how it would be impossible, given all relevant Nix code, to yield the set of lines relevant in order to compute/evaluate a given attribute. This is the real goal, I think, unless someone can come up with something better. I basically want to answer the question “which files and which lines of which files are/were needed in order to evaluate this attribute”. If it’s a derivation then it’s the set of all lines necessary in order to evaluate/realize the derivation.
Am I talking crazy or does my request/idea make sense?
In other words “I know/guess that my NixOS system depends on a module requiring capitaine-cursors-4 but I have no idea which module that is. But, I can use @mrene’s magic expression to inverse-map for which module (whose source code is described in the file located at the value of the "file" attribute) has capitaine-cursors-4 in its list of required packages. (“inverse-map” because the output of .nixosConfigurations.${name}.options.environment.systemPackages.definitionsWithLocations is associating the module file with the set of dependencies (and not the other way around - what we’re seeking).”
It’s a little clunky - I don’t actually see inkscape in this dependency tree (I guess because the elements of name only include top-level dependencies). But, it’s definitely a tool in my toolbox.
I wonder, though, still, if I can achieve my holy grail of source code + attribute -> set of all relevant source code lines.
P.s. I’m wondering how @mrene stumbled on .definitionsWithLocations. My guess is REPL exploration - but looking for what exactly…
You’re spot on on REPL exploration. I was watching this talk about the nix module system and the presenter (@infinisil) alluded that .options is a super-set of .config (because it contains the schema on top of the values (which .config contains)). I was surprised to see a definitionsWithLocations, I was fully expecting this information to have been lost by the time it reached this point.
From what I understood the whole module system is implemented in nix code, so if you want to access more detail you could instrument the module system and add more relevant information to the mix.
There is however a fundamental difference between the (excuse the spontaneous naming) “configuration-plane” (nix code) and the “derivation-plane” (which nix why-depends queries) which can be non-trivial to overcome. For example, if you depended on a package from `home-manager (and are using its nixos module) the dependency chain could have looked like this instead:
❯ nix why-depends .#nixosConfigurations.beast.config.system.build.toplevel /nix/store/g4slvksnx7ba07jviigb700nc2xklnrd-gnome-shell-extension-space-bar-19
warning: Git tree '/home/mrene/dotfiles' is dirty
/nix/store/r36r9ysaqwb9q0b4qnina4cdzzka2w3b-nixos-system-beast-23.11.20230814.dirty
└───/nix/store/gmz7xrffn8f5qnw5c701cch465175mmf-etc
└───/nix/store/dyjglqz7wqb80hn1mrknn78aarbm9y5d-system-units
└───/nix/store/ygpmjq4k9dppr1fvxslx00lwy5hh2qp8-unit-home-manager-mrene.service
└───/nix/store/djgj1pkyhwfy4y5n9vzq7s0ni8vsn9fx-home-manager-generation
└───/nix/store/q577bw310zcz0pgib4mky1dw6mnjw8zj-home-manager-path
└───/nix/store/g4slvksnx7ba07jviigb700nc2xklnrd-gnome-shell-extension-space-bar-19
This indicates an indirect relationship because home-manager calls its activation script inside a systemd user service. From the nixos modules’ perspective all we would see is that the home-manager module defined a systemd service which has a dependency on some other derivation.
Finding which attribute is responsible for configuring the inputs to the home-manager-path derivation is also a challenge. You can search through home-manager and find out where it’s defined, find the .....options.home-manager.users.value.mrene.home.packages path, but hit a wall because home-manager has a nested module system which doesn’t seem exposed to the NixOS side of things.
There is hope in that any module system will end up using nixpkgs.lib.evalModules, but merging these two planes may require deep changes (and a lot more thought that I have given at this point).