If I instantiate a derivation, the resulting .drv
file declares a list of inputDrvs
, which AFAICT is a complete set of all derivations that this one depends on prior to building. I can also use nix show-derivation
to get a JSON equivalent. However, what I can’t figure out how to do is reconstruct this same list from within Nix itself. Is there any way to do this?
The use-case here is I’m trying to write a tool to help me explore dependencies and I thought it would be really handy to be able to write it in Nix directly, if only I could get at this data, rather than wrapping nix show-derivation
.
2 Likes
Not a direct answer to your question, but if you haven’t seen it already you may want to read this:
https://www.tweag.io/posts/2019-02-06-mapping-open-source.html
They share the source code for how it’s all produced, including some nix expressions to walk the graph, so it might help for drawing inspiration.
3 Likes
I use something similar to this to ensure that my CI has all build time deps cached:
opened 01:37AM - 22 Feb 17 UTC
stale
If I have:
```nix
{
exportReferencesGraph = [ "foo" package ];
}
```
…
I get the runtime dependencies of `package`. If instead I:
```nix
{
exportReferencesGraph = [ "foo" package.drvPath ];
}
```
I get _all_ the recursive build-time dependencies of `package`. What I actually want is the closure of the _runtime_ dependencies of `package`'s build-time dependencies, but I don't know how to get that.
E.g., if `package` is `hello` and building it depends on `stdenv`, I want the _runtime_ closure of `stdenv`, not its build-time closure, since the latter will include everything all the way back to `bootstrap-tools` and I get a ton of stuff I don't need. When Nix builds `package`, it doesn't pull in all the transitive build junk.
How can I get the behavior I want?
cc @edolstra @shlevy @7c6f434c @cleverca22
Edit: this seems like the logic I want: https://github.com/NixOS/nix/blob/master/src/libstore/build.cc#L1251-L1266
2 Likes
Both of these are useful. Looks like the “Mapping a Universe” approach is to evaluate all of the packages into a custom store in order to produce the .drv files, then walk the store with Haskell and parse them using Nix.Derivation.parseDerivation
. That’s cute, and I didn’t realize Nix had a Haskell library for parsing derivations. The build-time dependencies approach is pure Nix but is using regular expressions to parse the derivation >_<
I ended up writing my tool around nix show-derivation
with a scripting language after all, which is irritating, but it works reasonably enough. If I were more competent with Haskell I would have been tempted to use that, but I’m not.
1 Like
@lilyball Besides the already proposed super interesting ideas (see also blog post ), which I had no idea of, I’ve implemented a pure Nix function that gets all of the propagatedBuildInputs
and buildInputs
of a derivation here:
NixOS:master
← doronbehar:declarative-wrappers
opened 08:50PM - 12 Apr 20 UTC
###### Motivation for this change
Create a new wrapper to rule them all! :smi… ling_imp:
Now Seriously: This has been talked over since like for ever and now I'd like to present to you a Pure Nix function that creates a wrapper environment for any derivation you'll give it, using `symlinkJoin`. It does so by recursively searching for a derivation's inputs' for derivations with an attribute such as:
```nix
passthru = {
propagateEnv = {
GST_PLUGIN_SYSTEM_PATH_1_0 = "%out%/lib/gstreamer-1.0";
};
};
```
This recursive search means, that you don't have to take care of orchestrating `wrapGAppsHook` and `wrapQtAppsHook` or others with stuff like this:
```nix
preFixup = ''
makeWrapperArgs+=("''${qtWrapperArgs[@]}")
''
```
I tested in this new function in wrapping [picard](https://github.com/NixOS/nixpkgs/blob/e3510bc553d86efd75d708f25702ddc8694c9dc4/pkgs/applications/audio/picard/default.nix), [gucview](https://github.com/NixOS/nixpkgs/blob/e3510bc553d86efd75d708f25702ddc8694c9dc4/pkgs/os-specific/linux/guvcview/default.nix) (which is currently not wrapped but there's an open PR that wraps it the old way https://github.com/NixOS/nixpkgs/pull/84449) and [arandr](https://github.com/NixOS/nixpkgs/blob/6283f15bb9b65af64571a78b039115807dcc2958/pkgs/tools/X11/arandr/default.nix).
Here's an example of the environments (new lines inserted to help readability) created for guvcview with the old way and the new way:
#### OLD
```sh
GIO_EXTRA_MODULES=
'/nix/store/z7gzw2bb3ypvf1wq1fjjn5r45idfgg0g-dconf-0.34.0-lib/lib/gio/modules'${GIO_EXTRA_MODULES:+':'}$GIO_EXTRA_MODULES
GIO_EXTRA_MODULES=
'/nix/store/z7gzw2bb3ypvf1wq1fjjn5r45idfgg0g-dconf-0.34.0-lib/lib/gio/modules'${GIO_EXTRA_MODULES:+':'}$GIO_EXTRA_MODULES
GDK_PIXBUF_MODULE_FILE=
'/nix/store/18l8ly5235r94s2rr122i3cfichccxqn-librsvg-2.46.4/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache'
XDG_DATA_DIRS=
'/nix/store/nwksqszw0dakmizrrlql9pamcf39vnxg-gtk+3-3.24.14/share/gsettings-schemas/gtk+3-3.24.14
/nix/store/slmsmbyfgf1rqj9rp6s1y33cqaz8q203-gsettings-desktop-schemas-3.34.0/share/gsettings-schemas/gsettings-desktop-schemas-3.34.0
/nix/store/nwksqszw0dakmizrrlql9pamcf39vnxg-gtk+3-3.24.14/share/gsettings-schemas/gtk+3-3.24.14
/nix/store/slmsmbyfgf1rqj9rp6s1y33cqaz8q203-gsettings-desktop-schemas-3.34.0/share/gsettings-schemas/gsettings-desktop-schemas-3.34.0'${XDG_DATA_DIRS
+'
'}$XDG_DATA_DIRS
XDG_DATA_DIRS=
'/nix/store/92p57svcsqgwa5fgb8z9qg7m7r66nx7a-guvcview-2.0.6/share'${XDG_DATA_DIRS:+':'}$XDG_DATA_DIRS
```
#### NEW
```sh
GDK_PIXBUF_MODULE_FILE=
${GDK_PIXBUF_MODULE_FILE-'/nix/store/a7dwk7qamr2j7220rsfvn8w386b8a3lc-librsvg-2.48.0/lib/gdk-pixbuf-2.0/2.10.0
loaders.cache'}
GIO_EXTRA_MODULES=
'/nix/store/0byixyx6l833y9a7ah8ygbh4d4nfgcin-dconf-0.36.0-lib/lib/gio/modules'${GIO_EXTRA_MODULES:+':'}$GIO_EXTRA_MODULES
GI_TYPELIB_PATH='/nix/store/2chzii7gwmlcii9x73l0v3bl7lf2awvb-gtk+3-3.24.14/lib/girepository-1.0
/nix/store/ly22x5s6v91i840xkkj82dhblkfgl79v-atk-2.35.1/lib/girepository-1.0
/nix/store/kah2hha1zcbv0xvyq7fdx26gxd03002x-gdk-pixbuf-2.40.0/lib/girepository-1.0
/nix/store/xjg7kpw38mfaab2fsarll2d39nk06ha5-gsettings-desktop-schemas-3.36.0/lib/girepository-1.0
/nix/store/r4vrzisk85nfkmmafvvi348czb8l4p86-gobject-introspection-1.64.0/lib/girepository-1.0
/nix/store/y9sn0hll1m90zdi8bwa5126lmdkdp8i9-pango-1.44.7/lib/girepository-1.0
/nix/store/a7dwk7qamr2j7220rsfvn8w386b8a3lc-librsvg-2.48.0/lib/girepository-1.0
/nix/store/kah2hha1zcbv0xvyq7fdx26gxd03002x-gdk-pixbuf-2.40.0/lib/girepository-1.0
/nix/store/ly22x5s6v91i840xkkj82dhblkfgl79v-atk-2.35.1/lib/girepository-1.0
'${GI_TYPELIB_PATH:+':'}$GI_TYPELIB_PATH
XDG_DATA_DIRS=
'/nix/store/2chzii7gwmlcii9x73l0v3bl7lf2awvb-gtk+3-3.24.14/share/gsettings-schemas/gtk+3-3.24.14
/nix/store/xjg7kpw38mfaab2fsarll2d39nk06ha5-gsettings-desktop-schemas-3.36.0/share/gsettings-schemas/gsettings-desktop-schemas-3.36.0
/nix/store/y2fj89hq03nxdpc7bv7k9dpbc3f082ah-runtime-env/share
'${XDG_DATA_DIRS:+':'}$XDG_DATA_DIRS
```
As you can see, there are duplicates entries in the old wrapper which increases the chances of hitting obsurd issues such as what https://github.com/NixOS/nixpkgs/pull/84689 fixed.
## What ~~didn't~~ works?
(EDIT) Everything I tested.
## How would a user override a derivation wrapped this way?
Say `all-packages.nix` has:
```nix
my-awesome-pkg = wrapGeneric (callPackage ../applications/my-awesome-pkg { }) { };
```
Assuming the user knows `my-awesome-pkg` is wrapped with `wrapGeneric`, he would need to use an overlay like this:
```nix
self: super:
{
my-awesome-pkg = super.wrapGeneric (
super.my-awesome-pkg.unwrapped.overrideAttrs(oldAttrs: {
preFixup = ''
overriding preFixup from an overlay!!
'';
})
) {};
}
```
###### Things done
- [x] Tested using sandboxing ([nix.useSandbox](http://nixos.org/nixos/manual/options.html#opt-nix.useSandbox) on NixOS, or option `sandbox` in [`nix.conf`](http://nixos.org/nix/manual/#sec-conf-file) on non-NixOS linux)
- Built on platform(s)
- [x] NixOS
- [ ] macOS
- [ ] other Linux distributions
- [ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
- [ ] Tested compilation of all pkgs that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review wip"`
- [x] Tested execution of all binary files (usually in `./result/bin/`)
- [ ] Determined the impact on package closure size (by running `nix path-info -S` before and after)
- [x] Ensured that relevant documentation is up to date
- [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md).
It’s also a sort of hack because you can reach infinite recursion with it, unless you pass it a list of known packages you don’t want it to iterate.
I noted my self to try the other methods once I’ll have more time to work on that PR.
3 Likes