How does Nix compute runtime dependencies?

Help
1

How does Nix compute runtime dependencies for commands like nix-copy-closure and nix copy?

Reading the Nix Manual, it appears to say that runtime dependencies are computed when needed, by scanning output paths for other /nix/store/* paths.

I have a couple questions about this:

  1. Is the above explanation correct? This information is computed on-the-fly, and not present in the .drv file, correct? Can anyone point out the place in the Nix source code where this scanning step takes place?

  2. There are a couple commands I know of to let you easily inspect the runtime (or build-time) closure:

    However, are there any tools that show which files in a given output path have which runtime dependencies? (edit: err, yes, there is: nix why-depends. See the next question where I show using this…)

    For instance, hello has a reference to glibc:

    $ nix-build '<nixpkgs>' -A hello
/nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
$ tree /nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
/nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
├── bin
│   └── hello
└── share
    ├── info
    │   └── hello.info
    ├── locale
    │   ├── bg
    │   │   └── LC_MESSAGES
    │   │       └── hello.mo
...
$ nix-store --query --references /nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31

    I can confirm this by grepping:

    $ grep -r 9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31 /nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
Binary file /nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10/bin/hello matches

    Is this a legitimate way to figure this out?

    It seems like it would be nice to have Nix give me a better interface for this.

  3. Are there any good ways to go about figuring out why a given output path has some dependency?

    For instance, nix why-depends appears to do some of this:

    $ nix why-depends /nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10 /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31
/nix/store/w9yy7v61ipb5rx6i35zq1mvc2iqfmps1-hello-2.10
╚═══bin/hello: …...................../nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/ld-linux-x86-64.…
    => /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31

    However, is there any general advice for figuring out why a given binary contains a /nix/store path? (I imagine the answer to this will depend on what type of binary it is…)

  4. If I know that a given runtime dependency is not actually needed (so it can be excluded from the closure), is there any way to tell nix copy or nix-copy-closure to exclude it?

    I imagine I can modify the original derivation to remove unneeded /nix/store/ path in its output files, but I am just wondering what other options I have here.

  5. When writing a derivation, is there any way to force a runtime dependency?

    I’ve seen /nix/store/*/nix-support/propagated-build-inputs files. Is that effectively what these files are for?

6 Likes
2

I suggest reading the nix-pills which explain exactly these kinds of detail. If you still have questions after reading them, I suggest asking again in this discourse: I think that one question per thread would be easier to answer.

3

You kind of answered this yourself. As long a nix can read a /nix/store path in the outputs, it will consider it a runtime dependency. Nix can’t know about these paths til after it builds a package, so this is why why-depends will usually build the package before giving output.

The out paths from the inputs is known, and nix can assert that the paths exist.

It would probably be located in the nix store realize code, but I’m not super familiar with the code base.

nix-store --query --references for direct runtime dependencies
nix-store --query --requisites for all runtime dependencies

if you pass a .drv, you will get the related build dependencies.

nix why-depends and grep are all that I’m aware of.

You can use disallowedReferences or disallowedRequisites when creating your derivation to prevent this Introduction

As long as the store path appears in the out path, it will be a runtime dependency.

I believe so https://github.com/NixOS/nixpkgs/blob/1c24ff1f1f5fa9d42a6fa5a377287623def78fe3/pkgs/development/tools/build-managers/bazel/bazel_0_29/default.nix#L548. One example I can think of is .jar files needing a given library to be available, but since they aren’t plain text, nix won’t read that the store path is there. So it’s common to write the libraries to $out/nix-support/<file> to ensure that it’s added as a runtime dependency.

Although propagated-build-inputs is special to the default builder https://github.com/NixOS/nixpkgs/blob/1c24ff1f1f5fa9d42a6fa5a377287623def78fe3/pkgs/stdenv/generic/setup.sh#L1111 as this will also be consumed by direct downstream dependencies if they also use the default builder.

1 Like
4

It’s nice that the SQLite schema is also very straightforward. So, it’s also easy to query references in that way:

$ nix-store -q --references /nix/store/jr56bx8a41h1x0wm7ih2mic5i3cjfcm1-coreutils-8.32
/nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35
/nix/store/g6441q8xrb8lq3lk5r4s7zjdv23bpxdx-attr-2.4.48
/nix/store/kzw4p3v89r2cbvbnm7g8kf9fh4z3c5m7-acl-2.2.53
/nix/store/xivj0vwm0pjpmwsjrc8qcca9fcfyxm4b-openssl-1.1.1i
/nix/store/jr56bx8a41h1x0wm7ih2mic5i3cjfcm1-coreutils-8.32

$ sqlite3 /nix/var/nix/db/db.sqlite
sqlite> select id from ValidPaths where path = "/nix/store/jr56bx8a41h1x0wm7ih2mic5i3cjfcm1-coreutils-8.32";
48929
sqlite> select path from ValidPaths where id in (select reference from Refs where referrer = 48929);
/nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35
/nix/store/g6441q8xrb8lq3lk5r4s7zjdv23bpxdx-attr-2.4.48
/nix/store/kzw4p3v89r2cbvbnm7g8kf9fh4z3c5m7-acl-2.2.53
/nix/store/xivj0vwm0pjpmwsjrc8qcca9fcfyxm4b-openssl-1.1.1i
/nix/store/jr56bx8a41h1x0wm7ih2mic5i3cjfcm1-coreutils-8.32
3 Likes
5

@jonringer Thanks for all this information.

These look like they could be really helpful when actually writing derivations. I’d mainly image using them as debugging tools.

This is a great example. It is very simple and shows exactly what is going on. It makes sense that the .jar files are compressed, so the /nix/store/* paths inside them need to explicitly added in a nix-support file.

@danieldk Ah, I didn’t realize that this information is stored in the Nix DB.

I had assumed this information was computed at runtime when needed by commands like nix copy and nix-copy-closure.

However, given that there are special attributes like disallowedReferences that can be used from within Nix, I guess runtime dependencies are calculated after a derivation is built and preemptively stored in the Nix DB (as opposed to be computed lazily when needed).

I’m still interested in reading the actual Nix code that scans files and computes runtime dependencies if anyone knows where to find it in the Nix codebase.

6

There is also the exportReferencesGraph special attribute.

This is used a couple places in Nixpkgs to do some interesting things:

writeReferencesToFile: This creates a derivation with a single file that contains a list of references to the given input derivation.

https://github.com/NixOS/nixpkgs/blob/24e5fe6075bc7a137bb701eb8a378f5a8689e10d/pkgs/build-support/trivial-builders.nix#L389-L403

closureInfo: This is similar to writeReferencesToFile but appears to also create some other related files.

https://github.com/NixOS/nixpkgs/blob/24e5fe6075bc7a137bb701eb8a378f5a8689e10d/pkgs/build-support/closure-info.nix

3 Likes