How to get a missing .drv file for a derivation from nixpkgs?

psychic · March 4, 2019, 2:41am

I have a package that has some dependencies.

nix-store -qR /nix/store/...

One of the dependencies: /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23

Trying to get its derivation with: nix-store --query --deriver /nix/store/53wi...
yields: /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv

This derivation, however, does not exist on my computer.
Despite having in the nix.conf option (which is on by default, but just in case) to keep derivation files.

So I thought of regetting this .drv file by calling nix-instantiate with the
relevant nix-expression (I know it’s the correct expression because when I
build it I get the correct output path), but I’m getting a DIFFERENT store
path than the one with nix-store --query --deriver.

/nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv

To summarise:

$ cat $(nix-store --query --deriver /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23)
cat: /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv: No such file or directory

$ nix-build -E '(import <nixpkgs> {}).bash'
/nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23

$ nix-instantiate -E '(import <nixpkgs> {}).bash'
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv

The reason I stumbled upon this issue in the first place is that vulnix was
crashing because it couldn’t find the .drv files for dependencies in the
closure I was scanning.

Oddly enough this problem did not occur to me on a revision from unstable: ce8c243699808355e0a550c7bdebf4ca94f0480d
But when upgrading to the most recent commit in stable-19.03 the problem surfaced: 07e2b59812de95deeedde95fb6ba22d581d12fbc
And this happens with multiple dependencies, not just with the one I used in
the example.

What should my approach be to find the missing .drv file?
Why does this problem happen in the first place?
Why does nix-instantiate on the nix expression and nix-store --query --deriver on the build output, return different .drv files?

Thanks for the help!

vcunat · March 4, 2019, 8:50am

There may be multiple different *.drv that produce the same output. That’s quite normal. For example, some dependency may change the URL from which it downloads or update the utilities by which it downloads – that will change the “recipe” but not any output, thanks to fixed-output-ness of source fetchers.

psychic · March 4, 2019, 10:47am

Indeed, that answers (3) by giving an example of when the case is as such. Thanks!

I’m still curious about (1),(2) and why do .drv files disappear in the first place (when nix is not set to remove .drv files on GC)

danbst · March 4, 2019, 12:47pm

Wonder if --deriver uses local DB or cache.nixos.org DB. Can you try method described in Introspect nix-shell? - #2 by danbst ?

zimbatm · March 4, 2019, 1:17pm

Not a direct reply but I think it’s possible to pull .drv files from the binary cache using nix-store --realize /nix/store/...drv. That might solve the more immediate problem.

psychic · March 5, 2019, 1:04am

[nix-shell:~]$ sqlite3 "/nix/var/nix/db/db.sqlite"       "select deriver from ValidPaths where path = '/nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23'"
/nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv

Returns the same result. I’m guessing that --deriver does use the local DB.

psychic · March 5, 2019, 1:05am

@zimbatm Good idea. But did not work for me.

$ nix-store --realize /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv
don't know how to build these paths:
  /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv
cannot build missing derivation '/nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv'
error: build of '/nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv' failed

psychic · March 6, 2019, 1:21am

Any ideas about why derivations were missing in the first place? I’ve looked at garbage collection settings and they aren’t seem to be relevant to the problem.

I assume this is not vulnix’s fault but I wonder if other people who use vulnix come across this missing .drv problem. Perhaps those people don’t? Or perhaps they don’t use vulnix? In which case I’d like to know what is their policy regarding security vulnerabilities? Are they using an alternative tool? (If need be I’ll start another thread).

On the one hand I’d like to introduce Nix into my company, on the other hand this does not inspire confidence. I know nix is in used in production by other companies, and that is why I’m asking.

EDIT: I found out a way to circumvent the problem that may shed some light as to the origin of the problem (still unclear to me however)

It seems that running vulnix <build-output> works even for the derivations that were before reportedly as missing. The vulnix invocation I ran before that was generating the error looks like:

vulnix -R $(nix-store -qR <build-output>) and that one stumbled upon the missing derivation.
The reason I used this invocation is that vulnix returns advisories regarding build-time dependencies deep down the build chain which adds a lot of noise to the advisories I care about more which are the run-time dependencies.

Does this shed some light as to why this is happening in the first place?

EDIT: I think I understand the reason, feel free to confirm or reject the theory:
nix-store -qR <build-output> returned run-time dependencies, some of which were taken from binary cache and with them the deriver of the binary cache was registered in my local database (this clashes with the fact that @zimbatm 's method of realizing the derivation did not work, however.
Conversely when running vulnix on directly it used my derivation file for it and from that it got all dependencies, inside which run-time dependencies are as well, through their derivations that must be on my computer since it generated them to start the build in the first place.

That is my guess, for now.

7c6f434c · March 6, 2019, 10:30am

Any ideas about why derivations were missing in the first place? I’ve looked at garbage collection settings and they aren’t seem to be relevant to the problem.

Do you have keep-derivations=true in nix.conf? This option specifies whether a derivation can be GC-ed when its output is present in store.

On the one hand I’d like to introduce Nix into my company, on the other hand this does not inspire confidence.

This is a deliberate and documented configurable behaviour, so please do not take it as evidence that Nix fails to achieve GC behaviours it aims at.

If the way it is described in Introduction - Nix Reference Manual or some problem with discoverability are the reasons for lack of confidence, then indeed I can only say I read all the three (Nix, Nixpkgs, NixOS) manuals before trying out NixOS and liked them, but different people prefer different styles.

psychic · March 7, 2019, 1:18am

Thanks for your reply.

Do you have keep-derivations=true in nix.conf? This option specifies whether a derivation can be GC-ed when its output is present in store.

Yes I do, I added it for extra-safety but the default value is true so I doubt it has to do with the problem.

This is a deliberate and documented configurable behaviour, so please do not take it as evidence that Nix fails to achieve GC behaviours it aims at.
If the way it is described in Introduction - Nix Reference Manual or some problem with discoverability are the reasons for lack of confidence, then indeed I can only say I read all the three (Nix, Nixpkgs, NixOS) manuals before trying out NixOS and liked them, but different people prefer different styles.

The lack of confidence was regarding missing .drv files that are not missing due to GC, at least according to what is documented. But perhaps my guess in the previous post is the reason as to why this happened.

Of course I read all of nixos.org’s manuals and basically all written manual/guide/tutorial that exists online. I’m at the stage where to learn more I have to read code, and unfortunately that code is not statically-typed

danbst · March 7, 2019, 1:18pm

Here’s what Hydra says for this package: Hydra - Build 89205122 of job nixpkgs:trunk:runtimeShellPackage.x86_64-linux
And here’s what Hydra says for it’s (non-existing) drv: Hydra - Build 88528297 of job nixpkgs:staging-next:runtimeShellPackage.x86_64-linux

So, somehow information about derivers is fetched from staging-next.

danbst · March 7, 2019, 1:35pm

Hydra says, that /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23 was

Cached from: nixpkgs:cross-staging:avr.libcCross.x86_64-linux build 88317396

Looking into that build → Build Steps → Bash build details we see:

Below is the build log of derivation /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv. It was built on root@hydra.ewi.tudelft.nl.

However, on original Details page we see:

Derivation store path: /nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv

How I understand it: hydra instantiates derivation, computes it’s store-path, finds it in some previous build, reuses that binary cache together with old .drv information. When you fetch NAR, you fetch also old .drv information…

So, --deriver is pretty much flaky. I don’t know why nix-store --realise doesn’t see it. Probably Hydra has some cleanup script, which makes 1-to-1 mappings between store output and derivation?

psychic · March 8, 2019, 2:08am

Thanks for the analysis, it is quite educational!

May I ask how did you go about deciding the specific hydra build to look at?
Also what if the package wasn’t as major as bash how would you find its details? The search at the top does not work for specific derivations. In the case of bash as luck would have it when going to any build and checking build dependencies bash comes out at the top so that was easy to find.

danbst · March 8, 2019, 8:44am

@psychic I’ve just inserted /nix/store/… path into search in top. It indeed doesn’t work for many locally built derivations, but those should have correct acompany .drv-s locally.

Also what if the package wasn’t as major as bash how would you find its details?

I hope Hydra can search everything that was built recently. Some old results can be collected, yes. Not sure how long are build information preserved.

I’ve added bug report to Nix Rewrite derivers in local DB for `keep-derivations=true` · Issue #2712 · NixOS/nix · GitHub

So, fix until issue is resolved, is:

get .drv of top derivation you want to verify via Vulnix

➜ nix-instantiate '<nixpkgs/nixos>' --arg configuration '{ boot.isContainer = true; }' -A system                                                                       
/nix/store/pqxy99yaqry0nhwn51dxdrvj01fzmrgi-nixos-system-nixos-19.03pre167327.11cf7d6e1ff.drv

list all referenced .drvs:

$ nix-store -qR /nix/store/pqxy99yaqry0nhwn51dxdrvj01fzmrgi-nixos-system-nixos-19.03pre167327.11cf7d6e1ff.drv
/nix/store/6v3y7s4q4wd16hsw393gjpxvcf9159bv-patch-shebangs.sh
/nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh
/nix/store/9ny6szla9dg61jv8q22qbnqsz37465n0-multiple-outputs.sh
/nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh
/nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv
...

for each of them run nix-store -q --outputs
for each collected output check if it is present in local Nix DB, and if present, update deriver (SQL query)

I’ve written a small Bash script to find out all inconsistencies, you may find it useful.

#!/usr/bin/env bash

top=$(nix-instantiate '<nixpkgs/nixos>' --arg configuration '{ boot.isContainer = true; }' -A system 2>/dev/null)
nix-store -qR $top 2>&1 | grep -E 'drv$' | while read drv; do
  nix-store -q --outputs $drv 2>&1 | grep -v warning | while read output; do
    if [ ! -e $output ]; then
      #echo " => $drv is not built locally"
      continue
    fi
    deriver="$(nix-store -q --deriver $output 2>/dev/null)"
    [[ "$deriver" != "$drv" ]] && (
        echo " !!! => mismatch for $output: deriver should be $drv"
    ) || (
        echo "$output is correct"
    )
  done
done

This produces ALOT of mismatches on my system. Feel free to use output of that script to fire correct SQLs to nix local DB

psychic · March 8, 2019, 9:02am

@danbst Thanks!

I’ll try it out! It’ll be great if it fixes the inconsistencies!

What I did for now is I ran vulnix on the top deriavation, outputted in JSON format and then intersected it with the output of $(nix-store -qR /nix/store/...) on my derivation in order to have only run-time dependencies.

fzakaria · June 23, 2024, 6:39pm

I posted Why isn't deriver and realize identity functions (see example) which is maybe related.

This UX is terrible for anyone trying to understand Nix.

> which htop
/Users/fzakaria/.nix-profile/bin/htop

> nix derivation show $(which htop) | jq -r "keys[0]"
/nix/store/hgnz7dz10vi0n81s341iihqm1azq4lb6-htop-3.2.2.drv

> nix-store --query --deriver $(which htop)
/nix/store/nshxi52ldsv03sc8k810kw5j3hwa69bv-htop-3.2.2.drv

Those two derivations are different and only one exists on my system.