One of the dependencies: /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23
Trying to get its derivation with: nix-store --query --deriver /nix/store/53wi...
yields: /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv
This derivation, however, does not exist on my computer.
Despite having in the nix.conf option (which is on by default, but just in case) to keep derivation files.
So I thought of regetting this .drv file by calling nix-instantiate with the
relevant nix-expression (I know it’s the correct expression because when I
build it I get the correct output path), but I’m getting a DIFFERENT store
path than the one with nix-store --query --deriver.
$ cat $(nix-store --query --deriver /nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23)
cat: /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv: No such file or directory
$ nix-build -E '(import <nixpkgs> {}).bash'
/nix/store/53wi068kjrqfr2j0hzcxhbw2xaa990jr-bash-4.4-p23
$ nix-instantiate -E '(import <nixpkgs> {}).bash'
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv
The reason I stumbled upon this issue in the first place is that vulnix was
crashing because it couldn’t find the .drv files for dependencies in the
closure I was scanning.
Oddly enough this problem did not occur to me on a revision from unstable: ce8c243699808355e0a550c7bdebf4ca94f0480d
But when upgrading to the most recent commit in stable-19.03 the problem surfaced: 07e2b59812de95deeedde95fb6ba22d581d12fbc
And this happens with multiple dependencies, not just with the one I used in
the example.
What should my approach be to find the missing .drv file?
Why does this problem happen in the first place?
Why does nix-instantiate on the nix expression and nix-store --query --deriver on the build output, return different .drv files?
There may be multiple different *.drv that produce the same output. That’s quite normal. For example, some dependency may change the URL from which it downloads or update the utilities by which it downloads – that will change the “recipe” but not any output, thanks to fixed-output-ness of source fetchers.
Not a direct reply but I think it’s possible to pull .drv files from the binary cache using nix-store --realize /nix/store/...drv. That might solve the more immediate problem.
Any ideas about why derivations were missing in the first place? I’ve looked at garbage collection settings and they aren’t seem to be relevant to the problem.
I assume this is not vulnix’s fault but I wonder if other people who use vulnix come across this missing .drv problem. Perhaps those people don’t? Or perhaps they don’t use vulnix? In which case I’d like to know what is their policy regarding security vulnerabilities? Are they using an alternative tool? (If need be I’ll start another thread).
On the one hand I’d like to introduce Nix into my company, on the other hand this does not inspire confidence. I know nix is in used in production by other companies, and that is why I’m asking.
EDIT: I found out a way to circumvent the problem that may shed some light as to the origin of the problem (still unclear to me however)
It seems that running vulnix <build-output> works even for the derivations that were before reportedly as missing. The vulnix invocation I ran before that was generating the error looks like:
vulnix -R $(nix-store -qR <build-output>) and that one stumbled upon the missing derivation.
The reason I used this invocation is that vulnix returns advisories regarding build-time dependencies deep down the build chain which adds a lot of noise to the advisories I care about more which are the run-time dependencies.
Does this shed some light as to why this is happening in the first place?
EDIT: I think I understand the reason, feel free to confirm or reject the theory: nix-store -qR <build-output> returned run-time dependencies, some of which were taken from binary cache and with them the deriver of the binary cache was registered in my local database (this clashes with the fact that @zimbatm 's method of realizing the derivation did not work, however.
Conversely when running vulnix on directly it used my derivation file for it and from that it got all dependencies, inside which run-time dependencies are as well, through their derivations that must be on my computer since it generated them to start the build in the first place.
Any ideas about why derivations were missing in the first place? I’ve looked at garbage collection settings and they aren’t seem to be relevant to the problem.
Do you have keep-derivations=true in nix.conf? This option specifies whether a derivation can be GC-ed when its output is present in store.
On the one hand I’d like to introduce Nix into my company, on the other hand this does not inspire confidence.
This is a deliberate and documented configurable behaviour, so please do not take it as evidence that Nix fails to achieve GC behaviours it aims at.
If the way it is described in Introduction - Nix Reference Manual or some problem with discoverability are the reasons for lack of confidence, then indeed I can only say I read all the three (Nix, Nixpkgs, NixOS) manuals before trying out NixOS and liked them, but different people prefer different styles.
Do you have keep-derivations=true in nix.conf? This option specifies whether a derivation can be GC-ed when its output is present in store.
Yes I do, I added it for extra-safety but the default value is true so I doubt it has to do with the problem.
This is a deliberate and documented configurable behaviour, so please do not take it as evidence that Nix fails to achieve GC behaviours it aims at.
If the way it is described in Introduction - Nix Reference Manual or some problem with discoverability are the reasons for lack of confidence, then indeed I can only say I read all the three (Nix, Nixpkgs, NixOS) manuals before trying out NixOS and liked them, but different people prefer different styles.
The lack of confidence was regarding missing .drv files that are not missing due to GC, at least according to what is documented. But perhaps my guess in the previous post is the reason as to why this happened.
Of course I read all of nixos.org’s manuals and basically all written manual/guide/tutorial that exists online. I’m at the stage where to learn more I have to read code, and unfortunately that code is not statically-typed
Below is the build log of derivation /nix/store/2amas84m6w99q8lazd9fww6zrkakv6f9-bash-4.4-p23.drv. It was built on root@hydra.ewi.tudelft.nl.
However, on original Details page we see:
Derivation store path: /nix/store/zvy7mbpxqlplqpflqn5xk9szx25s4mhg-bash-4.4-p23.drv
How I understand it: hydra instantiates derivation, computes it’s store-path, finds it in some previous build, reuses that binary cache together with old .drv information. When you fetch NAR, you fetch also old .drv information…
So, --deriver is pretty much flaky. I don’t know why nix-store --realise doesn’t see it. Probably Hydra has some cleanup script, which makes 1-to-1 mappings between store output and derivation?
May I ask how did you go about deciding the specific hydra build to look at?
Also what if the package wasn’t as major as bash how would you find its details? The search at the top does not work for specific derivations. In the case of bash as luck would have it when going to any build and checking build dependencies bash comes out at the top so that was easy to find.
@psychic I’ve just inserted /nix/store/… path into search in top. It indeed doesn’t work for many locally built derivations, but those should have correct acompany .drv-s locally.
Also what if the package wasn’t as major as bash how would you find its details?
I hope Hydra can search everything that was built recently. Some old results can be collected, yes. Not sure how long are build information preserved.
I’ll try it out! It’ll be great if it fixes the inconsistencies!
What I did for now is I ran vulnix on the top deriavation, outputted in JSON format and then intersected it with the output of $(nix-store -qR /nix/store/...) on my derivation in order to have only run-time dependencies.
