;
tryEval = builtins.tryEval;
in
lib.mapAttrs (k: v:
let name = (tryEval v.name or "");
out = (tryEval v.outPath or "");
in {
name = name.value;
out = out.value;
}
) pkgs
```
This one has some odd failure:
```auto
❯ nix eval -f example.nix
error: store path '/nix/store/sqpnqdjc2mpzixcy2kbbgkddf8awhq99-nixpkgs-21.05pre278688.c0e88185200' is not allowed to have references
(use '--show-trace' to show detailed location information)
```
## nix-eval-jobs
Tried [nix-eval-jobs](https://github.com/nix-community/nix-eval-job) but it doesn’t emit _outPath_.
Was fairly easy to add it via this [pull request](https://github.com/nix-community/nix-eval-jobs/pull/17) but it doesn’t work with recursive packages and some other errors.
```auto
error: OVMF-CSM has been removed in favor of OVMFFull
{"attr":" __splicedPackages.__ splicedPackages. __splicedPackages.__ splicedPackages.OVMF-CSM","error":"error: OVMF-CSM has been removed in favor of OVMFFull"}
error: OVMF-secureBoot has been removed in favor of OVMFFull
```
@matthewbauer had a great [blog post](https://matthewbauer.us/blog/all-the-versions.html) about doing almost similar which was very good for inspiration.
PS: If someone wants to help me with this project, please reach out 🙂 It’s in the same vein as [Searching and installing old versions of Nix packages – Marcelo Lazaroni – Developing for the Interwebs](https://lazamar.github.io/download-specific-package-version-with-nix/)
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [January 5, 2022, 11:47pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/2 "2022-01-05T23:47:32Z")
nix build .#cowsay
nix path-info -rS ./result
is fun!
nix-instantiate --eval --json --strict -A cowsay
---
### Author:  [@kamadorueda](https://discourse.nixos.org/u/kamadorueda)
#### Post date: [January 6, 2022, 12:23am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/3 "2022-01-06T00:23:25Z")
You may want to check out this project if you haven’t:
> **[GitHub - on-nix/pkgs: Curated information about every package on Nix!](https://github.com/on-nix/pkgs)**
>
> Curated information about every package on Nix! Contribute to on-nix/pkgs development by creating an account on GitHub.
For your specific use case it would be:
- Generate the list of commits with git
- For each commit:
- Check it out
- Use `nix-env --json` and `jq` to compute all offered attributes by this nixpkgs set (something like [this](https://github.com/on-nix/pkgs/blob/730c522af5d6944c4130b267eb6b913b9b670d48/data/nixpkgs/attrs.json)
- For each attribute use nix-instantiate to get its out-path
Expect lots of storage to be used since what you want to compute is a lot of information, but you can implement a bisection algorithm or perform sampling and still get good enough results. You can exploit the fact that not all packages change in every commit
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [January 6, 2022, 1:54am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/4 "2022-01-06T01:54:01Z")
Wow what an awesome search engine.
Basically what I want to add @kamadorueda is adding the **outPath** so you can also search by that instead of the name or version.
Is this something I can contribute to your project?
happy to discuss this over email or DM
---
### Author:  [@kamadorueda](https://discourse.nixos.org/u/kamadorueda)
#### Post date: [January 6, 2022, 2:26am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/5 "2022-01-06T02:26:02Z")
Yeah, sounds awesome, definitely
The constraints on this project are the 10GB per repo that Github allows, and that we don’t have Github Actions at the moment. Also the search engine (run client-side, stored on Github Pages) has a limit per page of 100MB, so I’ve not been able to add to the search index features like “search by package output path (like /bin/nix as [nix-index](https://github.com/bennofs/nix-index))” because it would exceed the limit. The data is already in the repository, but not under the website
So, just take that into account if you decide to contribute, I’ll also be happy if you decide to fork or just take some ideas from it. It’s free, libre and open source!
With enough resources (S3 buckets, some machines to run periodic updates, a proper search engine like elasticsearch) one can definitely make something awesome out of this data. But I have not found a way to make paying those expenses sustainable, so that’s why I decided to go with Github free tier
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [January 6, 2022, 3:11am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/6 "2022-01-06T03:11:07Z")
I know how to get output path for an individual attribute.
I was seeking help on how to generate it for _all attribites_ in nixpkgs
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [January 6, 2022, 3:18am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/7 "2022-01-06T03:18:49Z")
@kamadorueda thanks for the advice.
Do you know of a better way to generate the outPath in a single invocation or by evaluation of a nix script?
(like the one I was attempting in my original message).
It would need just need to evaluate the derivation for the outPath and not actually build it…
---
### Author:  [@kamadorueda](https://discourse.nixos.org/u/kamadorueda)
#### Post date: [January 6, 2022, 4:15am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/8 "2022-01-06T04:15:14Z")
> [@fzakaria](#):
>
> Do you know of a better way to generate the outPath in a single invocation or by evaluation of a nix script?
I’ve tried to do it, and it’s not possible. There are a few builtins.abort and even a few syntax errors on Nixpkgs that cannot be handled inside a Nix expression. You cannot evaluate it all. On the other hand Nix-env and hydra know very well how to handle those, so that’s the only way, and thus computing what you want require many steps
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [January 6, 2022, 5:31am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/9 "2022-01-06T05:31:01Z")
Captured my approach so far here: [https://fzakaria.com/2022/01/05/computing-all-output-paths-for-every-attribute-in-nixpkgs.html](https://fzakaria.com/2022/01/05/computing-all-output-paths-for-every-attribute-in-nixpkgs.html)
It’s far from perfect and in fact **very slow** lol
---
### Author:  [@kamadorueda](https://discourse.nixos.org/u/kamadorueda)
#### Post date: [January 6, 2022, 5:53am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/10 "2022-01-06T05:53:35Z")
Hard problems are the best ones!
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [January 6, 2022, 12:29pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/11 "2022-01-06T12:29:56Z")
Remember nothing is slow if you can distribute it over 1000 machines.
🙂
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [January 6, 2022, 5:48pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/12 "2022-01-06T17:48:13Z")
Ok, I had a brainwave.
Would a postbuildhook help you?
[https://nixos.org/manual/nix/stable/advanced-topics/post-build-hook.html](https://nixos.org/manual/nix/stable/advanced-topics/post-build-hook.html)
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [January 6, 2022, 7:48pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/13 "2022-01-06T19:48:43Z")
I don’t want to build everything though.
To calculate output paths is merely an evaluation…
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [January 6, 2022, 7:52pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/14 "2022-01-06T19:52:05Z")
you want your cake and eat it!! 🙂 . Hard problem… hope you get a solution.
---
### Author:  [@adisbladis](https://discourse.nixos.org/u/adisbladis)
#### Post date: [April 26, 2022, 10:00am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/15 "2022-04-26T10:00:12Z")
> [@fzakaria](#):
>
> Tried [nix-eval-jobs](https://github.com/nix-community/nix-eval-job) but it doesn’t emit _outPath_ .
> Was fairly easy to add it via this [pull request](https://github.com/nix-community/nix-eval-jobs/pull/17) but it doesn’t work with recursive packages and some other errors.
I have just added support for `recurseForDerivations` to nix-eval-jobs in [Add support for recurseForDerivations by adisbladis · Pull Request #62 · nix-community/nix-eval-jobs · GitHub](https://github.com/nix-community/nix-eval-jobs/pull/62) which makes it actually possible to eval nixpkgs.
Example expression:
```auto
import {
allowAliases = false;
}
```
Edit:
@fzakaria If it wasn’t clear `nix-eval-jobs` now also contains output paths.
---
### Author:  [@ryantm](https://discourse.nixos.org/u/ryantm)
#### Post date: [April 27, 2022, 3:24am UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/16 "2022-04-27T03:24:16Z")
I somehow missed this thread earlier. The goal here sounds pretty similar to what ofborg does to get the number of rebuilds:
>
We also discussed this a bit here:
[How to find all reverse transitive dependencies of a package?](https://discourse.nixos.org/t/how-to-find-all-reverse-transitive-dependencies-of-a-package/18407)
---
### Author:  [@fzakaria](https://discourse.nixos.org/u/fzakaria)
#### Post date: [April 27, 2022, 2:17pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/17 "2022-04-27T14:17:50Z")
I think I covered in the blog post why that script doesn’t work (it’s been a while so my memories a bit hazy).
It’s a bit interesting that you need such a crazy script or tools just to evaluate all attributes of the set.
nix-env can do it but you can’t do it natively with the repl.
---
### Author:  [@misuzu](https://discourse.nixos.org/u/misuzu)
#### Post date: [July 1, 2022, 12:04pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/18 "2022-07-01T12:04:53Z")
Something like this could be very useful for binary cache mirror 🤔
---
### Author:  [@jeff-hykin](https://discourse.nixos.org/u/jeff-hykin)
#### Post date: [October 24, 2023, 9:35pm UTC](https://discourse.nixos.org/t/evaluate-all-outpaths-for-every-package-in-nixpkgs/16995/19 "2023-10-24T21:35:41Z")
AFAIK no script, even those used by nix-env and hydra, were able to fully list all derivations. Which is why derivations like `darwin.apple_sdks.frameworks.CoreServices` have never showed up on [search.nixos.org](http://search.nixos.org).
I’ve spent a LOT of time on this, and have a script for it!
But first I need to mention:
- Even excluding the syntax errors and abort problems
- Even excluding the equality checking problem; which is that nix can’t tell if two functions are equal (e.g. if `a=x: x` then following is false: `a == a`) and therefore nix can’t tell if two attr sets are deeply equal, and therefore can’t tell if an attr has already been explored/seen earlier in the tree. (e.g. makes a finite tree with back references look like an infinte tree)
- Even then the actual attr tree is a non-converging infinite tree. E.g. its like a fractal. As the recursion deepens, some hash values keep changing.
- Even if we explore the tree using BFS with a hardcoded max depth (packages can occur at any depth), the amount of ram required explodes (I maxed out a 256Gb machine)
So.
To get around all those, I made [a Deno script](https://github.com/jeff-hykin/snowball/blob/6c3dcda90c93d25ec7d0f4646243cc7df872a7f1/scrape.versions.js) that abuses the hell out of 40 concurrent `nix repl` subprocesses, and uses iterative deepening to get around the BFS memory problem.
It has some config vars at the top for:
- which nixpkgs hash
- a start attr-path (defaults to root)
- child attr names to ignore (this list can be empty but having a few things in it makes the search MUCH faster)
- number of concurrent subprocesses (dont use too many or there will be nix locking errors)
Finally:
- It writes the attr path, child attr names, and some other info to a file, one line per attr-path.
NOTE: I think I used nix v2.11 with it
I think this is the only one that truly prints every attr path. It probably takes 24 hours to get to a depth of 4, so its still not practical for most things, but its progress.
It is really sad that it takes this much effort and this hacky of a solution to literally just iterate over all packages.