Evaluating possibly-nonfree derivations

Help
1

I’m trying to write a script that produces the derivation path and attribute path for all nixpkgs packages. While nix-env -qaP --drv-path gets me a long way, I thought it would be fun, educational and more flexible (for example, I’d also like to see all haskellPackages later) to write a small nix script.

So far I got:

{ pkgs ? import <nixpkgs> {} }:

with builtins;
with pkgs;
with pkgs.lib;

let
  m = mapAttrs (name: value:
    let
      e = (tryEval value);
    in
      optionalString (e.success) (optionalString (hasAttr "drvPath" e.value) (name + " -> " + e.value.drvPath))
  ) pkgs;
in
  writeTextFile {
    name = "test";
    text = (toJSON m);
  }

However running this with nix-instantiate test.nix produces an error because some of those are ‘unfree’ packages. Does this mean tryEval doesn’t catch those kinds of errors, or is there something about nixs’ laziness that I am misunderstanding? (adding --show-trace didn’t really enlighten me)

2

If you try to do it in nix, you do have a somewhat fundamental problem: how do you keep the output from depending on every package in nixpkgs, and thus requiring you to actually build/substitute them all? (keep in mind some are broken…)

Are you sure you want to do this in nix?

If you’re really sure, you can use builtins.unsafeDiscardStringContext to get the store paths as strings without keeping the dependency on the object they reference.

However, I don’t think this solves the unfree/marked broken problem, although it does prevent actual builds. You’ll have to configure nixpkgs to allow unfree and broken and insecure packages in order to get around those eval-time checks.

Are you sure you want to do this in nix?

3

No, I just figured it might be nice - if possible :slight_smile:. I definitely wasn’t intending to build everything, I was optimistically assuming as long as I was only referencing things that can be determined by ‘just’ evaluating the nix expressions without actually building the derivation (like drvPath), the fact that Nix is lazy might avoid actually having to build those. Looks like I was wrong there - though it does seem to work like that in nix repl?

I would be interested in understanding why/how this (doesn’t) work in nix, but also on how you’d recommend approaching this instead :wink:

4

The only reason it works in the repl is because the string is the end goal in and of itself. If the string makes its way into a derivation of any kind, the dependency goes with it, which is good, because it’s normally what you want.

Here, you’re doing something rather strange in that you want a link to these packages yet you don’t want a real link to them. You essentially want to create dangling pointers, in the terminology of memory management. Nix isn’t designed to do that, though as I mentioned above it can be done with unsafeDiscardStringContext.

A bigger question here is why you want to list store paths that won’t actually exist on your system. It seems like whatever goal you actually wanted this for could be achieved in another way. However, if you really want to do this, it seems like an external script which calls nix a number of times to get various data might be a good idea.

More generally, it’s worth knowing that nixpkgs is really not designed to by walked indiscriminately, as the error messages that show up when you try to evaluate broken packages will tell you. Things that look at all of nixpkgs do many distinct evals in order to contain the effect of packages which fail to evaluate. The degree of safe walking that’s possible is roughly what the nix tools like nix search already do.

5

Ofborg does this to figure out how many derivations are changed by a Nixpkgs PR. You can find the script here: https://github.com/NixOS/ofborg/blob/2beea85aedd48baa7596d8254d4452033acc96af/ofborg/src/outpaths.nix

1 Like
6

Sure. In this case I don’t really need this to be a derivation, I’m just interested in the mapping between attribute paths and derivation paths

Well, I’m ‘just’ referring to the .drv, not to the actual dependency - but I guess the .drv refers to its output.

Yes, thanks for that!

I want to make the reports such as the one at https://arnout.engelen.eu/nixos-iso-gnome-r13y/ produce more ‘human-readable’ attribute paths when possible.

I don’t think I want much beyond that, just some more flexibility (e.g. also entering the haskellPackages even though they are dontRecurseIntoAttrs), and it seemed overkill to create a whole C++ project just to achieve that.

Thanks, I’ll have a look! Also relying on nix-env I see - I guess I’ll finally have to learn what exactly that does :laughing:

2 Likes
7

So indeed it’s quite easy to get the desired functionality and flexibility with nix-env:

#!/usr/bin/env nix-shell
#!nix-shell -p nix -i "nix-env -qaP --no-name --drv-path -f"
{ pkgs ? import <nixpkgs> {} }:

(pkgs // {
  haskellPackages = pkgs.recurseIntoAttrs
    (if pkgs.stdenv.hostPlatform.isGhcjs
     then pkgs.haskell.packages.native-bignum.ghcHEAD
     # Prefer native-bignum to avoid linking issues with gmp
     else if pkgs.stdenv.hostPlatform.isStatic
     then pkgs.haskell.packages.native-bignum.ghc92
     else pkgs.haskell.packages.ghc92);
})

… though it seems a bit weird to reach for nix-env for this (as I’m not interested in ‘user environments’ at all). Also it takes about 1m30s on my machine, which is definitely a lot better than ‘building/downloading everything’, but still feels like maybe we aren’t taking full advantage of the Nix ‘laziness’.

So: direct problem solved, but some room for further exploration :slight_smile:

1 Like