Creating a derivation that depends on all non-broken Haskell packages

Help
1

I’m trying to create a derivation that I can use to get a nix-shell in which all non-broken Haskell packages are in the GHC package db (such that the ghc-pkg list command outputs all these packages).

To achieve this I have created the below .nix file:

let
  hs-pkgs = pkgs.haskell.packages.ghc962;
  pkgs =
    let
      src = builtins.fetchTarball {
        url = "https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz";
        sha256 = "12mbg7rj7yybwqc5xx01r5ryq56wn3igyrypml27agwwln1x2par";
      };
    in import src { system = "x86_64-linux"; };
  nonBroken = name: x: !((x.meta or {}).broken or false);
  hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs nonBroken hs-pkgs;
in hs-pkgs-filtered

This, however, does not appear to work. I think the problem is that a package may be marked non-broken while one of its dependencies are broken. For example, the package zuramaru itself is not marked broken but it depends on the broken package throwable-exceptions which causes an evaluation error.

How can I get a shell with all non-broken Haskell packages installed in the GHC package db?

2

toString pkg evals the package as if you instantiated it directly; including throws for broken deps or unfreeness.

Wrap that in a tyrEval and that should allow you to filter out packages with broken deps.

1 Like
3

How should I use tryEval? The following

let
  hs-pkgs = pkgs.haskell.packages.ghc962;
  pkgs =
    let
      src = builtins.fetchTarball {
        url = "https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz";
        sha256 = "12mbg7rj7yybwqc5xx01r5ryq56wn3igyrypml27agwwln1x2par";
      };
    in import src { system = "x86_64-linux"; };
  nonBroken = name: x: !((x.meta or {}).broken or false);
  hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs nonBroken hs-pkgs;
in
pkgs.lib.attrsets.filterAttrs (name: x: (builtins.tryEval x).success) hs-pkgs-filtered

gives this error during nix-build

error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'AC-Vector-Fancy-2.4.0'
         whose name attribute is located at /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/pkgs/stdenv/generic/make-derivation.nix:303:7

       … while evaluating attribute 'propagatedBuildInputs' of derivation 'AC-Vector-Fancy-2.4.0'

         at /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/pkgs/stdenv/generic/make-derivation.nix:357:7:

          356|       depsHostHostPropagated      = lib.elemAt (lib.elemAt propagatedDependencies 1) 0;
          357|       propagatedBuildInputs       = lib.elemAt (lib.elemAt propagatedDependencies 1) 1;
             |       ^
          358|       depsTargetTargetPropagated  = lib.elemAt (lib.elemAt propagatedDependencies 2) 0;

       error: Package ‘AC-Vector-2.3.2’ in /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/pkgs/development/haskell-modules/hackage-packages.nix:246 is marked as broken, refusing to evaluate.
4

As I mentioned, you need to toString a drv to make it throw on broken deps (or itself being broken) the same way it would on direct eval.

nix-repl> (builtins.tryEval (haskellPackages.AC-Vector)).success
true

nix-repl> (builtins.tryEval (toString (haskellPackages.AC-Vector))).success
false
1 Like
5

When I try this I get a type error because some of the attributes in haskell.packages.ghc962 (e.g. buildHaskellPackages) point to something that’s not a derivation — and therefore cannot be converted to a string.

I guess I need a isDerivation function to remove all the stuff that is not a derivation. But I don’t know how to write this.

Code:

let
  hs-pkgs = pkgs.haskell.packages.ghc962;
  pkgs =
    let
      src = builtins.fetchTarball {
        url = "https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz";
        sha256 = "12mbg7rj7yybwqc5xx01r5ryq56wn3igyrypml27agwwln1x2par";
      };
    in import src { system = "x86_64-linux"; };
  nonBroken = name: x: !((x.meta or {}).broken or false);
  hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs nonBroken hs-pkgs;
  f = name: x: builtins.typeOf x == "set" && (builtins.tryEval (toString x)).success;
in
pkgs.lib.attrsets.filterAttrs (name: x: builtins.trace (name + " " + builtins.typeOf x) (f name x)) hs-pkgs-filtered

Output:

trace: 4Blocks set
trace: ABList set
trace: AC-Angle set
trace: AC-Boolean set

[...]

trace: bugsnag-yesod set
trace: bugzilla-redhat set
trace: build-env set
trace: buildFromCabalSdist lambda
trace: buildHaskellPackages set
error:
       … from call site

         at /home/rune/code/haskell-graph/demo2.nix:14:1:

           13| in
           14| pkgs.lib.attrsets.filterAttrs (name: x: builtins.trace (name + " " + builtins.typeOf x) (f name x)) hs-pkgs-filtered
             | ^
           15|

       … while calling 'filterAttrs'

         at /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/lib/attrsets.nix:305:5:

          304|     # The attribute set to filter
          305|     set:
             |     ^
          306|     listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));

       … while calling anonymous lambda

         at /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/lib/attrsets.nix:306:29:

          305|     set:
          306|     listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
             |                             ^
          307|

       … from call site

         at /nix/store/0iqihbvgw5r24rvff4mkikzz0v0lalh1-source/lib/attrsets.nix:306:62:

          305|     set:
          306|     listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
             |                                                              ^
          307|

       … while calling anonymous lambda

         at /home/rune/code/haskell-graph/demo2.nix:14:38:

           13| in
           14| pkgs.lib.attrsets.filterAttrs (name: x: builtins.trace (name + " " + builtins.typeOf x) (f name x)) hs-pkgs-filtered
             |                                      ^
           15|

       … from call site

         at /home/rune/code/haskell-graph/demo2.nix:14:90:

           13| in
           14| pkgs.lib.attrsets.filterAttrs (name: x: builtins.trace (name + " " + builtins.typeOf x) (f name x)) hs-pkgs-filtered
             |                                                                                          ^
           15|

       … while calling 'f'

         at /home/rune/code/haskell-graph/demo2.nix:12:13:

           11|   hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs nonBroken hs-pkgs;
           12|   f = name: x: builtins.typeOf x == "set" && (builtins.tryEval (toString x)).success;
             |             ^
           13| in

       error: cannot coerce a set to a string
6

Looks like I can use the type attribute. This code seems to work:

let
  hs-pkgs = pkgs.haskell.packages.ghc962;
  pkgs =
    let
      src = builtins.fetchTarball {
        url = "https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz";
        sha256 = "12mbg7rj7yybwqc5xx01r5ryq56wn3igyrypml27agwwln1x2par";
      };
    in import src { system = "x86_64-linux"; };
  nonBroken = name: x: !((x.meta or {}).broken or false);
  hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs nonBroken hs-pkgs;
  f = name: x: (x.type or "?") == "derivation" && (builtins.tryEval (toString x)).success;
in
pkgs.lib.attrsets.filterAttrs (name: x: builtins.trace (name + " " + x.type or "?") (f name x)) hs-pkgs-filtered
2 Likes
7

That’s lib.isDerivation!

An alternative solution would be to filter out all packages that have meta.hydraPlatforms == [] which in practice amounts to the same thing you are trying and is probably a bit faster. (Technically it filters out all packages not built on Hydra which is a) versioned packages which often don’t work anyways and b) packages that depend on packages that are broken or are broken themselves. This is not a strong guarantee, but indeed how the Haskell infrastructure has worked for years now.)

1 Like
8

Thanks for the tip @sternenseemann! Unfortunately, I can’t get either the solutions to build.

My Nix code looks as follows:

pkgs.nix:

let
  release-23-05 =
    import (builtins.fetchTarball {
      url = "https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz";
      sha256 = "12mbg7rj7yybwqc5xx01r5ryq56wn3igyrypml27agwwln1x2par";
    });
  unstable =
    import (builtins.fetchTarball {
      url = "https://github.com/nixos/nixpkgs/archive/ca6912ef4e004e09637dcbeed71dd352f11aaa76.tar.gz";
      sha256 = "06s217ci10z75ab9w88qpl9wjbsjk28pz3dgl21hxfa0az71q71v";
    });
in
  unstable

non-broken-haskell-packages.nix:

{ pkgs ? import ./pkgs.nix { system = "x86_64-linux"; }
, get-hs-pkgs ? pkgs: pkgs.haskell.packages.ghc962
}:
let
  hs-pkgs = get-hs-pkgs pkgs;
  isNonBroken = name: x: !(x.meta.broken or false)
  hs-pkgs-filtered = pkgs.lib.attrsets.filterAttrs isNonBroken hs-pkgs;
  isNonBrokenDerivation = name: x:
    (x.type or "?") == "derivation" &&
      (builtins.tryEval (toString x)).success;
  non-broken =
    pkgs.lib.attrsets.filterAttrs
    isNonBrokenDerivation
    hs-pkgs-filtered;
  non-broken2 =
    pkgs.lib.attrsets.filterAttrs
    (name: x: (x.meta.hydraPlatforms or []) != [])
    hs-pkgs;
in
{ inherit non-broken pkgs hs-pkgs non-broken2; }

Trying to build the attribute non-broken in non-broken-haskell-packages.nix I get the error:

error: builder for '/nix/store/sslwlwk5hk2gdlv599vfiar06vm94h7j-CC-delcont-0.2.1.0.drv' failed with exit code 1;
       last 10 log lines:
       > [2 of 5] Compiling Control.Monad.CC.Seq ( Control/Monad/CC/Seq.hs, dist/build/Control/Monad/CC/Seq.o, dist/build/Control/Monad/CC/Seq.dyn_o )
       > [3 of 5] Compiling Control.Monad.CC ( Control/Monad/CC.hs, dist/build/Control/Monad/CC.o, dist/build/Control/Monad/CC.dyn_o )
       >
       > Control/Monad/CC.hs:82:13: error: [GHC-88464]
       >     Variable not in scope:
       >       ap :: CCT ans m (a -> b) -> CCT ans m a -> CCT ans m b
       >     Suggested fix: Perhaps use ‘map’ (imported from Prelude)
       >    |
       > 82 |     (<*>) = ap
       >    |             ^^
       For full logs, run 'nix log /nix/store/sslwlwk5hk2gdlv599vfiar06vm94h7j-CC-delcont-0.2.1.0.drv'.

And trying to build the attribute non-broken2 in non-broken-haskell-packages.nix this is the error I get:

Error: Setup: Encountered missing or private dependencies:
transformers >=0.2 && <0.6

error: builder for '/nix/store/izp7diszfnqg7nnfafm1dhqrydg8x6if-exception-transformers-0.4.0.11.drv' failed with exit code 1;
       last 10 log lines:
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > configureFlags: --verbose --prefix=/nix/store/5m6naakmr4fs5q06q1fplbc0iyfhqsyw-exception-transformers-0.4.0.11 --libdir=$prefix/lib/$compiler/lib --libsubdir=$abi/$libname --docdir=/nix/store/4k98rb918g54g3bb2vglml013pq1rqlz-exception-transformers-0.4.0.11-doc/share/doc/exception-transformers-0.4.0.11 --with-gcc=gcc --package-db=/build/tmp.tAN446ma9s/package.conf.d --ghc-options=-j8 +RTS -A64M -RTS --disable-split-objs --enable-library-profiling --profiling-detail=exported-functions --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --enable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --ghc-options=-haddock --extra-lib-dirs=/nix/store/00914xy1p14vd8qy23lrjd165rnxy3h2-ncurses-6.4/lib --extra-lib-dirs=/nix/store/g8l012l0q2xbl27da8zipg39mnpf40gb-libffi-3.4.4/lib --extra-lib-dirs=/nix/store/i59h6pqnqsfffp03dk0plvpq798ymhan-elfutils-0.189/lib --extra-lib-dirs=/nix/store/c6v3ix7r8gy6fgq1jssswkdk4xnk8fwk-gmp-with-cxx-6.2.1/lib
       > Using Parsec parser
       > Configuring exception-transformers-0.4.0.11...
       > CallStack (from HasCallStack):
       >   withMetadata, called at libraries/Cabal/Cabal/src/Distribution/Simple/Utils.hs:368:14 in Cabal-3.10.1.0:Distribution.Simple.Utils
       > Error: Setup: Encountered missing or private dependencies:
       > transformers >=0.2 && <0.6
       >
       For full logs, run 'nix log /nix/store/izp7diszfnqg7nnfafm1dhqrydg8x6if-exception-transformers-0.4.0.11.drv'.
9

The broken flags are based on whatever package set haskellPackages points to. Thus they are not accurate for any other package set (in fact it is planned to disable them for non-standard sets altogether, see hackage2nix: Correctly flag broken Haskell packages · Issue #554 · NixOS/cabal2nix · GitHub).

1 Like
10

Note that the set of marked as broken packages is not equal to the set of packages that do not build or work.

broken = false is not a guarantee that the package will build.

11

In haskellPackages it is a comparatively strong indication that the package will build on x86_64-linux, though.

12

FWIW, I ended up writing a simple bash script that builds each package to check whether it actually works. I couldn’t rely on any attribute to tell me whether it would build on my machine so this was the only solution I could find. Bash script in question:

#!/usr/bin/env bash

trap "echo Exited!; exit;" SIGINT SIGTERM

FILE="non-broken-haskell-packages.nix"
ATTR="non-broken"
TIMEOUT_SECONDS=3600

cat good.txt >> good.txt.bak
cat bad.txt >> bad.txt.bak
rm good.txt
rm bad.txt

for f in $(nix-instantiate -A $ATTR $FILE); do
  echo "##### Building $f ..."
  nix-build -j8 --timeout $TIMEOUT_SECONDS $f
  if [ $? -eq 0 ]; then
    echo $f >> good.txt
  else
    echo $f >> bad.txt
  fi
done