Slow evaluation of recursive dependency resolution (spago2nix-ree)

thought2 · April 8, 2020, 1:36pm

I’m developing this ‘*2nix’ tool for the language purescript: GitHub - thought2/spago2nix-ree: Generates a lock file for spago projects that can be built with nix

My main motivation was that dependency resolution is done by nix - so that only actual project sources have to be rebuilt if something changes in your project sources. In the already existing spago2nix this is not the case, everything has to be recompiled on any change.
With spago2nix-ree dependencies that are built once are stored in the nix store so you never have to build them again.

I’m glad that this feature actually already works! But there’s one huge problem: The evaluation phase is very long. For a project with about 10 dependencies one rebuild (without any code change) already takes one minute. No good It’s really just evaluation, the dependencies don’t get rebuilt or so. Tbh, I thought that this would not happen because of the lazy evaluation and caching of derivations which inputs did not change.

Is there anything that I can do to that? The recursive build function can be found here: spago2nix-ree/build-package.nix at d9a6c9d767082a9a4b6134fc65989d7c911c0bef · thought2/spago2nix-ree · GitHub

FRidh · April 8, 2020, 1:53pm

maybe you can store the dependencies of a single package in the passthru and have the recursion function use that?

thought2 · April 8, 2020, 3:46pm

Awesome! That did the trick. Now the evaluation time is almost instantaneous.
So I’m passing down (and up again) a cache attr set that holds the built dependencies, so the lookup in the eval phase is much faster.

github.com

thought2/spago2nix-ree/blob/5a70c55c154852cfb45f19eb0cfc66769ebcf591/build-package.nix#L26

    
      
          let
            forEach = f: xs: builtins.concatStringsSep "\n" (map f xs);
          
          
  buildPackage = { spagoPackages, package }:
              (buildPackage' {
                inherit spagoPackages;
                inherit package;
                cache = { };
              }).package;
          
          
  buildPackage' = { spagoPackages, package, cache }:
              let
          
          
      dependencies =
                  map (key: builtins.getAttr key spagoPackages) package.dependencies;
          
          
      source = package.source;
          
          
      dependenciesBuilt = builtins.foldl' (memo: unbuildPackage:
          
          
        if (builtins.hasAttr unbuildPackage.name memo.cache) then