Request explanation of cross-compilation-related package sets (buildPackages, targetPackages, etc)

[cdepillabout]

I created a derivation that rebuilt GHC with a newer version of clang than it was using by default:

How to get llvm-hs building on MacOSX in nixpkgs

We’re using llvm-hs at work. However, there is a problem building it on MacOSX. Through some trail-and-error, I was able to get it building. It was somewhat complicated, so I wanted to create a post here that explained how to do it. From what I could tell, llvm-hs needs to be built in an environment with the LLVM version it expects (so llvm-hs-8.0.0 needs LLVM-8 available). However, GHC is built with clang-7 (from LLVM-7), and this seems to mess up llvm-hs-8.0.0 when building. In order to …

In doing so, I had to override some of the cross-compilation-related arguments to GHC, like targetPackages and pkgsBuildTarget.

I have read the cross-compilation section of the nixpkgs manual, but I still don’t really understand what is going on.

I had a couple specific questions come up while doing this, and I’d be really interested in answers to them:

1. What are targetPackages, pkgsBuildTarget, buildPackages, etc? Where are they defined? When not doing cross compilation, do they all end up being the same thing?

2. Is it safe to override things like targetPackages, pkgsBuildTarget, etc in an overlay to nixpkgs? Is there anything I should watch out for when doing this?

   For example, if I wanted to make sure I built a compiler using a different stdenv, do I have to override the stdenv in the normal top-level attribute-set, the targetPackages attribute set, and the pkgsBuildTarget attribute set?

   I ended up directly modifying arguments to GHC in How to get llvm-hs building on MacOSX in nixpkgs instead of using an overlay. It was tricky to figure out. Is there some easier way to do this?

3. The GHC derivation takes arguments of pkgsBuildTarget, targetPackages, and buildLlvmPackages (which gets passed in as buildPackages.llvmPackages). Why doesn’t GHC just take buildPackages as an argument and pull llvmPackages out of it?

   https://github.com/NixOS/nixpkgs/blob/db3beda6b56563777254156351b9bed349cf7d07/pkgs/development/compilers/ghc/8.6.5.nix#L1-L17

   Alternatively, why doesn’t GHC just take the individual packages it needs from pkgsBuildTarget and targetPackages as arguments instead of taking the whole set?

4. The GHC derivation pulls out a separate targetPackages from pkgsBuildTarget:

   https://github.com/NixOS/nixpkgs/blob/db3beda6b56563777254156351b9bed349cf7d07/pkgs/development/compilers/ghc/8.6.5.nix#L82

   What is going on here? Is pkgsBuildTarget.targetPackages different from targetPackages?

5. In the top-level haskell-packages.nix file, why is callPackage redefined:

   https://github.com/NixOS/nixpkgs/blob/db3beda6b56563777254156351b9bed349cf7d07/pkgs/top-level/haskell-packages.nix#L19-L22

   I imagine this doesn’t have anything to do with cross-compilation stuff, but I was still wondering…

6. In the top-level haskell-packages.nix file, when the haskell package sets are created, they are given a reference to GHC from buildPackages.haskell.compiler.ghc865:

   https://github.com/NixOS/nixpkgs/blob/db3beda6b56563777254156351b9bed349cf7d07/pkgs/top-level/haskell-packages.nix#L113-L117

   Why can’t they just take the compiler from haskell.compiler.ghc865? Why specifically buildPackages.haskell.compiler.ghc865?

7. In the top-level haskell-packages.nix file, when the haskell package sets are created, they are given a reference to buildPackages.haskell.packages.ghc865 called buildHaskellPackages:

   https://github.com/NixOS/nixpkgs/blob/db3beda6b56563777254156351b9bed349cf7d07/pkgs/top-level/haskell-packages.nix#L113-L117

   What is this? Is buildPackages.haskell.packages.ghc865 different from haskell.packages.ghc865? If not, then isn’t this just a circular reference? How come this doesn’t give an infinite recursion error when it is used? Is it due to nix’s laziness?

I guess I should ping people I’ve seen talk about cross compilation stuff, since they would probably know the answers to these questions: @peti, @matthewbauer, @Ericson2314 (Recently @peti has been doing weekly streams where he works on Haskell stuff. I’d love to see a long-form video explanation of the types of things I asked above.)