Why are these derivations so different?

mhwombat · March 18, 2022, 10:27pm

One thing that has long baffled me is why derivations that apparently do the same thing look so different.
For example, I’m curious about the difference between these two derivations. The first derivation is the one that I use to build a simple Haskell program.

let
  pkgs = import <nixpkgs> { };
in
  pkgs.haskellPackages.developPackage {
    root = ./.;
  }

The second derivation is the one I use to install the same program in my NixOS configuration. It’s in an overlay.*

{ mkDerivation, base, cmdargs, directory, fetchgit, filepath
, process, lib, time
}:
mkDerivation {
  pname = "jot";
  version = "1.4";
  src = fetchgit {
    url = "https://github.com/mhwombat/jot.git";
    sha256 = "1slnsh2jmqbhrsll6xlwa7nks35agmx2hrwxiy26shvnnfcv6m0d";
    rev = "006ac6d2d97e2cfdf42baa0e6d18dd9eb06c0c05";
    fetchSubmodules = true;
  };
  isLibrary = false;
  isExecutable = true;
  executableHaskellDepends = [
    base cmdargs directory filepath process time
  ];
  license = lib.licenses.publicDomain;
}

I assume that pkgs.haskellPackages.developPackage returns a derivation. Is there documentation for pkgs.haskellPackages.developPackage? I searched both the nix and nixpkgs manuals for it, but no luck. I even searched both the nix and nixpkgs repos for the string “developPackage” to see if I could find the source code.

Including a executableHaskellDepends attribute in mkDerivation seems to be the magic word to tell nix to build a standard Haskell executable or library. Is executableHaskellDepends processed by mkDerivation, or plain old derivation? The documentation for mkDerivation doesn’t mention it.

I understand that one reason the two derivations are different is that the first one gets the source from the local directory, and the second fetches it from github. And of course the second needs some extra packaging information such as version number and license. But I am suprised that the Haskell part of the two derivations is so different. Is pkgs.haskellPackages.developPackage basically a convenient wrapper around mkDerivation?

Just FYI, here’s the overlay:

final: prev: {
  jot = final.haskellPackages.callPackage ./jot {};
  pandoc-select-code = final.haskellPackages.callPackage ./pandoc-select-code {};
  pandoc-logic-proof = final.haskellPackages.callPackage ./pandoc-logic-proof {};
}

TLATER · March 19, 2022, 1:59am

Github’s built-in search features are poor in the best of cases, they’re catastrophically bad for something as big as nixpkgs.

I usually search using grep on my computer, but for quick web browsing, @nixinator taught me about sourcegraph: make-package-set.nix - NixOS/nixpkgs - Sourcegraph

There you can also find the docs I’m not sure they’re actually exposed to the outside world, a lot of these deeper ecosystem functions are only documented in internal comments.

cdepillabout · March 19, 2022, 2:10am

Let me try to answer your questions, or at least give you some pointers where you can find more information.

It appears that one of your derivations is just a normal Haskell derivation, and the other one is using developPackage.

Here’s a short overview of the different ways to produce a derivation with the Haskell stuff in Nixpkgs:

Most of the important Haskell-related files are in pkgs/development/haskell-modules. And then pkgs/top-level/haskell-packages.nix sort of pulls everything together.
Just like stdenv, haskellPackages provides a mkDerivation function. haskellPackages also provides a callPackage function that works similarly to the top-level callPackage. If you dig through the above files, you’ll find that they are defined here:
- haskellPackages.mkDerivation
  
  Similarly to stdenv.mkDerivation, haskellPackages.mkDerivation takes an attribute set for building a Haskell package, and returns a derivation. It takes arguments like isLibrary, executableHaskellDepends, pname, etc. Internally, it uses the builder pkgs/development/haskell-modules/generic-builder.nix. (This file is quite complex, but if you read through it, you can figure out exactly what happens with the values passed in as executableHaskellDepends. This generic Haskell builder is effectively a wrapper around stdenv.mkDerivation for specifically building Haskell packages.)
- haskellPackages.callPackage
  
  This is similar to the top-level callPackage, but it knows about all the Haskell packages in the Nixpkgs Haskell package set. For example, you showed a Haskell derivation that started out like this:
```
{ mkDerivation, aeson, base, cmdargs, directory, fetchgit, filepath, process, lens, lib, time}:
mkDerivation {
  pname = "jot";
  version = "1.4";
  ...
```
  If you pass this function (or file) to haskellPackages.callPackage, it knows how to automatically find the arguments for Haskell packages like aeson, lens, etc. It also knows to fall back to looking for packages in the top-level Nixpkgs for things that aren’t Haskell libraries. So things like fetchgit, lib, etc will be pulled from the top-level Nixpkgs.
It is admittedly somewhat confusing that haskellPackages.mkDerivation and stdenv.mkDerivation are named the same, but (completely) different functions (as well as callPackage).

cabal2nix

This is a tool that reads a .cabal file and produces a function call like you have above. For instance:

$ nix-shell -p cabal2nix --run 'cabal2nix https://github.com/mhwombat/jot.git' > my-haskell-package.nix
$ cat my-haskell-package.nix
{ mkDerivation, base, cmdargs, directory, fetchgit, filepath, lib, process, time}:
mkDerivation {
  pname = "jot";
  version = "1.4";
...

You should be able to pass this package to haskellPackages.callPackage in order to build it. For example:

$ nix repl '<nixpkgs>'
nix-repl> haskellPackages.callPackage ./my-haskell-package.nix {}
«derivation /nix/store/zaimhwcg11z7z8r5g87m1hwwdiza5j84-jot-1.4.drv»

nix-repl> :b haskellPackages.callPackage ./my-haskell-package.nix {}
This derivation produced the following outputs:
  doc -> /nix/store/5pm4m9ml7pb3g5s9rs6vnm0wwy4rp9hp-jot-1.4-doc
  out -> /nix/store/8aq64ikzrllr8kll3sfmgbgdwzk48nzg-jot-1.4

haskellPackages.callCabal2nix

The big problem with the cabal2nix CLI tool is that you have to manually run it, redirect output to a file on disk, then calls haskellPackages.callPackage on that output. haskellPackages.callCabal2nix wraps up this whole process for you. For example:

$ nix repl '<nixpkgs>'
nix-repl> haskellPackages.callCabal2nix "jot" (builtins.fetchGit "https://github.com/mhwombat/jot.git") {}
«derivation /nix/store/1wgfvjy7k3y66378214d9k3cwrp7z340-jot-1.4.drv»

nix-repl> :b haskellPackages.callCabal2nix "jot" (builtins.fetchGit "https://github.com/mhwombat/jot.git") {}
This derivation produced the following outputs:
  doc -> /nix/store/hfnfmv4sv0q5jhaq5yjrkysa4kiij54y-jot-1.4-doc
  out -> /nix/store/ckq3xw2qg696pm4iaclifzj7bjq0l3p2-jot-1.4

This is a little easier than manually running cabal2nix, and passing the output to haskellPackages.callPackage.

haskellPackages.callCabal2nix internally uses IFD, with all its advantages and disadvantages.

haskellPackages.developPackage

This is a function for easily doing Haskell development on a local package. With the jot example, this would be used for instance if you have the jot repo checked out locally, and you want to jump into an environment with cabal available, as well as all the Haskell dependencies that are needed. There is a good example of haskellPackages.developPackage at the top of your first post.

haskellPackages.developPackage is mostly a direct wrapper around haskellPackages.callCabal2nix, but it has a little extra logic for giving you a development environment if you call it from nix-shell.

Example of building:
```
$ nix repl '<nixpkgs>'
nix-repl> haskellPackages.developPackage { root = builtins.fetchGit "https://github.com/mhwombat/jot.git"; }  
«derivation /nix/store/0gx99ynmfkpn60bmr3jgm37vqn3v9ags-jot-1.4.drv»

nix-repl> :b haskellPackages.developPackage { root = builtins.fetchGit "https://github.com/mhwombat/jot.git"; }
This derivation produced the following outputs:
  doc -> /nix/store/wlrijn4irrpasam4kxk20rn5b8dnrpgi-jot-1.4-doc
  out -> /nix/store/7fgbai1xswflaw61wl9ccj1zf6k7vyi2-jot-1.4
```
Quick example of shell environment:
```
$ nix-shell -E 'with import <nixpkgs> {}; haskellPackages.developPackage { root = builtins.fetchGit "https://github.com/mhwombat/jot.git"; }'
```
We are now in a shell environment with a ghc available that has all the dependencies of jot. For example:
```
$ ghc-pkg list | grep -i cmdargs
cmdargs-0.10.21
```
There is also a good explanation of using developPackage, of which you’re likely aware
haskellPackages.shellFor

This is for creating a development environment that contains multiple Haskell packages. This is useful when you’re working on a “big” Haskell project, where you have a bunch of individual Haskell packages.

In general, you’d define each of your individual packages with callCabal2nix, and then pass them all to shellFor. See the documentation of haskellPackages.shellFor for an example of this.

When doing development on Haskell packages locally, we generally recommend that users define an overlay for their local packages. The derivations for each of the local packages are generally defined with haskellPackages.callCabal2nix. Then haskellPackages.shellFor is used to define a shell environment.

cdepillabout · March 19, 2022, 2:30am

Sort of. There is a little documentation in the source code. But that doesn’t really give a high-level overview.

There is an open PR that is bringing the Haskell-related documentation back into Nixpkgs. In theory the developPackage documentation should be added to this. This is definitely the type of thing we’d love to get help with if you’re interested in helping out!

I explained this above a little bit, but exectuableHaskellDepends is an argument to the generic Haskell builder in Nixpkgs. The generic Haskell builder is exposed through haskellPackages.mkDerivation. It is somewhat confusing, but haskellPackages.mkDerivation is completely(?) different from stdenv.mkDerivation.

haskellPackages.developPackage is a direct wrapper around haskellPackages.callCabal2nix. But haskellPackages.callCabal2nix is more-or-less a convenient wrapper around cabal2nix, haskellPackages.callPackage, and haskellPackages.mkDerivation.

ryantm · March 19, 2022, 2:44am

When you use ./. in your config it makes a derivation that is named based on the name of the folder the nix file is in. This can lead to surprising results when CI might give the checkout folder a weird name.

This issue is common and insidious enough to have earned a place in the nix anti patterns:

https://nix.dev/anti-patterns/language#reproducibility-referencing-top-level-directory-with

mhwombat · March 19, 2022, 3:29pm

Thank you so much everyone for the explanations. I’m learning so much from this one thread, and it will take me a few days to fully digest all this excellent information.

@cdepillabout, I’m reading that PR and would love to contribute to the documentation effort. For now I will continue to write short tutorials on things as I figure them out. Then I will discuss with the team if any of the things I’ve written would be useful to incorporate into the docs, and if so, where the information should go.

joee · April 25, 2023, 5:38pm

I found a simple solution to the question “How do I install a (cabal-based) haskell executable in my NixOS environment?” Not exactly your original question, but Google sent me here, so sharing… Starting from a similar nix-build environment:

{ pkgs ? import <nixpkgs> { } }:
pkgs.haskellPackages.developPackage {
  root = ./.;
}

I was able to install it in my NixOS configuration.nix using just:

let
  mypackage = import /path/to/mypackage {};
in
...
    environment.systemPackages = with pkgs; [
      mypackage
    ];