Nix Haskell Development (2020)

I’ve been using Nix to develop and deploy Haskell for quite some time now and I’d like to share some of the gems I have found/figured out here. For this post I want to focus on just project development. I’m no expert and I’m sure community members will have additions/improvements on what I post. Hopefully we can distill this post into a new wiki entry afterwards.

Pinning Nixpkgs

There are two easy ways to pick a specific commit of sources, niv or nix-prefetch-git (install with nix-env -i nix-prefetch-git. For this guide I use the later since you will end up with a better understanding of what is involved with fetching sources. However, niv is the better choice when you want to abstract away the details.

nix-prefetch-git will give you back some JSON that you can import and fill a fetchFromGitHub command with. I recommend creating a nix directory in the root of your project to put this and other nix files in (not shell.nix or default.nix).

Here is an example of how to get this for unstable:

nix-prefetch-git https://github.com/nixos/nixpkgs-channels.git refs/heads/nixpkgs-unstable > nix/nixpkgs-unstable.json

Here is an example of how to get this for nixos-19.09:

nix-prefetch-git https://github.com/nixos/nixpkgs-channels.git refs/heads/nixos-19.09 > nix/nixos-19-09.json

If want to import the nixos-19.09 package set then I would create nix/nixos-19-09.nix:

let
  hostNix = import <nixpkgs> {};
  nixpkgsPin = hostNix.pkgs.lib.importJSON ./nixos-19-09.json;

  pinnedPkgs = hostNix.pkgs.fetchFromGitHub {
    owner = "NixOS";
    repo  = "nixpkgs-channels";
    inherit (nixpkgsPin) rev sha256;
  };
in
  pinnedPkgs

Now you can import ./nix/nixos-19-09.nix in your default.nix or shell.nix which is now pinned to that specific commit.

hoogle and callCabal2nix

To work effectively with Haskell you need a local hoogle instance with the dependencies you have in your cabal or package.yaml file and the versions of those dependencies you are compiling with. It is important to note you can create either a shell.nix or a default.nix in the root of your project and both can be loaded with nix-shell. The important difference is you may want to use default.nix for deployment since nix-build defaults to using default.nix. Here is an example of how do to this I learned from the state machine testing course:

{ nixpkgs ? import ./nix/nixos-19-09.nix }:
let
  overlay = self: super: {
    myHaskellPackages = 
      super.haskell.packages.ghc865.override (old: {
        overrides = self.lib.composeExtensions (old.overrides or (_: _: {})) 
          (hself: hsuper: {
            ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
            ghcWithPackages = hself.ghc.withPackages;
          });
      });
  };

  pkgs = import nixpkgs {
    overlays = [overlay];
  };

  drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};

  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        ghcid
        # Add other development tools like ormolu here
      ];
      shellHook = ''
         source .config/secrets     '';
      }
  );
in
  drvWithTools

Lets break down what is going on here.

{ nixpkgs ? import ./nix/nixpkgs.nix }:

The first line is the arguments to this derivation that I called nixpkgs. It has a default value, the import after the ?, so you can just call nix-shell to load up the file. That import is pulling in the pinned nix packages we discussed in the previous section.


  overlay = self: super: {
    myHaskellPackages = 
      super.haskell.packages.ghc865.override (old: {
        overrides = self.lib.composeExtensions (old.overrides or (_: _: {})) 
          (hself: hsuper: {
            ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
            ghcWithPackages = hself.ghc.withPackages;
          });
      });
  };

This is an overlay, meaning it is a function that takes a self and super as arguments. Here is a good guide on how to write overlays. What its doing is setting the attribute myHaskellPackages to the package set of ghc865 in this case with a few adjustments to the package set. Later on we can add other overrides to the package set here.


pkgs = import nixpkgs {
    overlays = [overlay];
  };

Here we are importing the pinned nixpkgs with the overlay. Importantly we can import other overlays like ormolu.


 drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};

This is where we create the derivation for your project. It uses the package set myHaskellPackages we just modified with the overlay and uses callCabal2nix to convert our cabal file into a nix derivation without us having to call cabal2nix ourselves externally.

Note: This also works if you have just a project.yaml, but you have to set the path to ./. instead of ./name-of-project.cabal or ./package.yaml (I wish it would). This can have the unintended concequence of extra rebuilds since Nix is importing the current directory and all its subdirectories as a dependency of your project. In this case (builtins.filterSource (path: type: type != "directory" || baseNameOf path != ".svn") ./.;) can be used to filter out unwanted files.


  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        ghcid
        # Add other development tools like ormolu here
      ];
      shellHook = ''
         source .config/secrets
      '';
      }
  );

Lastly we have the environment for the shell. This is made for you by callCabal2nix and is in the env attribute of drv. However, we want to add some extra packages for development, namely ghcid. Plus I often like to use the shellHook attribute to pull in development secrets (think tokens for development) or other cleanup functions. The overrideAttrs function input is the old attributes of the env. In this case we use it to append our development tools to nativeBuildInputs, otherwise we wouldn’t get hoogle. The with pkgs.myHaskellPackages; means we don’t have to type pkgs.myHaskellPackages before ghcid or any other tools you want (hlint, apply-refactor, doctest, …).

Adding packages or package overrides

Lets say you realize the version of req you want to use is not in the package set. One way is to use callHackageDirect to create a derivation that will use hackage as its source for the version of req you want. A trick to get the sha256 for the package is to put 52 zeros (the correct length of a sha256) as a placeholder, then invoke nix-shell and when the build fails copy over the correct sha256. Keep in mind not to just copy another sha256 from another derivation (like what nix-prefetch-git generated). See this issue for more information.

This would look like:

  overlay = self: super: {
    myHaskellPackages = 
      super.haskell.packages."${compiler}".override (old: {
        overrides = self.lib.composeExtensions (old.overrides or (_: _: {})) 
          (hself: hsuper: {
            ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
            ghcWithPackages = hself.ghc.withPackages;
            ## Overridden Package here!
            req = hself.callHackageDirect 
              { pkg = "req";
                ver = "3.1.0";
                sha256 = "0000000000000000000000000000000000000000000000000000";
              } {};
          });
      });
  };

Now this will fail to build since the sha256 nix will calculate won’t be all zeros, but once you replace it with the one Nix reports back then it will build (Note, this is what Niv handles for you). The result is req is replaced with verison 3.1.0 in your package set and your local hoogle instance will generate haddocks for 3.1.0. Note, you will have to reload the nix-shell that hoogle is running in as well as restart hoogle (direnv and Iorri can automate part of that process).

Note: You can also use callHackage which does not need a sha256, but req was not in the all-cabal-hashes so I needed to use callHackageDirect. Search for callHackageDirect on this if you want to read more.

dontCheck and justStaticExecutables

A common issue you will run into by pulling in extra or different packages is that you have to build them yourself. Haskell build times are reasonable (depending on the library), but often the tests are slow. You can fix this by adding dontCheck in front of the dependency you know works or just want to test out. Note, Distributed Builds can help when you don’t want to skip tests or just want faster build times. Anyways, the line we added above would become:

req = self.haskell.lib.dontCheck 
  (hself.callHackageDirect 
    { pkg = "req";
      ver = "3.1.0";
      sha256 = "0rwbdmp2g74yq7fa77fywsgm0gaxcbpz9m6g57l5lmralg95cdxh";
    } {}
  );

Another issue is when you are pulling in a tool, not a library. Lets say you want the latest version of ghcid because it has --allow-eval. You would start much the same way, but instead you want to put it in your nativeBuildInputs:

(pkgs.haskell.lib.justStaticExecutables (pkgs.haskellPackages.callHackage "ghcid" "0.8.2"))

In context this would be:

  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        (pkgs.haskell.lib.justStaticExecutables (pkgs.haskellPackages.callHackage "ghcid" "0.8.2"))
      ];
      shellHook = ''
         source .config/secrets     '';
      }
  );

The justStaticExecutables will just make the executable, not the haddock, etc.

However, when this doesn’t work (version not on hackage, won’t compile, etc.) you can take a more direct approach and callCabal2nix on the source yourself. For example, when I tried getting that version of ghcid building the package set was missing extra. So, here is what you do:

  1. Get the source (use same sha256 trick):
  ghcidSrc = pkgs.fetchFromGitHub {
    owner = "ndmitchell";
    repo = "ghcid";
    rev = "e3a65cd986805948687d9450717efe00ff01e3b5";
    sha256 = "1xhyl7m3yaka552j8ipvmrbfixgb8x0i33k2166rkiv7kij0rhsz";
  };

  1. Add to nativeBuildInputs (This didn’t build for me since etxra was missing)
  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        # Added here!
        (pkgs.haskell.lib.justStaticExecutables 
          (pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {}))
      ];
  );
  1. Get the missing library for ghcid
  extra = pkgs.haskell.lib.dontCheck 
          (pkgs.haskellPackages.callHackageDirect 
            { pkg = "extra";
              ver = "1.6.20";
              sha256 = "0fy5bkn8fn3gcpxksffim912q0ifk057x0wmrvxpmjjpfi895jzj";
            } {}
          );
  1. Add the missing libary to the callCabal2nix call
  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        (pkgs.haskell.lib.justStaticExecutables 
        # Fixed here!
        (pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {inherit extra;}))
      ];
  );

All together in a full example
{ nixpkgs ? import ./nix/nixos-19-09.nix }:
let
  overlay = self: super: {
    myHaskellPackages = 
      super.haskell.packages.ghc865.override (old: {
        overrides = self.lib.composeExtensions (old.overrides or (_: _: {})) 
          (hself: hsuper: {
            ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
            ghcWithPackages = hself.ghc.withPackages;
          });
      });
  };

  pkgs = import nixpkgs {
    overlays = [overlay];
  };

  drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};

  ghcidSrc = pkgs.fetchFromGitHub {
    owner = "ndmitchell";
    repo = "ghcid";
    rev = "e3a65cd986805948687d9450717efe00ff01e3b5";
    sha256 = "1xhyl7m3yaka552j8ipvmrbfixgb8x0i33k2166rkiv7kij0rhsz";
  };

  extra = pkgs.haskell.lib.dontCheck 
          (pkgs.haskellPackages.callHackageDirect 
            { pkg = "extra";
              ver = "1.6.20";
              sha256 = "0fy5bkn8fn3gcpxksffim912q0ifk057x0wmrvxpmjjpfi895jzj";
            } {}
          );

  drvWithTools = drv.env.overrideAttrs (
    old: with pkgs.myHaskellPackages; {
      nativeBuildInputs = old.nativeBuildInputs ++ [
        (pkgs.haskell.lib.justStaticExecutables 
          (pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {inherit extra;}))
      ];
  );
in
  drvWithTools

Adding local packages

Sometimes you find a bug in one of your dependencies and you want to try out a quick fix. Now you could fork the repo, make some changes, push them up and use callCabal2nix to create a derivation for that commit and overide that package just like above. However, that is a lot of work for a change that might not even work. Instead you can just clone the repository you want to modify and use callCabal2nix to import it directly. The addition to the overlay looks like this:

project = self.haskell.lib.dontCheck (hself.callCabal2nix "project" /full/path/to/project/folder {});

Notes: The dontCheck is useful when your change breaks the tests, but you want to test it out before refactoring tests. The callCabal2nix needs the full path to the project folder, not the full path to the cabal file. Also, this will generate new haddocks for the project that you can view in hoogle. If you don’t want the haddocks you can wrap the whole call in self.haskell.lib.dontHaddock (...)

Alternative shellFor

So for we have shown how to create an overlay with your custom haskell package set, tweak that package set to have what you need, use that custom set to create derivation for your project and add some build tools/hooks. There is a shortcut you can take, a function called shellFor that lets you create a shell based on a custom package set and lets you add extra tools/options. You can use this to skip creating the overlay and drvWithTools I create. For now I’ll leave this tutorial as is since overlays are used thoughout nixpkgs, not just Haskell, while the shellFor concept is only setup for some languages.

15 Likes

Nice post! The wiki would really benefit from this information.

There are only a few things I would add here:

  1. Whenever pinning nixpkgs, I’d recommend adding a comment with the date you used to pin nixpkgs. It is sometimes non-trivial to figure out the date of the channel given just a commit hash, so it is often convenient to know what date a given channel commit corresponds to.

  2. In general, you don’t need to do something like the following:

    (hself: hsuper: {
            ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
            ghcWithPackages = hself.ghc.withPackages;
    })
    

    Instead, I’d recommend most people use shellFor. It works for projects with multiple packages. You can also pass nativeBuildInputs directly to shellFor. (ghcid should technically be in nativeBuildInputs, but in this case it doesn’t really matter.)

    There is a withHoogle argument on shellFor.

    (edit: I originally suggested nativeBuildInput instead of nativeBuildInputs, but that was a mistake. The correct spelling is nativeBuildInputs (plural).)

  3. There are a lot of examples of Haskell-related nix code that take compiler as a argument, giving the ability to compile with multiple versions of GHC.

    In practice, I rarely see people actually use this. Since nixpkgs only contains packages in a single stackage version, there are a whole bunch of packages that don’t compile with different compiler versions. I’d recommend taking this out, just to simplify it somewhat. (Of course, this is a very minor nitpick.)

  4. Unless you have a good reason, I’d recommend against overwriting the top-level haskellPackages attribute in an overlay. Instead, I’d just give it a separate name, like myHaskellPackages. Then you can just refer to it as nixpkgs.myHaskellPackages.

    This can reduce confusion if you need to get development tools (like ghcid) from haskellPackages. Or even if you need to pull in some system library that has a Haskell dependency.

  5. For package overrides, I’d strongly recommend against using cabal2nix to generate Nix expressions on the command line. I consider this an anti-pattern.

    Instead, you should be using callHackage, callHackageDirect, or just callCabal2nix after getting the source with fetchFromGithub.

    The only time you should be using cabal2nix to generate expressions is when your CI doesn’t allow IFD. Although, in that case, I’d recommend just getting a better CI.

  6. It is possible to disable tests, haddocks, profiling, etc for all derivations all at once. To do this, you can override mkDerivation:

    myHaskellPackages = super.haskellPackages.override {
      overrides = hself: hsuper: rec {
        mkDerivation = args: hsuper.mkDerivation (args // {
          doCheck = false;
          doHaddock = false;
          enableLibraryProfiling = false;
          enableExecutableProfiling = false;
          jailbreak = true;
        })
      };
    };
    

    This is nice if you are testing things and frequently rebuilding all Haskell packages, but you want everything to build faster.

    The big downside of this is that you will no longer be able to get any Haskell packages from the shared nix cache.

  7. Link to haskell.nix. Most non-trivial projects are now using haskell.nix.

    haskell.nix has a couple annoying points, but it seems like it will be the way forward for doing Haskell development using nix.


Also, if you end up adding this info to the Haskell wiki page, you may want to link to it from the Haskell reddit. It seems like there is some interest in Nix from the larger Haskell community.

7 Likes

niv is really handy for automating such things. It manages a JSON file with dependencies and provides a nice Nix wrapper. It has a command-line tool to quickly add/drop/update dependencies.

2 Likes

Nice reply! I’ve learned more from your post than hours of reading nix projects and googling. Each number is a reply to the corresponding number from your comment.

  1. Is the date your talking about the same as what nix-prefetch-git sets or do you mean the date you pinned nixpkgs?

  2. I did not know about shellFor, much cleaner. Not sure if I want to show both approaches or not. (I have not added this in yet). Note, changing buildInputs to nativeBulidInput doesn’t put ghcid on my path when using overrideAttrs. I will try with shellFor later.

  3. I was thinking the compiler argument could be used to test which versions of ghc your project works with, so you can annotate your cabal file with them. I bet that use case is better served with separate nix files for testing. (I have not make this change yet)

  4. Much better idea

  5. Learned about this from you, very helpful.

  6. I would still prefer to disable tests, haddocks, profiling on a per dependency basis simply so most things get pulled from the cache. Still, that is a nice tip when working on several distinct packages where there is no cached version.

  7. haskell.nix is on my todo list to try. I’m not sure if I should start linking people to this write-up before or after I take the time to learn how haskell.nix does things differently/better.

This post was better than any guide or manual I’ve read so far. Wish I could click the heart button more than once! I really hope you make this a wiki article.

1 Like

The way I pin nix packages I picked up from niv. Last I checked (quite some time ago) there was some issue I had combining it with my Haskell development setup (The problem might have just been my setup). I’ll check it out again. Thanks for mentioning it!

1 Like

I’m so glad it was useful to you!

I’m done for today, but tomorrow I’ll come back and update it with shellFor, niv and do some grammar checking.

Let me know if the updates get too cluttered.

Is the date your talking about the same as what nix-prefetch-git sets or do you mean the date you pinned nixpkgs?

Ah, so the date I’m referring to here is the date you used for the channel.

For instance, if I were to use the tip of the 19.09 channel as of today, I’d put in a comment in my nix code that says:

# This is the release-19.09 branch as of 2020-03-09.

At work, we use a release branch of nixpkgs, and sometimes things like CUDA-support or MacOSX support will stop working in the release branch. It is helpful to have an idea of the date of the release branch, so you can keep track of the various bugs you tend to run into when your project gets more complicated.

Although this is a very small nitpick. It might be easier just to recommend niv and not worry about it!

changing buildInputs to nativeBulidInput doesn’t put ghcid on my path when using overrideAttrs

Ah, sorry, this should be nativeBuildInputs (plural). I’ll correct it my original post.

(edit: I tried to google you an explanation of nativeBuildInputs vs. buildInputs, but I couldn’t find an easy-to-understand explanation. If anyone can help me out here, I’d appreciate it!)

the compiler argument could be used to test which versions of ghc your project works with, so you can annotate your cabal file with them

In this case, you may also have to change the nixpkgs checkout, so you can get a different LTS package set.

For instance, if you wanted to test GHC-8.6.5, you should use 19.09 (since it contains LTS-14, which uses GHC-8.6.5), but if you want to test GHC-8.8.2, you should use the current master (since it contains LTS-15, which is GHC-8.8.2). Now, there are some instances where you don’t have to do this (and you can instead just pass a different compiler like you suggest), but in practice, for many non-trivial projects, you’ll need to use a nixpkgs Haskell package set that is geared to the compiler you want to test.

Although, I’d say that if you end up wanting to test against different GHC’s, I’d recommend just going to something like haskell.nix.

I would still prefer to disable tests, haddocks, profiling on a per dependency basis simply so most things get pulled from the cache.

Yeah, I agree that is the best thing to do!

I’m not sure if I should start linking people to this write-up before or after I take the time to learn how haskell.nix does things differently/better.

haskell.nix has another big learning curve (although they do have some nice documentation), so my suggestion would be to release this write-up as-is, and maybe just point people to haskell.nix at the very end for more complicated projects.

Thank you all again for your help, especially @cdepillabout feedback. I think I’ve gotten this Howto to a point it can be published on the Nix Wiki, though I won’t do so until later tomorrow. Let me know if you spot any typos or other errors before I submit it.

(I’m also delaying to see if I get the motivation to write out and explain a shellFor example)

1 Like

Great tutorial, thanks; I remember it taking me me a lot of trial and error to figure those out, so I wish we had it sooner!

I found that one disadvantage for this pattern is that it makes starting the new project a lot more time consuming. So I wanted to mention the cookiecutter template I was using which results on a project pretty similar to your example: https://github.com/utdemir/hs-nix-template , it saves quite a bit of time setting all these up.

2 Likes

This is very nice! I’d like to put it as a tl;dr at the top of the guide. I would also like to make a PR adding some examples on how to use niv (importing and adding new packages to the right place). That way I can have your repository as a complete example of how to start using Nix to develop Haskell projects.

Its a much nicer experience to just call nix-shell -p cookiecutter git --run 'cookiecutter gh:utdemir/hs-nix-template' and call it a day. This tutorial has turned into a description of some aspects of how to use Nix to develop Haskell, not necessarily the best way to set it up.

One thing I would mention, don’t override haskellPackages with the custom package set (I made this mistake for awhile as well). Your better off creating a myHaskellPackages so you can still use the original haskellPackages for build tools. This helps with build times since haskellPackages on a recent nixpkgs checkout is often cached, but a modified version may require you to build them yourself.

2 Likes

I would also like to make a PR adding some examples on how to use niv (importing and adding new packages to the right place).

Of course! I’ll just give you commit access, feel free to decline the invitation it if you don’t prefer it :).

Your better off creating a myHaskellPackages so you can still use the original haskellPackages for build tools.

Oh, this is a good point, thanks! I’ll update the template.

1 Like

Thanks for giving me access! I’ll continue this tread with issues on the repo so we don’t clutter up this discussion.