I’ve been using Nix to develop and deploy Haskell for quite some time now and I’d like to share some of the gems I have found/figured out here. For this post I want to focus on just project development. I’m no expert and I’m sure community members will have additions/improvements on what I post. Hopefully we can distill this post into a new wiki entry afterwards.
Pinning Nixpkgs
There are two easy ways to pick a specific commit of sources, niv or nix-prefetch-git (install with nix-env -i nix-prefetch-git. For this guide I use the later since you will end up with a better understanding of what is involved with fetching sources. However, niv is the better choice when you want to abstract away the details.
nix-prefetch-git will give you back some JSON that you can import and fill a fetchFromGitHub command with. I recommend creating a nix directory in the root of your project to put this and other nix files in (not shell.nix or default.nix).
Here is an example of how to get this for unstable:
nix-prefetch-git https://github.com/nixos/nixpkgs-channels.git refs/heads/nixpkgs-unstable > nix/nixpkgs-unstable.json
Here is an example of how to get this for nixos-19.09:
nix-prefetch-git https://github.com/nixos/nixpkgs-channels.git refs/heads/nixos-19.09 > nix/nixos-19-09.json
If want to import the nixos-19.09 package set then I would create nix/nixos-19-09.nix:
let
hostNix = import <nixpkgs> {};
nixpkgsPin = hostNix.pkgs.lib.importJSON ./nixos-19-09.json;
pinnedPkgs = hostNix.pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs-channels";
inherit (nixpkgsPin) rev sha256;
};
in
pinnedPkgs
Now you can import ./nix/nixos-19-09.nix in your default.nix or shell.nix which is now pinned to that specific commit.
hoogle and callCabal2nix
To work effectively with Haskell you need a local hoogle instance with the dependencies you have in your cabal or package.yaml file and the versions of those dependencies you are compiling with. It is important to note you can create either a shell.nix or a default.nix in the root of your project and both can be loaded with nix-shell. The important difference is you may want to use default.nix for deployment since nix-build defaults to using default.nix. Here is an example of how do to this I learned from the state machine testing course:
{ nixpkgs ? import ./nix/nixos-19-09.nix }:
let
overlay = self: super: {
myHaskellPackages =
super.haskell.packages.ghc865.override (old: {
overrides = self.lib.composeExtensions (old.overrides or (_: _: {}))
(hself: hsuper: {
ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
ghcWithPackages = hself.ghc.withPackages;
});
});
};
pkgs = import nixpkgs {
overlays = [overlay];
};
drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
ghcid
# Add other development tools like ormolu here
];
shellHook = ''
source .config/secrets '';
}
);
in
drvWithTools
Lets break down what is going on here.
{ nixpkgs ? import ./nix/nixpkgs.nix }:
The first line is the arguments to this derivation that I called nixpkgs. It has a default value, the import after the ?, so you can just call nix-shell to load up the file. That import is pulling in the pinned nix packages we discussed in the previous section.
overlay = self: super: {
myHaskellPackages =
super.haskell.packages.ghc865.override (old: {
overrides = self.lib.composeExtensions (old.overrides or (_: _: {}))
(hself: hsuper: {
ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
ghcWithPackages = hself.ghc.withPackages;
});
});
};
This is an overlay, meaning it is a function that takes a self and super as arguments. Here is a good guide on how to write overlays. What its doing is setting the attribute myHaskellPackages to the package set of ghc865 in this case with a few adjustments to the package set. Later on we can add other overrides to the package set here.
pkgs = import nixpkgs {
overlays = [overlay];
};
Here we are importing the pinned nixpkgs with the overlay. Importantly we can import other overlays like ormolu.
drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};
This is where we create the derivation for your project. It uses the package set myHaskellPackages we just modified with the overlay and uses callCabal2nix to convert our cabal file into a nix derivation without us having to call cabal2nix ourselves externally.
Note: This also works if you have just a project.yaml, but you have to set the path to ./. instead of ./name-of-project.cabal or ./package.yaml (I wish it would). This can have the unintended concequence of extra rebuilds since Nix is importing the current directory and all its subdirectories as a dependency of your project. In this case (builtins.filterSource (path: type: type != "directory" || baseNameOf path != ".svn") ./.;) can be used to filter out unwanted files.
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
ghcid
# Add other development tools like ormolu here
];
shellHook = ''
source .config/secrets
'';
}
);
Lastly we have the environment for the shell. This is made for you by callCabal2nix and is in the env attribute of drv. However, we want to add some extra packages for development, namely ghcid. Plus I often like to use the shellHook attribute to pull in development secrets (think tokens for development) or other cleanup functions. The overrideAttrs function input is the old attributes of the env. In this case we use it to append our development tools to nativeBuildInputs, otherwise we wouldn’t get hoogle. The with pkgs.myHaskellPackages; means we don’t have to type pkgs.myHaskellPackages before ghcid or any other tools you want (hlint, apply-refactor, doctest, …).
Adding packages or package overrides
Lets say you realize the version of req you want to use is not in the package set. One way is to use callHackageDirect to create a derivation that will use hackage as its source for the version of req you want. A trick to get the sha256 for the package is to put 52 zeros (the correct length of a sha256) as a placeholder, then invoke nix-shell and when the build fails copy over the correct sha256. Keep in mind not to just copy another sha256 from another derivation (like what nix-prefetch-git generated). See this issue for more information.
This would look like:
overlay = self: super: {
myHaskellPackages =
super.haskell.packages."${compiler}".override (old: {
overrides = self.lib.composeExtensions (old.overrides or (_: _: {}))
(hself: hsuper: {
ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
ghcWithPackages = hself.ghc.withPackages;
## Overridden Package here!
req = hself.callHackageDirect
{ pkg = "req";
ver = "3.1.0";
sha256 = "0000000000000000000000000000000000000000000000000000";
} {};
});
});
};
Now this will fail to build since the sha256 nix will calculate won’t be all zeros, but once you replace it with the one Nix reports back then it will build (Note, this is what Niv handles for you). The result is req is replaced with verison 3.1.0 in your package set and your local hoogle instance will generate haddocks for 3.1.0. Note, you will have to reload the nix-shell that hoogle is running in as well as restart hoogle (direnv and Iorri can automate part of that process).
Note: You can also use callHackage which does not need a sha256, but req was not in the all-cabal-hashes so I needed to use callHackageDirect. Search for callHackageDirect on this if you want to read more.
dontCheck and justStaticExecutables
A common issue you will run into by pulling in extra or different packages is that you have to build them yourself. Haskell build times are reasonable (depending on the library), but often the tests are slow. You can fix this by adding dontCheck in front of the dependency you know works or just want to test out. Note, Distributed Builds can help when you don’t want to skip tests or just want faster build times. Anyways, the line we added above would become:
req = self.haskell.lib.dontCheck
(hself.callHackageDirect
{ pkg = "req";
ver = "3.1.0";
sha256 = "0rwbdmp2g74yq7fa77fywsgm0gaxcbpz9m6g57l5lmralg95cdxh";
} {}
);
Another issue is when you are pulling in a tool, not a library. Lets say you want the latest version of ghcid because it has --allow-eval. You would start much the same way, but instead you want to put it in your nativeBuildInputs:
(pkgs.haskell.lib.justStaticExecutables (pkgs.haskellPackages.callHackage "ghcid" "0.8.2"))
In context this would be:
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
(pkgs.haskell.lib.justStaticExecutables (pkgs.haskellPackages.callHackage "ghcid" "0.8.2"))
];
shellHook = ''
source .config/secrets '';
}
);
The justStaticExecutables will just make the executable, not the haddock, etc.
However, when this doesn’t work (version not on hackage, won’t compile, etc.) you can take a more direct approach and callCabal2nix on the source yourself. For example, when I tried getting that version of ghcid building the package set was missing extra. So, here is what you do:
- Get the source (use same
sha256trick):
ghcidSrc = pkgs.fetchFromGitHub {
owner = "ndmitchell";
repo = "ghcid";
rev = "e3a65cd986805948687d9450717efe00ff01e3b5";
sha256 = "1xhyl7m3yaka552j8ipvmrbfixgb8x0i33k2166rkiv7kij0rhsz";
};
- Add to
nativeBuildInputs(This didn’t build for me sinceetxrawas missing)
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
# Added here!
(pkgs.haskell.lib.justStaticExecutables
(pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {}))
];
);
- Get the missing library for ghcid
extra = pkgs.haskell.lib.dontCheck
(pkgs.haskellPackages.callHackageDirect
{ pkg = "extra";
ver = "1.6.20";
sha256 = "0fy5bkn8fn3gcpxksffim912q0ifk057x0wmrvxpmjjpfi895jzj";
} {}
);
- Add the missing libary to the
callCabal2nixcall
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
(pkgs.haskell.lib.justStaticExecutables
# Fixed here!
(pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {inherit extra;}))
];
);
All together in a full example
{ nixpkgs ? import ./nix/nixos-19-09.nix }:
let
overlay = self: super: {
myHaskellPackages =
super.haskell.packages.ghc865.override (old: {
overrides = self.lib.composeExtensions (old.overrides or (_: _: {}))
(hself: hsuper: {
ghc = hsuper.ghc // { withPackages = hsuper.ghc.withHoogle; };
ghcWithPackages = hself.ghc.withPackages;
});
});
};
pkgs = import nixpkgs {
overlays = [overlay];
};
drv = pkgs.myHaskellPackages.callCabal2nix "name-of-project" ./name-of-project.cabal {};
ghcidSrc = pkgs.fetchFromGitHub {
owner = "ndmitchell";
repo = "ghcid";
rev = "e3a65cd986805948687d9450717efe00ff01e3b5";
sha256 = "1xhyl7m3yaka552j8ipvmrbfixgb8x0i33k2166rkiv7kij0rhsz";
};
extra = pkgs.haskell.lib.dontCheck
(pkgs.haskellPackages.callHackageDirect
{ pkg = "extra";
ver = "1.6.20";
sha256 = "0fy5bkn8fn3gcpxksffim912q0ifk057x0wmrvxpmjjpfi895jzj";
} {}
);
drvWithTools = drv.env.overrideAttrs (
old: with pkgs.myHaskellPackages; {
nativeBuildInputs = old.nativeBuildInputs ++ [
(pkgs.haskell.lib.justStaticExecutables
(pkgs.haskell.packages.ghc865.callCabal2nix "ghcid" ghcidSrc {inherit extra;}))
];
);
in
drvWithTools
Adding local packages
Sometimes you find a bug in one of your dependencies and you want to try out a quick fix. Now you could fork the repo, make some changes, push them up and use callCabal2nix to create a derivation for that commit and overide that package just like above. However, that is a lot of work for a change that might not even work. Instead you can just clone the repository you want to modify and use callCabal2nix to import it directly. The addition to the overlay looks like this:
project = self.haskell.lib.dontCheck (hself.callCabal2nix "project" /full/path/to/project/folder {});
Notes: The dontCheck is useful when your change breaks the tests, but you want to test it out before refactoring tests. The callCabal2nix needs the full path to the project folder, not the full path to the cabal file. Also, this will generate new haddocks for the project that you can view in hoogle. If you don’t want the haddocks you can wrap the whole call in self.haskell.lib.dontHaddock (...)
Alternative shellFor
So for we have shown how to create an overlay with your custom haskell package set, tweak that package set to have what you need, use that custom set to create derivation for your project and add some build tools/hooks. There is a shortcut you can take, a function called shellFor that lets you create a shell based on a custom package set and lets you add extra tools/options. You can use this to skip creating the overlay and drvWithTools I create. For now I’ll leave this tutorial as is since overlays are used thoughout nixpkgs, not just Haskell, while the shellFor concept is only setup for some languages.
