Nix + Haskell monorepo tutorial

Announcements
1

Hey!

I tried to setup a Nix build (including a self-hosted Hydra instance) for a haskell monorepo at work and I tried to document the steps involved.

It mostly aggregates info from various sources + describes how I generate a shell.nix file for a Cabal multi-package (new-* commands) development setup.

I’m sure there’s a lot to be improved, so any feedback is welcome.
(I especially don’t like the fact that the monorepo packages are mixed up with the other haskell packages)

Hope someone finds it useful!

11 Likes
2

This is great! I wish I had something like this when we were rolling out our first use of Nix a few months ago.

I haven’t read through the whole tutorial, but one thing I wanted to share that we did with regards to pinning nixpkgs, we decided to pin a particular branch from the nixpkgs-channels repo, rather than a single commit:

builtins.fetchGit {
    url = https://github.com/NixOS/nixpkgs-channels;
    ref = "nixos-19.03";
}

Our reasoning for using this approach is that you get the relative stability of a single channel while addressing the cache issue that Gabriel Gonzalez mentions in his haskell-nix guide:

However, if you choose to go this route then you will need to set up an internal Hydra server to build and cache your project.

Without an internal cache your developers will likely need to build these tools from scratch whenever your pinned nixpkgs drifts too far from the publicly cached channels. ghc in particular is very expensive to rebuild.

So, our assumption was that if you pin a specific channel, you get the benefit of the public cache for a lot longer than if you pin a specific commit. This feels like a trade-off between the stability of a specific commit and the cache of a public channel. This assumption may be incorrect, however, so if I am wrong please correct me.

1 Like
3

For my first attempt I wanted to be sure that I’m working with something that is as fixed as possible (nix is hard to debug even as it is) and with a large project especially with non-public packages I don’t think a private cache is a problem.

My main issue with channels and caching was that I was not really able to find what I can expect to be cached.
For instance I think nothing was cached when I tried to use a ghc other than the default one.
Also I still didn’t figure out what the *-darwin channels are.

Another thing is that it’s easier to explain the setup with a fixed commit to other team-members.

Also I think that with your pinning setup you will be forced to setting them as inputs to the hydra builds (in the web UI), which I really wanted to avoid.

I’ll think over it, and maybe add it to the tutorial.
Thanks for the feedback! I really appreciate it.

4

Great post, thank you!

I’ve also spent a lot of time building monorepo Haskell projects with Nix, and I have a piece of configuration which works for me. I’ve come to mostly the same conclusions as you, but I have a few different habits:

  • I use a single default.nix file to do almost everything, I have a ci.nix file which invokes default.nix with different compilers, and a shell.nix file which only contains (import ./default.nix {}).shell.
  • I use callCabal2nix and use import-from-derivation functionality instead of pregenerating the files (see lines 9-26 on the eample). This also solves the gitignore problem nicely.
  • I use haskellPackages.shellFor function to easily have a shell with dependencies (lines 88-100).
  • I use cachix to get a binary cache with my projects dependencies (nix-build | cachix push utdemir), so others don’t need to recompile half of the hackage if they trust me enough.

Here’s is an example: https://github.com/utdemir/distributed-dataset/blob/ca39acfc8a10d26155a9fdac788198f9652691b3/default.nix

Thanks again, it was really usefu!

3 Likes
5

Thanks @utdemir , your suggestions were super-useful!

I updated the tutorial to include the usage of callCabal2nix and shellFor.

Only thing that still bothers me about the whole solution is Custom subscopes for `haskellPackages` , hopefully I’ll find a way to fix that soon.

6

Great tutorial!

I have one question though. I am trying to migrate my packages to this setup, but one of my packages have a default.nix file with custom patchPhase and postInstall scripts. How do I include such options in the monorepo setting?

1 Like
7

The whole approach of calling callCabal2nix stands on the assumption that you’re including “haskell” projects i.e. non-Nix-aware.

I guess it might be possible to modify the crawling nix function (findHaskellPackages) to analyze whether the package contains a default.nix file and flag the records with it. Then in release.nix you would conditionally call callCabal2nix or callPackage depending on this flag.

I haven’t tried this though, as I am currently only introducing Nix as an alternative to other build solutions and so all our packages have to work without Nix too.