A summary of the problems I met while using the current nixpkgs Haskell infrastructure (And my thoughts on how to solve them)

Development
1

I’ve been using nix to develop Haskell packages for about 8 months now. While my overall experience with nixpkgs is great, there are also some frustrating problems with the nixpkgs Haskell infrastructure. I figured out that others might have had the same problems and a good start for improvement would be to identify the problems. So I decided to make a summary.

  1. Dependency management
    Nixpkgs tries to have only one version of each Haskell package at the same time – the version of every Haskell package in the package set is determined in the following way: If this package exists in stackage nightly then use the version from stackage, otherwise use the latest version from hackage. This works well if you are okay with the versions from the package set, you get prebuilt binaries from hydra, and all of the infrastructures (callCabal2nix, developPackage etc.) work out of the box, but if the version does not satisfy your need then tough luck, you’ll have to override the dependency. For developers, this means that you’ll need to add an overlay to your haskellPackages, and for maintainers, to override the inputs in configuration-commons.nix. This is not only tedious but also requires constant maintenance: if the package updates then you’ll need to update the overrides as well, otherwise it will break again.

  2. Documentation
    The Haskell infrastructure is really lacking in documentation, the Haskell section in nixpkgs manual points to cabal2nix’s documentation which hasn’t received any meaningful update for years, is really sparse and does not cover the functions in the library at all. This makes it really hard for newcomers to contribute to the Haskell infrastructure. The interaction between hydra and nixpkgs and the update process (which is usually done by peti) should also be documented.

  3. Automation
    While we currently have some amazing tools to automate the packaging process, there is still a lot of manual work required to keep the packages buildable. Most of the time, when a package gets marked as broken, it is because of one of the following reasons:

    1. The versions provided by nixpkgs does not satisfy the constraint of the package
    2. cabal2nix cannot find a derivation that provides the external dependencies (extra-libraries or pkgconfig-depends) required by the package
    3. The test broke because of sandboxing
    4. Its dependencies broke

    Currently, all of these cases require manual intervention and it’s really tedious to unbreak packages.

  4. Multi-package project support

My thoughts:

  1. Let cabal to do the heavy lifting
    I view nixpkgs’s version fixing as inherently flawed, you are trying to create a huge global package set where every package works with every other package and it’s just not gonna work without immense efforts. On the contrary, we can use per-package package sets, where a different package set is generated for each package which is guaranteed to satisfy all the constraints. How do we do this, you might ask. Well, we can use cabal to do the heavy lifting. Cabal will write the build plan to plan.json after dependency resolution and we could generate the package set from it. We can also get multi-package project support for free. This is the approach used by haskell.nix.

  2. Keeping it pure
    Another problem arises when we chose to use cabal – how do we keep the build pure when cabal requires reading an index that changes over time? Well, we filter out all the entries that are created after a specific date so that the output is fixed no matter when do we download the index. We then configure cabal to use this truncated index as a local repository to resolve dependencies, this way we can keep the build pure and deterministic. We also need to generate nix expressions for all versions of the packages on hackage.

  3. Mixing with version fixing
    One problem with using per-package package sets is that depending on the package set it’s in, one package can potentially evaluate to multiple derivations. This will not only increase the workload on hydra but also the binary cache size. To solve this we could use per-package package sets only on a few packages that do not work well with the global package set but mixing version fixing with per-package package sets is gonna be complex as cabal does not permit packages to depend on multiple versions of the same package.

5 Likes
2

I believe that @xaverdh has a tool which substantially helps with issue 1. Unfortunately I cannot find it right now, hence I’m cc’ing xaverdh directly. :slight_smile:

3

You can build multi-package projects by explicitely passing packages into callPackage, e.g.:

let common = <ghc>.callCabal2nix "common" ./common {};
    app = <ghc>.callCabal2nix "app" ./app { inherit common };
in app

Now common can be declared as a normal build depdency in app.cabal. This is not yet documented, I should see that this is written down at the appropiate place. Took me a few hours to figure this out :innocent:

4

Yes, you can build multi-package projects this way. But you can’t have a dev shell that has all the dependencies installed and indexed this way, can you? Also, I’d like nix to do the plumbing for me instead.

5

You can have a dev shell for multiple nixpkgs haskell packages with shellFor.

2 Likes
6

At some point I started to write my own tooling, but didn’t find the time to make significant progress at the time.
Nowadays there is is haskell.nix, which I think solves most of the issues?

1 Like
7

And the workshop for it: Scrive Nix Workshop - Scrive Nix Workshop

1 Like
8

Yes, haskell.nix works but I think it should be a last resort. It uses its own GHC and binary cache which can take up a lot of disk space. I think we should incorporate some of haskell.nix’s features into nixpkgs.

3 Likes
9

For me the only issue I still can not crack is

<no location info>: error:
    `ld' failed in phase `Merge objects'. (Exit code: 1)

Regardless of nixpkgs, or haskell.nix approach. And I am not sure if it’s related to nix or not. I am on macos. Strangely, when I use nix-shell and call cabal build, for example, everything is fine (e.g. it compiles).

And now I think one of the core problems when using Nix and Haskell together is that it is no clear where to get help from. Information is scattered among sites, blog posts, GitHub readme files, some of the stuff is outdated, some of the stuff is too basic and doesn’t cover real world use cases. And then even on this forum I am not sure where to post my questions :slight_smile: So apologies for hijacking this thread. :pray:

10

What precisely are you doing when you get that error, and what channel are you using?
This sounds vaguely familiar to me. If you enter a nix-shell environment that will put binutils in your PATH, so that might be why ld works then.

11

If that question was addressed to me then, you can find more information here: Linker failures when building Haskell libraries on macos · Issue #126042 · NixOS/nixpkgs · GitHub