Call for Contributions: We are updating to GHC 9.10.3

Development
1

Hello everyone,

we are currently in the process of updating the default version of GHC from 9.8.4 to 9.10.2. This is accompanied with an update from the Stackage LTS 23 to the Stackage LTS 24 package set, so we will be getting, e.g pandoc 3.7 and shellcheck 0.11.0:

Some initial work has already been completed, mainly focussing on fixing eval regressions and getting basic tooling to compile again. Now, the first build of the entire package set on Hydra has been completed, boasting about 1300 new failures. We would like to ask for your help fixing as many of the remaining regressions as possible.

I want to thank all contributors already in advance, also in the name of @NixOS/haskell.

How to Help

The main ways to help is by looking into build regressions of Haskell packages and figuring out if and how to fix them. To find regressionsā€¦

  • you can look at the Hydra jobset for the haskell-updates branch. Why not check whether your favorite Haskell package is broken or packages you otherwise depend on have regressed?
  • A more concise build report based on the Hydra jobset is available (Iā€™ll probably post updated versions of the build report in the PR as time goes on). You can expand a list at the bottom that shows broken packages ordered by reverse dependencies ā€“ fixing these packages will probably have the biggest impact. If youā€™re a maintainer of a Haskell package that broke, you should have been tagged on the PR as well.

For a given regression, try to figure out what went wrong thenā€¦

  • If applicable, write a patch and submit it upstream. This is great because it doesnā€™t just help us and can also be done if youā€™re familiar with Haskell, but not necessarily with the sometimes unwieldy Nixpkgs Haskell infrastructure! Be sure to let us know about relevant PRs/patches e.g. on this thread.
  • Write a Nixpkgs pull request, targeting haskell-updates, to add a workaround for a regression. If applicable, also open an issue with the upstream project, so we can track when this workaround can be removed again. Iā€™ve included some pointers on how to do this below.

Unfortunately, for testing builds locally you need to be using x86_64-linux to be able to substitute builds from the haskell-updates jobset since it is the only platform we test at the moment. (It may be nice to add aarch64-darwin when we have the Hydra resources to spare.)

Adding a Workaround

(mostly copied from Call for Contributions: We are updating to GHC 9.0.2)

Assuming you found packages you are interested in, hereā€™s how to fix them: Package builds for Haskell are fixed by updating the base package description via a set of haskellPackages overlays located in pkgs/development/haskell-modules/configuration-*.nix. The ones youā€™ll probably be needing to touch are:

  • configuration-common.nix: for general fixes that are not tied to a specific GHC version or platform and expected to be temporary (i.e. the issue should be fixed upstream as well)
  • configuration-ghc-*.nix: Compiler version specific fixes. Note that we prefer version agnostic fixes, so rather use CPP and introduce a patch in configuration-common.nix than introduce a hard dependency on GHC >= 9.10 in a patch.
  • The other configurations are:
    • configuration-nix.nix: for fixes that are needed because of the nature of Nix/nixpkgs compared to other environments
    • configuration-arm.nix, configuration-darwin.nix: Platform specific workarounds

You can look into these files, especially configuration-common.nix, for inspiration how fixes typically look. The available helper functions are defined at the place of their definition and the nixpkgs manual. Typical fixes are:

  • Lifting version constraints on a dependency using the doJailbreak function, e.g. if a package has an unnecessarily strict upper constraint on base (< 4.20).
  • Picking upstream patches using appendPatches.
  • Disabling a broken test suite using dontCheck. Note, though, that we try to make them work whenever possible, so verify that the failure is definitely harmless before disabling a test suite.
  • If you need to inject something into the derivation phases etc. use overrideCabal.

When youā€™ve found a fix for a package, make sure to add a comment next to your override explaining why it is needed and ideally link an issue or PR that needs to be resolved upstream for us to drop the override (if no issues exist, do open one). All fixes should naturally be PR-ed against the haskell-updates branch.

Background information on how maintenance of haskellPackages works can be found here

Common Problems

Looking at the failures so far, it seems to me that the following things are responsible for most of the failures:

  • We are using QuickCheck >= 2.15 now. A lot of packages have a constraint of QuickCheck < 2.15 on their test suites. In most cases, this can just be relaxed (via doJailbreak or a patch).
  • We are using filepath >= 1.5 now which may break some packages.
  • We are using text >= 2.1.2 now. This breaks most packages that import Data.Text without further specification because text-2.1.2 added Data.Text.show which clashes with Prelude.show. These failures require applying an upstream patch or writing and submitting one.
  • We are using hashable >= 1.5 now. In most cases this just needs bounds to be relaxed.
  • We are using containers >= 0.7 now. A lot of packages have a constraint of <0.7, but this can probably be lifted in most cases.
  • We are using data-default >= 0.8 now. This means that all data-default-instances-* packages are more or less irreparabely broken and packages depending on them need to be migrated away. Packages that depend on data-default-class < 0.2 need their bounds relaxed if they donā€™t need any instances of Data.Default, otherwise migrated to data-default. C.f. the data-default changelog.
12 Likes
2

The automatically updated status page works again, so you can use that to look for failures again.

3

First Eval with Stackage LTS 24.6

1 Like
4

The test suite of say which a few packages depend on is currently affected by a bug in text 2.1.2. This is resolved by text 2.1.3 which will be bundled with GHC 9.10.3 which hopefully releases in early September. The announcement for 9.10.3-rc4 says:

If all goes well the final release will be available the week of 1 September 2025.

This has the potential to delay landing the update, unfortunately though we may be heading for that time frame anyways. I personally wonā€™t have the spare time to finish the bump until mid to late September unfortunately.

Also refer to my writeup on the haskell-updates PR.

3 Likes
5

We have merged the bulk update into staging on Saturday which has just become staging-next. This way weā€™ll have some time to notice and sort out any issues before 25.11.

You can see all packages that no longer work in this commit, make sure to look at both broken.yaml and transitive-broken.yaml, though some additional fixes have already landed in staging-next.

Any further fixes are welcome. You can target staging-next for trivial changes. More involved fixes or changes that require package updates may need to go to haskell-updates as usual.

3 Likes
6

The GHC update has finally reached the master branch and the nixos-unstable channel. Feel free to report regressions that affect you on the Nixpkgs issue tracker.

We are already accruing fixes in the subsequent haskell-updates PRs and I plan on raising regressions compared to NixOS 25.05 when ZHF for 25.11 is announced. Some leads can be found here and here.

I also want to thank everyone who contributed to this update (this list is reconstructed for the Git history, DM me if Iā€™ve missed anyone):

Also of course, thanks to the staging team, the Stackage maintainers and Hackage trustees.

Iā€™m still not happy with the time it took to deliver this update. Part of this was the extra delay due to the staging workflow (we finalized our branch on 2025-09-25, staging-next was merged on 2025-10-05). Iā€™m still not happy with the development time of 1.5-2 months, but it is a big improvement compared to the update to 9.8 which took almost 4 months. Weā€™ve been discussing changes that could reduce both of these factors which weā€™ll hopefully start implementing soon.

I am (once again) disappointed by the disparity between outside contributions and the Haskell team. Seeing a disparity here is only natural, though I am concerned by the magnitude (2 people account for ~80% of the commits relating to this update). Iā€™ve found having to fix regressions especially frustrating when the packages in question have maintainer(s) listed that had been notified for well over a month, but never so much as responded. I am entertaining a theory that this is related to our notification mechanism being prone to getting lost among irrelevant GitHub notifications, so maybe some changes are in order there.

7 Likes
7

I fully agree with this assessment, I think our attention is sucked away, because each PR notifies many people, and the more people you notify, the less everybody feels responsible to actually respond, especially if it gets merged within a couple days anyways, also causing a lack of maintainer authority.

I might have more time soon to dedicate towards general improvements to maintainer empowerment and attention preservation, but a small step is already in the works here: First-class GitHub team support for review requests Ā· Issue #447514 Ā· NixOS/nixpkgs Ā· GitHub

By making GitHub teams be requestable for reviews, we can take advantage of some advanced review request features, such as round-robin or load-balancing assignments, where only a limited number of people get assigned instead of everybody:

If this works well, we can consider extending this to individual maintainers as well, along with some backup mechanism in case maintainers donā€™t respond within some time, though this would need to be implemented by hand.

1 Like