haskellPackages.stm-containers fails to build

The haskellPackages.stm-containers fails to build.

According to https://hydra.nixos.org/job/nixpkgs/haskell-updates/haskellPackages.stm-containers.x86_64-linux

It shows that stm-containers fails because stm-hamt fails, and stm-hamt fails because it’s dependency constraint is not met specifically primitive and primitive-extras which is still on an older version.

https://nix-cache.s3.amazonaws.com/log/8pbyf5yx929xji9abhqc7k2178bfz03k-stm-hamt-1.2.0.4.drv

As a general question I’m quite confused how haskell packages get updated. Since all these packages are auto generated. How does one go about wanting to fix this problem, since it seems like a matter of also updating primitive/primitive-extras.

I’m usually working with the pythonPackages attribute set and updating things there is just a matter of submitting a PR to nixpkgs updating the expression. But Haskell Packages are incredibly opaque to me.

Even the CI system is a bit strange, why are haskell packages in their own jobset? And what’s the difference between haskell-updates job set and trunk jobset? Both have haskellPackages.

And why are there some jobs that are “inactive”? Like stm-containers is inactive right now for some reason?

1 Like

How do I know whether this PR https://github.com/NixOS/nixpkgs/pull/76982 fixes this problem?

Seems like since stm-hamt is asking for a later version of primitive, perhaps stm-hamt should be rolled back to a previous version?

The version https://hackage.haskell.org/package/stm-hamt-1.2.0.3 if the haskell packages were to roll back to that, would work.

The only difference between 1.2.0.3 and 1.2.0.4 is using a later primitive version: https://github.com/nikita-volkov/stm-hamt/commit/0f6e59ea8ded2ebc4bcd4bcb59479b9b92c342bd

@CMCDragonkai Sorry you’re having some trouble with the Haskell package set. Let me try to answer some of your questions.

It shows that stm-containers fails because stm-hamt fails, and stm-hamt fails because it’s dependency constraint is not met specifically primitive and primitive-extras which is still on an older version.

I believe your diagnosis here is correct:

https://hydra.nixos.org/build/98648183/nixlog/2

As a general question I’m quite confused how haskell packages get updated.

The main tool for updating the Haskell package set is hackage2nix.

Also, @peti has weekly streams where he goes through and actually updates the Haskell packages. I think he has uploaded some of them to YouTube as well, so I’d recommend actually watching through one:

I believe hackage2nix is run automatically at regular intervals (once a day)? It pushes an update to the haskell-updates branch. Here is an example of it running:


hackage2nix basically looks at two separate sets of information to decide what version of each Haskell package to use:

  1. haskellPackages tracks package versions that are available in the latest Stackage LTS. You can see a list of packages and the corresponding versions here:

    https://github.com/NixOS/nixpkgs/blob/3741b43294b4d807fc982fa671d3c52065d1ef02/pkgs/development/haskell-modules/configuration-hackage2nix.yaml#L45-L82

    For each of the packages in this list, the corresponding package in haskellPackages gets set to this same version.

    For instance, in the above list, we have - lens == 4.17.1. If you look at haskellPackages.lens.version for nixpkgs in that commit, you’ll see that it is at 4.17.1.

    hackage2nix ALSO creates a derivation for the latest package version on Hackage (if they are different).

    For instance, you’ll notice that the latest version of lens on Hackage is actually 4.18.1. hackage2nix helpfully creates haskellPackages.lens_4_18_1 that we can use as well.

    Haskell packages that are available in the latest Stackage LTS set are updated semi-automatically by @peti once a week. Here’s an example of this:

    https://github.com/NixOS/nixpkgs/pull/76982/commits/bae6c63431301926de5064754236f1e0eb0d3101

  2. For Haskell packages not in the latest Stackage LTS, hackage2nix just generates the package from the latest version on Hackage.

    For instance, stm-containers is not in the latest Stackage LTS, so hackage2nix just generates the latest version. haskellPackages.stm-containers.version is currently 1.1.0.4 on the haskell-updates branch.


You can probably see what the problem will be here. The primitive package is in Stackage LTS as version 0.6.4.0, so haskellPackages.primitive is also 0.6.4.0. However, stm-hamt is not in Stackage LTS, and requires primitive >= 7.0.

There are a couple ways to get stm-hamt to build. You could send a PR to the haskell-updates branch that does any of the following:

  1. Override stm-hamt to explicitly use the newest version of primitive. You might have to do something similar for stm-containers:

    In pkgs/development/haskell-modules/configuration-common.nix:

    stm-hamt = super.stm-hamt.override {
        primitive = self.haskellPackages.primitive_0_7_0_0;
        primitive-extras = self.haskellPackages.primitive-extras_0_8;
    };
    

    This is basically what you have to do, but I haven’t tested this. (edit: As you can see below, this is not sufficient, although it should give you an idea of basically what needs to be done.)

    The bad thing about this method is that at some point, when a new version of primitive or primitive-extras is released to Hackage, hackage2nix will remove the primitive_0_7_0_0 and primitive-extras_0_8 derivations. This will cause nixpkgs to not be able to be evaluated, and @peti or I will have to fix it up. This is annoying to do.

  2. Create an older stm-hamt derivation.

    There is a list of packages that will be used to generate older versions of packages from Hackage:

    https://github.com/NixOS/nixpkgs/blob/e116b4b331b855653be114b7a26e4a20fdc95b3e/pkgs/development/haskell-modules/configuration-hackage2nix.yaml#L2475-L2485

    You can see that - aeson < 0.8 is on this list. This means that hackage2nix will generate a derivation like aeson_0_7_0_6.

    You could do something similar for stm-hamt. Add a line like - stm-hamt ==1.2.0.3. This will cause hackage2nix to create the stm-hamt_1_2_0_3 package. Then you could create an override like above on stm-containers to use stm-hamt_1_2_0_3 instead of the normal stm-hamt.

    The bad thing about this method is that this extra-packages list rarely gets updated when it is no longer needed. For instance, you can see the comment on aeson < 0.8 is newer versions don't work with GHC 7.6.x or earlier. However, nixpkgs has long since dropped support for GHC-7.6. It is unlikely anyone is still using this.

    Unless you want to be proactive in supporting stm-hamt and stm-containers, it is likely that it will eventually be broken again in the future.

  3. Force stm-hamt (and stm-containers?) to be an older version.

    You can actually add a line like - stm-hamt ==1.2.0.3 right above the line # LTS Haskell 14.20.

    https://github.com/NixOS/nixpkgs/blob/3741b43294b4d807fc982fa671d3c52065d1ef02/pkgs/development/haskell-modules/configuration-hackage2nix.yaml#L70

    This will cause hackage2nix to generate the haskellPackages.stm-hamt as version 1.2.0.3 (and also create a haskellPackages.stm-hamt_1_2_0_4 package, since that is the latest on Hackage).

    stm-containers will then pick this up automatically, and hopefully be able to be built.

    The bad thing about this method is that there is no automatic process to remove the - stm-hamt ==1.2.0.3 line. This could also very easily break in the future.

  4. Get stm-hamt and stm-containers into Stackage. Then we can be sure that they will compile with the existing version of primitive in Stackage/nixpkgs.

    This is by far the easiest for the nixpkgs maintainers, since we often don’t have to do anything. Although it is often the most work for you. And it normally takes a couple months until you can actually use the package you want to use in nixpkgs.


As you can see, Haskell packages in nixpkgs often break if they are not in Stackage LTS.

Let me answer your other questions as well:

Even the CI system is a bit strange, why are haskell packages in their own jobset? And what’s the difference between haskell-updates job set and trunk jobset? Both have haskellPackages .

I believe the haskell-updates job set just builds Haskell packages on the haskell-updates branch.

The reason they are separate is because the haskell-updates branch gets automatic updates from the hackage2nix tool. These updates have the possibility of breaking the evaluation of nixpkgs (as explained above), so they are kept to a completely separate branch. We never want Haskell stuff to break the trunk job set (which is the nixpkgs master branch).

I believe the configuration file for the haskell-updates job set is https://github.com/peti/ci/blob/f5962fb3e49e9710921f7691fe8befee0a309548/haskell-nixpkgs.nix.

And why are there some jobs that are “inactive”? Like stm-containers is inactive right now for some reason?

I’m not sure. I don’t know enough about Hydra. Although I think this is a hydra question, and not specific to Haskell stuff. Maybe you could create a new thread about this?

How do I know whether this PR https://github.com/NixOS/nixpkgs/pull/76982 fixes this problem?

That PR is to merge the haskell-updates branch into the master branch. This is normally done once a week (or so) by @peti.

The easiest way to see if the package you are interested in is building correctly on the haskell-updates branch is to checkout the haskell-updates branch, and try building the package. For instance,

$ git clone ... # clone nixpkgs
$ cd nixpkgs/
$ git checkout origin/haskell-updates
$ nix-build -A haskellPackages.stm-hamt

It may be marked broken, in which case you might want to checkout the following video which talks about how to mark a package as unbroken (or to try building a broken package anyway):

I should add a way to fix up these types of problems locally, in case you don’t care about nixpkgs and just want to get some project working locally:

I tried to create a nix file for getting the current version of stm-hamt compiling. I came up with the following:

let
  nixpkgsSrc = builtins.fetchTarball {
    # The haskell-updates branch of nixpkgs as of 2020-01-10.
    url = "https://github.com/NixOS/nixpkgs/archive/3741b43294b4d807fc982fa671d3c52065d1ef02.tar.gz";
    sha256 = "sha256:04zg129pb1iis521f8pr5n6y9pmiv48xlzfflwm1smb65hg4wacd";
  };

  # This is a nixpkgs overlay.
  haskellOverlay = nixpkgsSelf: nixpkgsSuper: {

    haskellPackages = nixpkgsSuper.haskellPackages.override {
      overrides = self: super: {
        stm-hamt = nixpkgsSelf.haskell.lib.markUnbroken (super.stm-hamt.override {
          primitive-extras = self.primitive-extras_0_8;
        });

        primitive = nixpkgsSelf.haskell.lib.dontCheck super.primitive_0_7_0_0;

        primitive-unlifted = nixpkgsSelf.haskell.lib.markUnbroken super.primitive-unlifted;
      };
    };
  };

  nixpkgs = import nixpkgsSrc { overlays = [ haskellOverlay ]; };

in

nixpkgs.haskellPackages.stm-hamt

This works:

$ nix-build /path/to/above/file.nix
...
/nix/store/hvigqf64pgcf1badpzc2skjyjysfkzsm-stm-hamt-1.2.0.4

Although, in practice, it is normally just easier to use an older, known-working version of stm-hamt:

let
  nixpkgsSrc = builtins.fetchTarball {
    # The haskell-updates branch of nixpkgs as of 2020-01-10.
    url = "https://github.com/NixOS/nixpkgs/archive/3741b43294b4d807fc982fa671d3c52065d1ef02.tar.gz";
    sha256 = "sha256:04zg129pb1iis521f8pr5n6y9pmiv48xlzfflwm1smb65hg4wacd";
  };

  # Normally you can get this derivation automatically.  See below for details.
  stm-hamt-pkg =
    { mkDerivation, async, base, criterion, deferred-folds, focus, free
    , hashable, list-t, mwc-random, mwc-random-monad, primitive
    , primitive-extras, QuickCheck, quickcheck-instances, rebase
    , rerebase, stdenv, tasty, tasty-hunit, tasty-quickcheck
    , transformers
    }:
    mkDerivation {
      pname = "stm-hamt";
      version = "1.2.0.3";
      sha256 = "2389eae079a7c80013c1ac028ee4750cdfec1f192f521561f3078f80c4c4d72f";
      libraryHaskellDepends = [
        base deferred-folds focus hashable list-t primitive
        primitive-extras transformers
      ];
      testHaskellDepends = [
        deferred-folds focus QuickCheck quickcheck-instances rerebase tasty
        tasty-hunit tasty-quickcheck
      ];
      benchmarkHaskellDepends = [
        async criterion focus free list-t mwc-random mwc-random-monad
        rebase
      ];
      homepage = "https://github.com/nikita-volkov/stm-hamt";
      description = "STM-specialised Hash Array Mapped Trie";
      license = stdenv.lib.licenses.mit;
    };

  # This is a nixpkgs overlay.
  haskellOverlay = nixpkgsSelf: nixpkgsSuper: {
    haskellPackages = nixpkgsSuper.haskellPackages.override {
      overrides = self: super: {
        # This calls the stm-hamt-pkg we have defined above.
        # I was able to generate this derivation with cabal2nix.
        #
        # $ nix-shell -p cabal2nix --run 'cabal2nix cabal://stm-hamt-1.2.0.3'
        #
        # However, normally you can do this automatically with the callHackage function:
        #
        # stm-hamt = self.callHackage "stm-hamt" "1.2.0.3" {};
        #
        # It appears that there is a small bug in the nixpkgs derivation for ghc-8.8
        # causing this to currently not work.  (If you instead used the
        # `master` branch, or one of the nixpkg release branches, you shouldn't ever
        # see a problem like this).
        stm-hamt = self.callPackage stm-hamt-pkg {};
      };
    };
  };

  nixpkgs = import nixpkgsSrc { overlays = [ haskellOverlay ]; };
in

nixpkgs.haskellPackages.stm-hamt

This also works:

$ nix-build /path/to/above/file2.nix
...
/nix/store/pfka7xh1947bp3qzgvhxawvpp9cnwpq8-stm-hamt-1.2.0.3
1 Like

Cool, but why did you override primitive-unlifted?