Which is the latest commit at the time of writing on master.
We have the same situation on release-23.05 which seems to be the current mainline release.
However, it seems like a huge number of packages have been marked as broken.
One of the critical ones is amazonka, which is quite an important/popular package.
My question is: what is the recommendation for working on Haskell projects using Nix?
Should I not use HEAD on master but use a particular branch?
Reading the introduction to the Haskell manual section should give you the background on this. Basically the following points are relevant:
Nixpkgs aims for one package version per package in the package set.
Nixpkgs marks packages that fail to build on CI as broken semi-automatically.
The first property means that packages often start failing since they want certain dependency versions nixpkgs doesn’t ship (by default). A good way to prevent such a situation is have your package be part of Stackage since we are using their LTS package sets as a starting point. It is also possible of course to have a working package without Stackage, but it is more likely to require manual intervention, patching or workarounds from time to time. A Nix friendly and involved upstream can make this very pleasant, though.
The second property means that broken flags can get stale, since broken packages are not tested on CI.
This is relative. The package set is maintained by a quite small group of people (a core team of 4 and some additional maintainers with limited responsibilties). That is quite a small person to package ratio, considering the over 16000 packages available on Hackage. amazonka is, quite plainly, broken because no one cared when it started failing to build.
This is easy to alleviate (even without getting it added to Stackage which would also help): You can become a maintainer for the relevant packages which means that you will get notified about build failures and have a guaranteed grace period to look into them before the failure propagates to master.
I would recommend using any of the unstable channels (depending on your platform) which are based on master. The haskell package sets on the release branches are basically frozen forever (which may suit some usecases?).
old.overrides is in this case self.haskell.packageOverrides. It is missing, because your overlay essentially empties self.haskell, instead you ought to only update haskell.packages.ghc962 specifically. This can be achieved like this:
I’ve hit even more roadblocks.
I now am trying to override amazonka (since it is broken in haskell.packages.ghc962), so I now have amazonka.nix:
# First fetch a particular commit
let
sources = import ../nix/sources.nix;
mkHaskellOverlay = import ./../nix/haskell.nix;
src = sources.amazonka;
overrides = selfh: superh:
let
# takes a name, and builds a package with that subdir.
amzDer = name: subdir:
selfh.callCabal2nix name "${src}/lib/${subdir}" { };
svcDer = name: subdir:
selfh.callCabal2nix name "${src}/lib/services/${subdir}" { };
# same as amzDer, but uses the same string for the resulting package name and the subdir to navigate to.
amzDerDirHasNameOfPackage = name: amzDer name name;
in {
amazonka-s3 = svcDer "amazonka-s3" "amazonka-s3";
amazonka-sns = svcDer "amazonka-sns" "amazonka-sns";
amazonka-sqs = svcDer "amazonka-sqs" "amazonka-sqs";
amazonka-test = amzDerDirHasNameOfPackage "amazonka-test";
amazonka-sso = svcDer "amazonka-sso" "amazonka-sso";
amazonka-sts = svcDer "amazonka-sts" "amazonka-sts";
crypton =
selfh.callCabal2nix "crypton" sources.crypton { };
amazonka = amzDerDirHasNameOfPackage "amazonka";
amazonka-core = amzDerDirHasNameOfPackage "amazonka-core";
};
in mkHaskellOverlay overrides
where haskell.nix is:
# ourOverrides: a function of type selfh: superh : <HaskellOverrides>
# self, super :: nixpkgs.
ourOverrides: self: super: {
haskell = self.lib.recursiveUpdate super.haskell {
packages.ghc962 = super.haskell.packages.ghc962.override (old: {
overrides =
let
# Conserve older overrides; or default to empty overrides.
oldOverrides = old.overrides or (_: _: { });
in super.lib.composeExtensions oldOverrides ourOverrides;
});
};
}
Now, the funny part is: I can override crypton without issues. I can see the version is a latter version with my overlay.
However, I cannot do the same with amazonka, or amazonka-core. If I renameamazonka* to amazonka2, I can see that the new packages get added.
Now, from the documentation of https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.attrsets.recursiveUpdate states that the RHS is preferred, which should be the case here. But I’m very confused about what is going on.
The same overlay works perfectly with nixpkgs.haskellPackages.
EDIT:
Yet more strange behaviour: amazonka-core = "foo"; does overrideamazonka-core, but the derivation does not. Which leads me to think that recursiveUpdate is not updating things within the derivation, which I find unlikely.
What is also very curious is that if I try with recursiveUpdateUntil, I get errors about ghc92?!
I have:
# ourOverrides: a function of type finalh: prevh : <HaskellOverrides>
# final, prev :: nixpkgs.
ourOverrides: final: prev: {
haskell = final.lib.recursiveUpdateUntil (_: _: _: true) prev.haskell {
packages.ghc962 = prev.haskell.packages.ghc962.override (old: {
overrides =
let
# Conserve older overrides; or default to empty overrides.
oldOverrides = old.overrides or (_: _: { });
in final.lib.composeExtensions oldOverrides ourOverrides;
});
};
}
and test.nix, which now gives me:
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core
error: attribute 'ghc92' missing
at /nix/store/n0w3wy2a7z71wpf5ldfqmdkps3b3i4gg-nixpkgs-src/pkgs/top-level/all-packages.nix:15852:11:
15851| then haskell.packages.native-bignum.ghc92
15852| else haskell.packages.ghc92);
| ^
15853|
Did you mean ghc962?
Also, where is the broken information coming from?
error: Package ‘amazonka-core-1.6.1’ in /nix/store/n0w3wy2a7z71wpf5ldfqmdkps3b3i4gg-nixpkgs-src/pkgs/development/haskell-modules/hackage-packages.nix:29521 is marked as broken, refusing to evaluate.
a) To temporarily allow broken packages, you can use an environment variable
for a single invocation of the nix tools.
$ export NIXPKGS_ALLOW_BROKEN=1
Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
(Flake) command, `--impure` must be passed in order to read this
environment variable.
b) For `nixos-rebuild` you can set
{ nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.
c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
{ allowBroken = true; }
to ~/.config/nixpkgs/config.nix.
«derivation
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core.meta.broken
false
meta.broken is set to false, but nix is still telling me the package is broken?
The issue is that nixpkgs probably stores metadata about broken packages elsewhere: or via a mechanism that is completely unclear to me. meta.broken = false seems to make no difference.
Which means when you’re overriding a broken package, you do override it, but nixpkgs doesn’t evaluate it thinking it is already marked as broken.
Now, recursiveUpdate doesn’t change that. Why that is, is beyond my understanding.
What I now did was:
-> % export NIXPKGS_ALLOW_BROKEN=1
-> % nix repl
Welcome to Nix 2.13.3. Type :? for help.
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core
«derivation /nix/store/7wqcggr8kkgxhnqq0ykr391sg0pm9s9d-amazonka-core-2.0.drv»```
Note the version we’re reporting now: 2.0, which is correct.
Without and compare this to:
-> % export NIXPKGS_ALLOW_BROKEN=0
-> % nix repl
Welcome to Nix 2.13.3. Type :? for help.
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core
error: Package ‘amazonka-core-1.6.1’ in /nix/store/n0w3wy2a7z71wpf5ldfqmdkps3b3i4gg-nixpkgs-src/pkgs/development/haskell-modules/hackage-packages.nix:29521 is marked as broken, refusing to evaluate.
Note the version number nix is reporting: 1.6.1 which isn’t what we’re trying to override with, it seems to be the initial version.
I cannot seem to find any documentation on why this is.
EDIT:
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core.version
"2.0"
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core.meta.broken
false
nix-repl> (import ./test.nix).haskell.packages.ghc962.amazonka-core
error: Package ‘amazonka-core-1.6.1’ in /nix/store/n0w3wy2a7z71wpf5ldfqmdkps3b3i4gg-nixpkgs-src/pkgs/development/haskell-modules/hackage-packages.nix:29521 is marked as broken, refusing to evaluate.
a) To temporarily allow broken packages, you can use an environment variable
for a single invocation of the nix tools.
$ export NIXPKGS_ALLOW_BROKEN=1
Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
(Flake) command, `--impure` must be passed in order to read this
environment variable.
b) For `nixos-rebuild` you can set
{ nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.
c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
{ allowBroken = true; }
to ~/.config/nixpkgs/config.nix.
It looks like the prev fixpoint leaks into the final nixpkgs somehow in this case as well, interestingly here only for this like outPath, but not for version, meta etc. The workaround seems to be to rename the package set you are working with, e.g. just exposing it as haskell.packages.ghc962' works fine. I’ll look into this problem further in the coming weeks, unfortunately it is not yet clear to me what causes this, so I can’t really tell you any more.
Binary cache is completely independent from the way packages are layed out in nixpkgs. The only thing that matters are the resulting derivations / outPaths.