During the end of the LTS release cycle, the packages in the LTS become quite old compared to the latest on Hackage. This causes quite a lot of pain for the nixpkgs Haskell maintainers. Right now it is mostly @peti and @maralorn taking on the brunt of this work. Switching to Nightly will hopefully reduce the amount of work that needs to go into keeping our package set in good shape.
That’s fair! I don’t mean to try to get even more free work out of anyone.
I guess I still have quite a few lts commits to choose from before I can get into trouble with the nightlies.
I fear I may have overreacted, because you can still use resolver: nightly-YYYY-MM-dd in your stack.yaml. I appologise. I’m only still a bit scared about the instabilities of bleeding-edge packages.
@NorfairKing Thank you for the apology. I admit I wasn’t really motivated to answer to your first message.
I admit pinning nixpkgs to get a fix stackage-lts is a use case that I know about, (I even helped people using it) but I had kinda forgotten about it, because it is philosophically quite a different approach to how I develop my projects.
So it’s good to have this feedback. It’s of course a bit hard to tell how widespread this usecase is, but we should keep it in mind for future considerations.
Anyways I think if I needed to figure out what to do to develop a project that works friction free with nix and with stack I guess pinning a nixpkgs and the matching nightly snapshot would be the way to go now. Sometimes nightly wont be in a good state I guess, but in that time you just don‘t bump the pinning of both. (btw. right now nightly even contains more packages than LTS.)
To reiterate: Our aim was to increase the number of working packages in nixpkgs. That includes btw. stackage packages, which can sometimes need manual intervention even though they are in stackage.
Our impression was: If we can provide a larger, more modern set of working packages, users will profit from that. (An example were btw. all hs-gtk dependent packages, which were nearly unsupportable because nixpkgs master used newer system libraries then stackage LTS needed. One example for how nightly seems to be a better fit in nixpkgs.)
Regarding reverting this change: Yeah, we could, but it would break packages and throw away a lot of work and cause quite a lot of new work. Also as domen pointed out, it would mean version downgrades, which sounds quite ugly. I personally am open to discuss if this is a good permanent solution, but let’s try to flow with it for a while. We’ll collect maintenance experience and I guess in a few months we’ll see how this works out.
The fact that 8.10.2 has serious issues that affect our users is sad, but a little bit independent from the question of the stackage snapshot. (I doubt that this issue is the reason that there is no LTS release with 8.10.2). After all we had tested 8.10.x quite extensively in the last months and we run test-suites for a good part of hackage on hydra, which didn‘t surface serious issues. So the only thing wrong with switching to 8.10.2 in my impression is that it’s “too new for all bugs to surface”, but I feel like for that use case we have the stable branches? (We wouldn‘t have done this switch shortly before branch-off.)
I hope I don‘t sound to defensive here. Of course any change of this kind means friction. Let’s see if it was worth it. Having this discussions tells me what to be mindful of in the future.
It may be worth addressing who the consumers of the packages are supposed to be. Are it users that want to directly use the libraries, or are it Nixpkgs applications that need it.
This question was asked for Python as well. Like with Haskell, Python applications can choose a different version, however, if the package set is changing/breaking all the time, it is frustrating for application maintainers to keep up. I suppose the same goes here.
The ghc issue domen linked before and which I was talking about is three weeks old, and three weeks ago there already were reasons not to bump ghc. I was very specific on purpose.
But I have to admit if stackage had a good reason not switch to ghc 8.10.x it would (have been) interesting to know about that.
I think that might be a valid general question, and I have stuff to say to that, too. But I think something else is more important: We need a list of packages that we actively track with the intention to support them. We should definitely populate this list with packages needed in nixpkgs (pandoc e.g.) but surely popular libraries should also be in there. But the question is not so much how exactly to populate this list, but more to have it at all and to have good tooling around it.
what about making contributing to the haskell ecosystem easier ? I can update lua/python/vim stuff within nixpkgs with no hassle. For haskell I have to download and configure extra stuff to update the overlay, which looks like an underestimated barrier (did the whole setup once, but it was too much of a hassle to contribute: add maintainer scripts for haskell package generation by hyperfekt · Pull Request #86699 · NixOS/nixpkgs · GitHub would help). The broken field is also for the current ghc but could we have different broken field depending on the ghc ?
I’m new to using Haskell with Nix but having an issue related to this change, so figured I’d post as it may highlight an issue with the current workflow.
I’m trying to use hakyll-images for a static website. It is marked as broken on 20.09, so I tried master. hakyll is broken on master as of 2021-01-01, but after searching github issues it appears I was in luck as it was fixed in a pair of pull requests by @vaibhavsagar including haskellPackages.hakyll: make buildable by Hydra by vaibhavsagar · Pull Request #104375 · NixOS/nixpkgs · GitHub on 2020-11-20. But when checking out that commit, hakyll is still marked as broken even though hydra succeeded since nixpkgs/pkgs/development/haskell-modules/configuration-hackage2nix.yaml had not yet been updated. When that file is next updated, the switch is made to nightly, which breaks hakyll again. So sadly there is no commit where hakyll both builds and is not marked as broken.
It seems I can get around this by checking out the aforementioned pull-request commit and allowing unbroken, although this results in a very long compilation.
I can’t even imagine how much work is put into maintaining the Haskell package set, and it’s much appreciated. I mainly wanted to highlight that perhaps it makes sense to regenerate the broken/un-broken package lists separately from version bumps, to ensure that we don’t miss an opportunity to have commits with the fewest possible broken packages.
I think this is probably related to the fix being split into two PRs (see https://github.com/NixOS/nixpkgs/pull/104273). Since when backporting haskellPackages fixes to stable, the usually auto generated file hackage-packages.nix is changed manually.
configuration-hackage2nix.yaml is only read by hackage2nix when regenerating hackage-packages.nix which only happens on haskell-updates weekly before the merge into master.
A major disadvantage of this process is however that backported haskellPackages fixes for stable are in a way better than fixes for master: Fixes are merged into the haskell-updates branch and only take effect when the package set is regenerated and merged into master every friday (which could mean that a package is broken up to a week even though a fix has already been merged).
Thank you for the additional context! Good to know.
I think we may be talking around each other, though. My point is that we can have a commit (and thus hydra caching…) with more working packages, simply by remarking as broken after the week of bug fixes and before bumping the nightly version. IIUC, the Hakyll breakage wasn’t related to the split PR but rather because it was fixed on last week’s nightly but still marked as broken, and then the new nightly version bump broke the package in a new way.
To avoid this wasted work, I’m suggesting a cadence like:
Week of bug fixes
update broken
update version
update broken
(Repeat)
Edit: I think I have the details wrong for Hakyll—just realized the PR was a backport, not on master, but I think the problem I outlined is still conceptually accurate.