In the last months using the stackage lts has become a larger maintenance burden which was not very compatible with the way we maintain Haskell packages in nixpkgs. Also we realized that following stackage lts does not really have significant benefits for our use case. (It would make more sense on the stable nixos releases, but we currently don‘t update them automatically anyways.)
That’s why we decided to switch from following stackage-lts to stackage-nightly on master.
This has two important consequences:
The new default ghc version is 8.10.2 (before it was 8.8.4)
A large number of packages previously marked broken might built now and the other way around. (But the total number of working Haskell packages has actually slightly risen.) We have made an effort to get essential packages working, but some Haskell packages you rely on might be broken now. Of roughly 15000 packages on hackage 6000 build successfully in nixpkgs, the other 9000 would need manual intervention and are therefor marked broken. If something you care about is marked broken, feel free to open a PR fixing it or open an issue. If you need help with fixing something the haskellPackages maintainers team (roughly @peti, @cdepillabout and me) will be glad to help. If you want us to know in the future, that you care about a certain package, you can write yourself into the packages maintainer list.
A short explanation about the haskellPackages package set in nixpkgs, if you are confused about the above announcement:
We automatically generate derivations for all packages in Hackage and commit them to nixpkgs. As version of a package we take the version from the stackage snapshot (now stackage-nightly) if it’s on stackage or the latest version from hackage, if not. Then we hope that this builds and if not, we either fix it manually or mark it broken. We update the hackage packages automatically every night on the haskell-updates branch. The haskell-updates branch then gets merged into master manually once per week.
The above changes are being merged into master around the time of this post.
I’d really appreciate if we had some heads up before such decisions are made. In general we strive to keep latest releases in unstable and this really was announced at the hour it was merged.
I understand if there was some pressure, but we could have at least gotten a week?
Yeah, that’s true. I’m sorry. There was not much pressure other than that we stick quite rigorously to the weekly schedule, which has its merits. I suggested making this announcement last week and then it fell through the cracks. Now I decided to make the announcement at least now so that people at least know whats going on, when they encounter issues. I wasn‘t even really around the last time a ghc bump happened. But I have been told an announcement then came even later. So yeah, I’ll file that under lessons learned and will push for a smarter timeline next time.
This is pretty unsatisfying in multiple ways. We kind of regard all hackage packages nearly the same so it’s hard to prioritize. That get’s worsened by the fact that hydra mail notifications are not a thing in the moment. We have plans to improve the process to make it more reliable. I hope we can do that soon.
For a set of Haskell packages which should be broken with more deliberation, how about those which are listed directly in pkgs (for example cachix and nix-linter…)
So watching the stream, I’m somewhat puzzled by the decision.
Stackage LTS does indeed not guarantee 100% of stability, but then somehow switching to nightly to make things worse is better?
So quoting from the discussion in stream, Peter says:
“We could be at GHC 8.10 already. I mean what we could do, obviously, is uhm, we could follow the nightly package set, we would then notice which packages are broken, which packages work, right? We can see, we can see that on Hydra. And then we could basically make an effort, that the stuff we want to compile, compiles. We could contribute the appropriate patches upstream, and we could open pull requests and make that this package set is as stable as possible”
I don’t think breaking master (unfortunately misnamed nixpkgs-unstable) is the way to go, we want it to be as stable as possible.
The motivation seems to be that we’d have a newer GHC version, but I don’t understand how that helps, given that more software is incompatible with the latest GHC version rather than the compiler in Stackage LTS.
It’s important to realize that nixpkgs is more than just Haskell packages so such swift decisions without proper communication impact quite a number of other packages and degrade trust in our ability to keep things stable.
I think we should separate two different questions here:
Is following stackage-nightly a good idea?
Was the transition executed with maximum stability in mind?
The second answer is, as I said earlier, that we should have done better with 2, and I have apologized for that. But I think there probably are just different understandings of how “unstable” master is supposed to be. I remember in roughly the last year occasions were for example a python version bump broke a lot of stuff, previous Haskell bumps also broke stuff. If I remember correctly stuff like that nearly always gets announced widely when it hits master. And I am very used to not updating for a week because something broke packages I use. My impression could be wrong and I agree that we should strive for a higher standard. But it didn‘t feel like this merge was much below the usual standard. (But I might be biased here because I worked the whole week on making it as good as possible. In really regret not noticing that servant was broken, btw.)
Regarding 1. I think we actually only talked shorty about that on the stream and quickly agreed, because it felt so obvious the right choice, that we didn‘t even bother to bring more arguments. The point here is: Switching from LTS to nightly does not mean a quality loss in the haskellPackages set. It most likely means an improvement of quality and a lowering of maintenance burden. We have to fix quite a few broken packages on a weekly basis. A significant number of those manual interventions is caused by the fact, that recent Hackage packages and stackage LTS are often very incompatible because LTS is lagging behind (of course on purpose). Now if a package falls out of stackage-nightly for some reason we can still probably fix it in nixpkgs, since we already have a lot manual fixes and overrides anyways in nixpkgs. That doesn‘t mean that we just drop support for those packages. So my assumption is, that following stackage-nightly will move the maintenance burden to have a different focus, while probably reducing it and giving us newer versions of a lot of cool stuff as a bonus. Of course we could be wrong about that, if the maintenance in truth gets more cumbersome or low quality (maybe that happens when nightly switches to ghc 9.0.x, idk) we should react to that and move back. But for me the evidence points in a different direction: One week after the move we had a lot less overrides necessary while the number of working Haskell packages was already higher than before the switch.
I really appreciate the effort, first of all. I think it’s valid when the decision is made to break things in order to move forward. However the current Haskell release process has showed to have a few flaws:
a) Packages are marked as broken and PR is merged in the same step - within an hour. What bothers me further, is that I’ve asked to give me some time to fix my package (as it was 23h o’clock on Friday) on the PR about 10 minutes after my packages was marked as broken and that was about 10 minutes before PR was merged.
b) When you do releases, you typically fix a resource to get some predictability. It’s either time or scope. Weekly releases of haskell packages will then always have a tendency to break things, especially with such a drastic change. Sticking to the schedule by breaking master does more harm than good. All the rest of nixpkgs uses a scope based system where all maintainers are invoked and there’s no time pressure, negative feelings and potential for burnout.
c) We’re trying to fix the process to have less breakage on master and I’d kindly ask to learn from those mistakes and adapt rather than using them as an supporting evidence of status quo.
Regarding 1. I think we actually only talked shorty about that on the stream and quickly agreed, because it felt so obvious the right choice, that we didn‘t even bother to bring more arguments. The point here is: Switching from LTS to nightly does not mean a quality loss in the haskellPackages set. It most likely means an improvement of quality and a lowering of maintenance burden. We have to fix quite a few broken packages on a weekly basis. A significant number of those manual interventions is caused by the fact, that recent Hackage packages and stackage LTS are often very incompatible because LTS is lagging behind (of course on purpose). Now if a package falls out of stackage-nightly for some reason we can still probably fix it in nixpkgs, since we already have a lot manual fixes and overrides anyways in nixpkgs. That doesn‘t mean that we just drop support for those packages. So my assumption is, that following stackage-nightly will move the maintenance burden to have a different focus, while probably reducing it and giving us newer versions of a lot of cool stuff as a bonus. Of course we could be wrong about that, if the maintenance in truth gets more cumbersome or low quality (maybe that happens when nightly switches to ghc 9.0.x, idk) we should react to that and move back. But for me the evidence points in a different direction: One week after the move we had a lot less overrides necessary while the number of working Haskell packages was already higher than before the switch.
That should really be taken upstream, why is there no LTS release given that there was always one about each half a year? I’ve opened an issue long time ago, but things have since gotten worse.
Once GHC 9 comes out, package set will be way smaller than it currently is and then weekly releases will force a jump back to older LTS. Not only this is a rollercoaster for users, but also it’s going to be a maintenance nightmare.
My understanding (please correct me if I’m wrong) is that this will be still working the same way. The only difference is from which stackage set these currently weekly updates get generated but it doesn’t mean that this breaks reproducibility guarantees. Once stuff gets in to some nixpkgs version, versions are locked by nix expressions so pinning nixpkgs version will still be effective way of locking your dependencies.
No, @Samae you are right. We kinda loose that ability.
I mean before the ability to do so was kinda incidental. So I think we could argue, that it is okay to not have it in the future. On the other hand now we have that feature for nightly.
I think it’s important to note that this doesn‘t mean you can‘t built stack projects with nixpkgs anymore. But yeah, it might be that you need some manual overrides sometimes …
In the future nixpkgs should be as versatile as haskell.nix and let you generate your dependencies easily from a stackage generation.
This sounds like a bad idea. The stackage maintainers make an effort to keep the LTS-es stable in a way that they don’t for the nightlies. How should one go about choosing the right nixpkgs commit to correspond to a stackage lts in the future?
@maralorn Is there any chance we can undo this decision? Being able to follow a stackage LTS is a hard requirement for me, because I need to support both a stack build and a nix build. I’ll be forced to stop using either nix or stack if this change goes through (or fork nixpkgs). Is there maybe some compromise we could use?
Yes there are. @maralorn This change is a serious issue that breaks workflows as well as introducing broken versions of the compiler. Could we revert this change?
I took a look at this thread, but this seems to be a bug that has existed in a bunch of previous GHCs. It doesn’t look like it was introduced in GHC-8.10. Maybe you meant to link to a different bug?
It does sound like they now have patches for it, so it’d probably be a good idea to backport the patches to our GHC-8.10.2 in nixpkgs.
There are a couple ways to get Haskell packages built with both stack and nix. If the Stackage nightlies in nixpkgs doesn’t work, here are a couple alternatives:
regenerate the package set in nixpkgs with the LTS version you want to use (by directly calling the hackage2nix script). Although I don’t think I’ve ever seen anyone do this, so I don’t know how annoying this would be in practice.
Patch up stack2nix to work with recent versions of stack, and then use that to generate your package set.
Just switch to haskell.nix. This is quite heavy-handed, but it seems to be pretty popular lately, and it gives the ability to easily compile with both stack and nix.
regenerate the package set in nixpkgs with the LTS version you want to use (by directly calling the hackage2nix script). Although I don’t think I’ve ever seen anyone do this, so I don’t know how annoying this would be in practice.
Patch up stack2nix to work with recent versions of stack , and then use that to generate your package set.
These two are clearly not well-supported, just going off your own wording.
Just switch to haskell.nix . This is quite heavy-handed, but it seems to be pretty popular lately, and it gives the ability to easily compile with both stack and nix.
I just switched away from haskell.nix because it couldn’t build static executables for things that need terminfo. It’s also 2-4x slower and quite a bit more code to write.
I just don’t think it’s fair to break a common workflow without a deprecation path.
If you want something that is well-supported and widely-used(?), I think your only choice right now is haskell.nix.
While I personally don’t have a strong opinion on whether nixpkgs should track Nightly or LTS, I can understand the reason for switching to Nightly.
During the end of the LTS release cycle, the packages in the LTS become quite old compared to the latest on Hackage. This causes quite a lot of pain for the nixpkgs Haskell maintainers. Right now it is mostly @peti and @maralorn taking on the brunt of this work. Switching to Nightly will hopefully reduce the amount of work that needs to go into keeping our package set in good shape.
I agree that it would be nice if there was an LTS-compatible package set available from nixpkgs, but with only three main Haskell maintainers in nixpkgs (myself included), I don’t think we have the manpower to support it (in addition to Nightly).
Although if someone was really interested in this, I imagine it would be something possible to maintain out of tree.