offtopic:
You describe exactly also the reason why I would like us to keep providing consistent package sets as a distribution, even though it can be a pain to form such a package set for some languages (hello Python). It would be great if other languages/frameworks would also have such upstream distributions that we could piggyback on.
(Yes please! Of any mass changes affecting interfaces that I can think of, this is probably most wanted)
To clarify the rate of (runtime) bugs in general is quite low and well within expectations, this was more to point out how not-broken “broken” packages are, and wasn’t meant to be a criticism of “non-broken” packages.
I’d be curious what exactly the “disheartening level of breakage” you describe is.
I probably phrased that poorly.
Twice I encountered a situation where a package, written in Haskell, had a top-level expression which silently downloaded binaries to my machine in a situation where there was no reason for me to expect that it would do this. I would be a lot less sensitive to this kind of thing if the PR implementing RFC 89 wasn’t stuck in limbo. It is a real sore point for me that I have no way to block nixpkgs from downloading binaries, nor to submit PRs that designate such packages when I discover them.
In both cases the corresponding haskellPackages.* expression would not build, so I assume the binary-downloading was due to that being the only way to get people working software. Maybe I should not have jumped to that conclusion.
The first situation where I encountered this was purescript, which I’ve already mentioned. I tried to manually fix the haskellPackages.purescript expression, but hit a lot of problems with bower-json and old versions of aeson. So then I gave up trying to use nixpkgs to build purescript and decided to use cabal directly, but starting with using nixpkgs to build cabal. It turned out that haskellPackages.cabal is marked broken, and <nixpkgs>.cabal does the downloading-binaries (of GHC!) thing. At that point I began to wonder how deep the rabbit-hole went…
Cabal requires what you can call “local consistency”,
If a package manager only permits installing or depending on one version of each of package then a software distribution for that package manager should bless one version of each package. The blessed version for each package must be compatible with the blessed version every other package.
Well, that “if clause” seems extremely inapplicable to nixpkgs…
I fail to see how just wrapping whatever cargo does in a derivation (using
cargo vendorandcargo build) is the Nix way to be honest.
It isn’t, which is why I prefer carnix/crate2nix.
I see lockfiles as a starting point for generating an “guaranteed to build” nix expression that allows overrides. I consider the current importCargoLock to be an interim solution at best.
- Having the only versions of a package available explicitly packaged and not guarded by a lock file allows you to effectively react to security advisories and to patch critical bugs.
Overrides are then applied to that expression in order to react to security issues that upstream decides to ignore. This should be possible unless the package depends on an old version of a library with a security problem and the fix hasn’t been backported; in this very specific situation the right response is to mark it broken.
Of course, this requires one IFD-rule exception for each language, so we can use carnix/crate2nix-style lang2nix tools to produce overridable nix expressions from lockfile s. @sternenseemann, I see that you are one of the shepherds for RFC 109 which makes it possible to move away from interim solutions like importCargoLock and towards tools like carnix/cargo2nix/lang2nix that turn a lockfile into an overridable derivation.
Perhaps my gripe here has more to do with Cabal than nixpkgs. I think it’s a bit weird that a programming language has a Long Term Support lassification and timeline. I depend on packages like tex, runit, and djbdns that were written decades ago and still work beautifully. Imagine if the C programming language simply “expired” all the code written more than a few years ago and declared it “beyond LTS!”
A minor tangent… If this existed, I think the most valuable version could be platform-variant and able to express the difference between things that are broken for a lack of effort/time (please try to fix me!) VS things that are intrinsically broken (I am broken and cannot be fixed until some pr/rfc/etc. lands; save yourself!)
This is already possible; meta = { broken = stdenv.isDarwin; } appears in many places in nixpkgs.
I think a simple string gets the job done here, no?
It would be a pretty simple RFC to propose that in pkgs/stdenv/generic/check-meta.nix the metaTypes.broken attribute be changed from bool to either bool str, where a pkg is considered to be broken if pkg.meta.broken or false != false.
I count only 25 occurrences of .broken in nixpkgs which are not assignments to the broken attribute (i.e. are not foo.broken = true):
$ rg -g \*.nix '\.[ ]*broken' | grep -v '\.broken =' | wc -l
25
(Technically there are contorted ways (let b = "broken"; in pkg.${b}) to access the broken field that wouldn’t match this regexp). The PR to implement the RFC would be one line needing careful attention and 25 lines of straightforward/LGTM updating.
Hopefully somebody will submit an RFC for this after the release and people have had time to decompress. I can do it if nobody else wants to.
How would you express that a package is currently broken on Linux and Darwin x86-64 because no one’s had success fixing it, and is broken on darwin-aarch64 because the upstream hasn’t addressed compatibility issues yet?
I don’t understand the either bool str part. Shouldn’t it be like (Haskell syntax) data Broken = Fine | BrokenBecause String?
What do you mean with false != false?
Couldn’t we say:
meta.broken = if isDarwin
then BrokenBecause "ReasonA"
else if isLinux
then BrokenBecause "ReasonB" ...
They meant (pkg.meta.broken or false) != false or if pkg.meta ? "broken" then pkg.meta.broken else false for brokennes need to not be equal to false. Any string is “not equal to false”.
That rabbit hole is not as deep as one might think. The package you want is pkgs.cabal-install which is basically the same as pkgs.haskellPackages.cabal-install. We follow the naming of the packages on hackage.
But yeah, we use a binary ghc for bootstraping because at this point it would require about a day to compile the chain from gcc to a current ghc.
just for the record, the haskell.nix project makes that approach (lockfile → one derivation per dependency with correct version) possible. Sadly it is incompatible with all Haskell packages in nixpkgs. I would love to have this functionality in nixpkgs.
Let’s disentagle that a bit.
stackage (independent of cabal-install the tool and Cabal the library) provides an LTS set of none to build together packages. Just because packages are not in there they are not “expired”. It’s just that no one took on the task to include them into stackage. Even very important projects like haskell-language-server are not on stackage, so being “beyond LTS” (i.e. not on stackage) is not in any way a form of dismissal. stackage is an addition to the ecosystem making life for maintainers and developers easier.
cabal-install or ghc do not know anything about stackage and will happily build anything no matter how old it is.
I doubt that bootstrapping GHC from source is actually possible at this point without a year or two of dedicated work. What you can theoretically do is compile GHC using the C backend and then use gcc to build a bootstrap GHC from the generated C – but at that point you can already use the binary GHCs since no one is going to audit that.
meta = {
broken =
with stdenv.hostPlatform;
if isX86_64 && (isLinux || isDarwin)
then "no one's had success fixing it"
if isAarch64 && isDarwin
then "upstream hasn't addressed compatibility issues yet"
else false;
}
Note the use of the hostPlatform.isXX predicates. Compared to string-comparison against builtins.currentSystem-type doubles, the predicates are much more readable and don’t encourage bogus assumptions (like "everything that isn’t x86_64-linux or aarch64-linux is some kind of Darwin).
This is also how Nix parses it: or has higher precedence than !=.
It looks a little funny because or requires whitespace, so you can write it even more misleadingly like this:
pkg.meta.broken or false!=false
but in fact Nix parses it as
(pkg.meta.broken or false) != false
If you think about it, this makes more sense. Most of the time, the thing after the or is going to be a literal (as it is here) or an identifier. So having or have higher precedence will save parentheses more often than it forces them to be added.
The Nix grammar is full of careful decisions like this; clearly somebody paid a lot of attention to usability. Almost makes me willing to forgive the mandatory semicolons 
Hey, thanks for your reply.
Just to be clear, my complaint was not about haskellPackages.ghc8107Binary. Every compiler that is written in the language it compiles for needs a binary seed (for C/C++ this is bootstrap-files).
I swear that at some point during my struggle, nix-env -iA for one of these tools (I thought it was cabal but now I’m not sure anymore) printed out some message like
downloading and installing GHC 9.0.X
this will be installed privately and will not interfere with the system installation
…or something like that. And it was very obvious that whatever it was downloading was not based on the nixpkgs ghc8107Binary bootstrapping chain. Unfortunately I’m having a hard time what led to the message. It was the last thing I tried before getting frustrated enough to set the project aside for a bit.
Yeah, it is also incredibly complex! I tried it and could not get it working. It kept trying to download some tarball-snapshot of nixpkgs, instead of using $NIX_PATH, which was not a good sign.
Fortunately I am very, very close to getting a cabal2nix-based build of purescript working. I really like cabal2nix; it’s simple and it just works, it has nix drive the build process (and invoke ghc) rather than some language-specific build tool, and it produces readable nix expressions that accept overrides.
I don’t know why cabal2nix declares itself deprecated, it’s really too bad. It and crate2nix look like the perfect examples of how nix ought to deal with language-specific package registries (which, let’s face it, are so trendy right now that we can’t keep trying to ignore them). It gives the package maintainers something that is good enough to commit verbatim probably ~80% of the time, and in the rest of the cases it’s a nice clean guaranteed-to-build starting point for manual intervention.
Together, I think these roughly underscore the point of my almost-but-not-quite-rhetorical question, which was roughly: if we’re going to fiddle with the type and form of the variable, we’d might as well make it easy to compose these without complex conditionals that are going to be easy to introduce bugs into (especially if we add more platforms, like other BSDs or Windows?)
That said, this point is worth wrestling with:
I think the string comparison part of this roughly has me, but I think I’m still on the fence on whether nested conditionals against a complex matrix of qualities will be much better?
I’m not sure I understand what alternative you are proposing. Could you be more concrete?
Keep in mind that nix has no static type system whatsoever. It also lacks statically resolvable identifiers. Things like enums and algebraic datatypes help prevent mistakes in other languages; trying to simulate them in Nix actually encourages mistakes.
Please be careful not to kill this effort with overengineering.
There is a close parallel here to RFC 89, which started out as a nice simple “add a meta boolean to indicate packages that download binaries”. A lot of bikeshedding happened, and what’s been approved is stuck with a bunch of complexity that none of the people active in the PR are particularly excited about, and now the PR appears to be languishing somewhat.
Oh, that already happened. Quite a while ago too. There are already predicates for both of those, and much more, including 8-bit AVR (!).
lib/systems is a bit of a hairball, but lib/systems/inspect.nix is one part of it that is nice and clean and works. Each predicate describes some subset of the platforms (both present and future) nixpkgs might be used with. If you can describe your needs with those predicates, do so. &&, ||, and ! correspond to set intersection, union, and complement. If you can’t, add a new predicate. When new platforms are added one of the duties of the person adding the platform is to go through the list of predicates and make sure they react properly to the newly-added platform – like I did when I added mips64el.
In that sense, lib/systems/inspect.nix is the present/future “rendezvous point” between the set of enabled platforms and packages that wish to query that set.
Am I allowed to pose the question without having the answer?
Yes–I know. I’m talking about adding more platforms to the conditionals.
And every time one of the predicates changes, there’s a chance they’re breaking the (often incorrect) assumptions underlying many existing conditionals. The semantic drift it forces is hard to ferret out after the fact unless the committer left a very good paper trail. The explanations will only help with this when the explanation and conditional meet the author’s expectations–otherwise they’ll just mislead.
I’m not sure this formulation leaves me room to do anything but capitulate, so I’m going to go ahead and disengage from the thread.
Does it? cabal2nix is the essential tool, which we use to create all Haskell packages in nixpkgs and there are no plans to change that. Also there is active development on cabal2nix going on, although not very intensive.
The haskell.nix project has a very different scope and is currently very unsuited and not intended for replacing cabal2nix.
You are correct.
Sorry, I got stack2nix (which is the one whose README.md says “Deprecated in favor of GitHub - input-output-hk/haskell.nix: Alternative Haskell Infrastructure for Nixpkgs”) mixed up with cabal2nix. I make the same mistake with crate2nix and cargo2nix, which are even more similarly named.
In any event, cabal2nix is definitely the one with all the awesomeness. I hope the rust-ecosystem tool that tries to emulate it (which I think is crate2nix) replaces importCargoLock as soon as policy allows it to.