I made splicing (nativeDrv and crossDrv are older, but were done
differently). Splicing is bad. I never wanted it. I causes major
headaches when getting anywhere near bootstrapping. It is gross. It is
probably slow. It should be removed.
Splicing is too magical. Probably three people really understand it.
Dozens of people think they understand it but really don’t, and get
pissed off when it breaks in weird ways. This is a major part of why
people hate on cross compilation.
The explanation sent me to one of the darkest and uncannier corners of Nixpkgs: the Splicing.
What I understood about splicing is that a spliced package acts like it was two slightly different packages; one of them is to be put at buildInputs and the other at nativeBuildInputs.
It can be noticed e.g. when building packages that depend on SDL.
Usually, splicing can be ignored without too much consequences, since most of the time we deal with leaf packages, and the hardcore machinery is mostly taken for granted.
However, I have bumped again with this idiosyncrasy when I adopted waf. Using waf.hook somehow breaks cross-compilation, however using a green alias wafHook = waf.hook magically works.
Indeed I inserted these three links in Nixpkgs so that I could return to them later:
Another annoying and stressful situation happens when a certain code is rejected and ruled unsuitable and prejudicial because “it breaks splicing”. This is doubly irritating:
If the code indeed breaks splicing, it requires patching the work in order to get rid of the offending code.
Example: the wafHook case above
If the code does not break splicing, it implies losing hours or even days in convoluted and stressful discussions.
And this source of disappointing and draining stress is precisely the dark-magic nature of splicing: anything said about splicing is treated with a huge shot of blind faith, since you most certainly are not one of the ones that really understand splicing.
This situation will become more and more ridiculous, given that huge mass migrations are being promoted via RFC 140. Luckily they do not break splicing by themselves (they are a mere callPackage hidden), however they can possibly break splicing when changing a top-level expression that uses a non-standard callPackage.
Venting a decade-old frustration, I have less than love for that less than beautiful dark-magic-accepted-by-blind-fanaticism thing called “splicing”.
It is not surprising that I agree with the quotes at the top…
Speculation
Since I am not so versed on those highly low-level things, I am a mere observer with a bit of interest (and a bucket of anger).
From the proposals I have seen, my feeling is that all of them pivot in a common motto: a new builder function. In other words, a callPackage2 (I like to call it build-package).
This build-package would receive a complete description of what should be “installed” in each machine of the host-build-target triplet.
Since the idea is to treat cross-compilation as first class, with native compilation as a particular case, the elements of the triplet should be pairwise distinct.
Most certainly this function will not be directly employed in all its grandeur. Most of the time it will be indirectly employed via wrappers to describe environments, to hide details for those not interested in so much detail.
References
List of GitHub issues related to and/or affected by the splicing incident
Yes I think it’s not splicing per se that is cursed, it’s the way we implement scopes and callPackage, which hinders both cross-compilation and situations like “there’s foobar in python3Packages but I need to ask for foobar from the top-level pkgs”. Then there’s also the general issue of expressing more complex graphs than makeScope newScope chains when working out of tree
I wish I could do a more explicit/fine-grained dependency injection, sort of like
We need those who know to explain it/try to document it in detail, and then review it and standardize our writing. I don’t think creating another new function that is generally inconsistent with the current callPackage assumptions is a good solution.
Where should this be explained? My hot take is we should remove the whole discussion of “machinery” from the nixpkgs manual, or somehow clearly separate it from the discussion of “how to use nixpkgs” which is what I believe the manual was intended to be.
@waffle8946 With https://github.com/NixOS/nixpkgs/pull/245243 I started implementing this: Use the Nixpkgs manual for stable user-facing documentation, while CONTRIBUTING.md and the various README.md files in the source code are for internal contributer-facing documentation. There’s still lots of work to do to fully implement this vision, but there’s already precedent
This would be a good alternative to splicing Discourage/prohibit inheriting from package sets in callPackage · Issue #204303 · NixOS/nixpkgs · GitHub.
Another alternative would be to simply explicitly use the different package sets which gets rid of the magic, but that would complicate .override(a lib func could probaly simplify it). People want invisible and magical cross-compilation which works in most cases and non-cross users don’t want to think about cross.
I don’t think we can expose the buildHost type names to the world without a revolt. I don’t even fully understand all the combinations myself. @roberth’s attempt to give them palatable names seems promising. Python packages now use build-system, dependencies, etc., which seem directionally nicer.
I can understand where you’re coming from but I have a different opinion on this matter: Consistent specific terms may not be as intuitive to a layman as well-chosen inconsistent unspecific terms but as long as they are clear and explained, a layman should be easily able to learn them. By learning the terms, people are exposed to the technology and theory behind the terms which IMO provides a much greater value than a term that is intuitively (mis-)understood.
buildInputs is a lot easier to intuitively understand than deps.hostHost; it’s the things that are put into the build, right?
It get a lot harder when you try to understand nativeBuildInputs intuitively though. IME even experienced Nix users/maintainers, while roughly knowing what it’s for and correctly using it (usually), struggle to explain it. Try it yourself, do you really know what happens when a package is put in nativeBuildInputs?
If you understood what deps.hostHost means, you will trivially be able to understand what deps.buildHost means. In fact, you will likely even have read about deps.buildHost while trying to understand deps.hostHost and understood them in concert.
If you then came across i.e. deps.buildTarget, you’d be much more likely to intuitively understand what that means or at least understand it after you’ve understood up the distinction between host and target. I think you’d also generally be far more likely to spot a cross-compilation bug armed with this knowledge.
For these reasons, as long as they are sensibly chosen and well explained, I always prefer consistent specific terms over intuitive inconsistent unspecific terms. (The masterclass are of course consistent specific intuitive terms but those are not always available.)
Can you explain to me why we don’t ~ever use depsHostHost because depsBuildBuild is preferred? I kind of get it, but I feel like you managed to pick one of the most confusing examples there…
In my opinion part of the problem is that “target” as part of the system sucks and is downstream of GCC mis‐design. If there was only build and host then the combinatorial explosion would be a lot more manageable. We’re probably stuck with it, though.