Frustrations about splicing

Development Nixpkgs Architecture
1

I made splicing (nativeDrv and crossDrv are older, but were done
differently). Splicing is bad. I never wanted it. I causes major
headaches when getting anywhere near bootstrapping. It is gross. It is
probably slow. It should be removed.

Splicing is too magical. Probably three people really understand it.
Dozens of people think they understand it but really don’t, and get
pissed off when it breaks in weird ways. This is a major part of why
people hate on cross compilation.

Frustration

A long time ago I bumped with this PR: https://github.com/NixOS/nixpkgs/issues/204303

The explanation sent me to one of the darkest and uncannier corners of Nixpkgs: the Splicing.

What I understood about splicing is that a spliced package acts like it was two slightly different packages; one of them is to be put at buildInputs and the other at nativeBuildInputs.
It can be noticed e.g. when building packages that depend on SDL.

Usually, splicing can be ignored without too much consequences, since most of the time we deal with leaf packages, and the hardcore machinery is mostly taken for granted.

However, I have bumped again with this idiosyncrasy when I adopted waf. Using waf.hook somehow breaks cross-compilation, however using a green alias wafHook = waf.hook magically works.

Indeed I inserted these three links in Nixpkgs so that I could return to them later:

https://github.com/NixOS/nixpkgs/issues/204303
https://github.com/NixOS/nixpkgs/issues/211340
https://github.com/NixOS/nixpkgs/issues/227327

Another annoying and stressful situation happens when a certain code is rejected and ruled unsuitable and prejudicial because “it breaks splicing”. This is doubly irritating:

And this source of disappointing and draining stress is precisely the dark-magic nature of splicing: anything said about splicing is treated with a huge shot of blind faith, since you most certainly are not one of the ones that really understand splicing.

This situation will become more and more ridiculous, given that huge mass migrations are being promoted via RFC 140. Luckily they do not break splicing by themselves (they are a mere callPackage hidden), however they can possibly break splicing when changing a top-level expression that uses a non-standard callPackage.

Venting a decade-old frustration, I have less than love for that less than beautiful dark-magic-accepted-by-blind-fanaticism thing called “splicing”.

It is not surprising that I agree with the quotes at the top…

Speculation

Since I am not so versed on those highly low-level things, I am a mere observer with a bit of interest (and a bucket of anger).

From the proposals I have seen, my feeling is that all of them pivot in a common motto: a new builder function. In other words, a callPackage2 (I like to call it build-package).

This build-package would receive a complete description of what should be “installed” in each machine of the host-build-target triplet.
Since the idea is to treat cross-compilation as first class, with native compilation as a particular case, the elements of the triplet should be pairwise distinct.

Most certainly this function will not be directly employed in all its grandeur. Most of the time it will be indirectly employed via wrappers to describe environments, to hide details for those not interested in so much detail.

References

List of GitHub issues related to and/or affected by the splicing incident
6 Likes
2

Yes I think it’s not splicing per se that is cursed, it’s the way we implement scopes and callPackage, which hinders both cross-compilation and situations like “there’s foobar in python3Packages but I need to ask for foobar from the top-level pkgs”. Then there’s also the general issue of expressing more complex graphs than makeScope newScope chains when working out of tree

I wish I could do a more explicit/fine-grained dependency injection, sort of like

# package.nix
{
  dependencies.wayland-scanner.name = "wayland";
  dependencies.wayland-scanner.offsets = "buildHost";
  dependencies.wayland-host.name = "wayland";
  dependencies.wayland-host.offsets = "hostTarget";
  recipe = { stdenv, wayland-scanner, wayland-host, ... }: ...
}
2 Likes
3

We need those who know to explain it/try to document it in detail, and then review it and standardize our writing. I don’t think creating another new function that is generally inconsistent with the current callPackage assumptions is a good solution.

5 Likes
4

Where should this be explained? My hot take :fire: is we should remove the whole discussion of “machinery” from the nixpkgs manual, or somehow clearly separate it from the discussion of “how to use nixpkgs” which is what I believe the manual was intended to be.

6 Likes
5

@waffle8946 With https://github.com/NixOS/nixpkgs/pull/245243 I started implementing this: Use the Nixpkgs manual for stable user-facing documentation, while CONTRIBUTING.md and the various README.md files in the source code are for internal contributer-facing documentation. There’s still lots of work to do to fully implement this vision, but there’s already precedent :slight_smile:

6 Likes
6

I believe no explanation will convince of things like wafHook = waf.hook not allowing to change wafHook to waf.hook.

One of the selling points of a functional language is equational reasoning:

And it is hard to understand why Nix has no referential transparency in this case…

https://wiki.haskell.org/Referential_transparency

2 Likes
7

waf.hook would work if derivations inside derivations were spliced, but it would likely be expensive, alternatively add a function to splice a single drv by Artturin · Pull Request #267792 · NixOS/nixpkgs · GitHub could be used hook = splice callPackage ./hook.nix. splice would also fix the common issue of overrideAttrs removing the __spliced

This would be a good alternative to splicing Discourage/prohibit inheriting from package sets in callPackage · Issue #204303 · NixOS/nixpkgs · GitHub.
Another alternative would be to simply explicitly use the different package sets which gets rid of the magic, but that would complicate .override(a lib func could probaly simplify it). People want invisible and magical cross-compilation which works in most cases and non-cross users don’t want to think about cross.

8

In a certain sense we are doing this already.
I did a quick ripgrep and find

  • 800 matches for buildPackages\.
  • 1400 for buildPackages

People want invisible and magical cross-compilation which works in most cases and non-cross users don’t want to think about cross.

I believe breaking this expectation of “I want invisible and magic” would be a benefit in the long run.

1 Like
9

Most of these, I suspect, are just bad compromises which break overrides

10

I’ve been thinking about whether we need splicing at all. Couldn’t we just manually splice our packages as necessary?

Why don’t we just:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  nativeBuildInputs = with pkgsBuildHost; [
    meson
    ninja
  ];
  buildInputs = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

or even:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps =
    (with pkgsHostHost; [
      libfoo
    ]) ++ (with pkgsBuildHost; [
      meson
      ninja
    ]);

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

?

In order to override usage of a package, you’d then simply do:

package.override { pkgsBuildHost = pkgsBuildHost // { libfoo = ...; }; }

or use any other extension mechanism of your choice (i.e. overlays).

You could also build an override abstraction that would behave similar to the old .override with explicit spliced deps:

package.overridePkgs (prev: { libfoo = prev.libfoo.override...; })

It’d then apply that function to pkgsHostHost, pkgsBuildHost and any other pkgsXY.

2 Likes
11

Please note that combining the dependency attributes will break hook offsets.

1 Like
12

Ah, good point, didn’t know that. Combining the withs in a sane manner looks kinda ugly like that anyways and something like this would be better:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = with pkgsBuildHost; [
    meson
    ninja
  ];
  deps.hostHost = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

(We could even get fancy and do deps.build.host but I digress.)

This would also finally free us from the confusing legacy buildInputs and nativeBuildInputs names.

1 Like
13

We can be more explicit, in the sense of a function like

buildPackage = setFromPkgsHostHost : setFromPkgsBuildHost : . . .
14

While we’re at it, we could even improve the API a bit and do this:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = {
    inherit (pkgsBuildHost)
      meson
      ninja
      ;
  ];
  deps.hostHost = {
    inherit (pkgsHostHost) libfoo;
  };

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

as lists are notoriously hard to override when you attempt to do anything but append/prepend.

2 Likes
15

I don’t understand what that is supposed to achieve, could you elaborate?

16

See also: simpler, saner cross-compilation · Issue #227327 · NixOS/nixpkgs · GitHub.

I don’t think we can expose the buildHost type names to the world without a revolt. I don’t even fully understand all the combinations myself. @roberth’s attempt to give them palatable names seems promising. Python packages now use build-system, dependencies, etc., which seem directionally nicer.

1 Like
17

While I was elaborating, you provided a better idea:

18

Haha :smiley:

I can understand where you’re coming from but I have a different opinion on this matter: Consistent specific terms may not be as intuitive to a layman as well-chosen inconsistent unspecific terms but as long as they are clear and explained, a layman should be easily able to learn them. By learning the terms, people are exposed to the technology and theory behind the terms which IMO provides a much greater value than a term that is intuitively (mis-)understood.

buildInputs is a lot easier to intuitively understand than deps.hostHost; it’s the things that are put into the build, right?
It get a lot harder when you try to understand nativeBuildInputs intuitively though. IME even experienced Nix users/maintainers, while roughly knowing what it’s for and correctly using it (usually), struggle to explain it. Try it yourself, do you really know what happens when a package is put in nativeBuildInputs?

If you understood what deps.hostHost means, you will trivially be able to understand what deps.buildHost means. In fact, you will likely even have read about deps.buildHost while trying to understand deps.hostHost and understood them in concert.
If you then came across i.e. deps.buildTarget, you’d be much more likely to intuitively understand what that means or at least understand it after you’ve understood up the distinction between host and target. I think you’d also generally be far more likely to spot a cross-compilation bug armed with this knowledge.

For these reasons, as long as they are sensibly chosen and well explained, I always prefer consistent specific terms over intuitive inconsistent unspecific terms. (The masterclass are of course consistent specific intuitive terms but those are not always available.)

4 Likes
19

Can you explain to me why we don’t ~ever use depsHostHost because depsBuildBuild is preferred? I kind of get it, but I feel like you managed to pick one of the most confusing examples there…

In my opinion part of the problem is that “target” as part of the system sucks and is downstream of GCC mis‐design. If there was only build and host then the combinatorial explosion would be a lot more manageable. We’re probably stuck with it, though.

2 Likes
20

Example:
Why, with strictDeps, cmake goes to nativeBuildInputs and cmake-extra-modules goes to buildInputs?