Frustrations about splicing

Development Nixpkgs Architecture
9

Most of these, I suspect, are just bad compromises which break overrides

10

I’ve been thinking about whether we need splicing at all. Couldn’t we just manually splice our packages as necessary?

Why don’t we just:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  nativeBuildInputs = with pkgsBuildHost; [
    meson
    ninja
  ];
  buildInputs = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

or even:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps =
    (with pkgsHostHost; [
      libfoo
    ]) ++ (with pkgsBuildHost; [
      meson
      ninja
    ]);

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

?

In order to override usage of a package, you’d then simply do:

package.override { pkgsBuildHost = pkgsBuildHost // { libfoo = ...; }; }

or use any other extension mechanism of your choice (i.e. overlays).

You could also build an override abstraction that would behave similar to the old .override with explicit spliced deps:

package.overridePkgs (prev: { libfoo = prev.libfoo.override...; })

It’d then apply that function to pkgsHostHost, pkgsBuildHost and any other pkgsXY.

2 Likes
11

Please note that combining the dependency attributes will break hook offsets.

1 Like
12

Ah, good point, didn’t know that. Combining the withs in a sane manner looks kinda ugly like that anyways and something like this would be better:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = with pkgsBuildHost; [
    meson
    ninja
  ];
  deps.hostHost = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

(We could even get fancy and do deps.build.host but I digress.)

This would also finally free us from the confusing legacy buildInputs and nativeBuildInputs names.

1 Like
13

We can be more explicit, in the sense of a function like

buildPackage = setFromPkgsHostHost : setFromPkgsBuildHost : . . .
14

While we’re at it, we could even improve the API a bit and do this:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = {
    inherit (pkgsBuildHost)
      meson
      ninja
      ;
  ];
  deps.hostHost = {
    inherit (pkgsHostHost) libfoo;
  };

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

as lists are notoriously hard to override when you attempt to do anything but append/prepend.

2 Likes
15

I don’t understand what that is supposed to achieve, could you elaborate?

16

See also: simpler, saner cross-compilation · Issue #227327 · NixOS/nixpkgs · GitHub.

I don’t think we can expose the buildHost type names to the world without a revolt. I don’t even fully understand all the combinations myself. @roberth’s attempt to give them palatable names seems promising. Python packages now use build-system, dependencies, etc., which seem directionally nicer.

1 Like
17

While I was elaborating, you provided a better idea:

18

Haha :smiley:

I can understand where you’re coming from but I have a different opinion on this matter: Consistent specific terms may not be as intuitive to a layman as well-chosen inconsistent unspecific terms but as long as they are clear and explained, a layman should be easily able to learn them. By learning the terms, people are exposed to the technology and theory behind the terms which IMO provides a much greater value than a term that is intuitively (mis-)understood.

buildInputs is a lot easier to intuitively understand than deps.hostHost; it’s the things that are put into the build, right?
It get a lot harder when you try to understand nativeBuildInputs intuitively though. IME even experienced Nix users/maintainers, while roughly knowing what it’s for and correctly using it (usually), struggle to explain it. Try it yourself, do you really know what happens when a package is put in nativeBuildInputs?

If you understood what deps.hostHost means, you will trivially be able to understand what deps.buildHost means. In fact, you will likely even have read about deps.buildHost while trying to understand deps.hostHost and understood them in concert.
If you then came across i.e. deps.buildTarget, you’d be much more likely to intuitively understand what that means or at least understand it after you’ve understood up the distinction between host and target. I think you’d also generally be far more likely to spot a cross-compilation bug armed with this knowledge.

For these reasons, as long as they are sensibly chosen and well explained, I always prefer consistent specific terms over intuitive inconsistent unspecific terms. (The masterclass are of course consistent specific intuitive terms but those are not always available.)

4 Likes
19

Can you explain to me why we don’t ~ever use depsHostHost because depsBuildBuild is preferred? I kind of get it, but I feel like you managed to pick one of the most confusing examples there…

In my opinion part of the problem is that “target” as part of the system sucks and is downstream of GCC mis‐design. If there was only build and host then the combinatorial explosion would be a lot more manageable. We’re probably stuck with it, though.

2 Likes
20

Example:
Why, with strictDeps, cmake goes to nativeBuildInputs and cmake-extra-modules goes to buildInputs?

21

A lot of this is actually in the nixpkgs manual, but I consider it really overwhelming due to too much prose and too many analogies. Let me try to summarise.

The three platforms are relevant when building a compiler:

  • buildPlatform = platform where the compiler is built
  • hostPlatform = platform where the compiler is run
  • targetPlatform = platform for which the compiler emits code, if the compiler has the limitation of only being able to emit code for a single platform

The word “compiler” here is relevant, because if you’re not building a compiler, then targetPlatform is a bit nonsensical here and we should really just care about the first two platforms. (s/compiler/package/g, in that case…) Moreover, if you can choose the compiler’s target at runtime, then targetPlatform becomes irrelevant.

So, I’ll discuss for now the case of a non-compiler package, where we only care about build and host, because that’s really what us mere mortals are generally dealing with.

In such a case:

  • depsBuild*: things that should only be available at build-time, i.e. never get into the runtime closure!

    • depsBuildBuild: used to emit more build-time code (e.g. some executables/libs used to build build-time tooling)
    • depsBuildHost nativeBuildInputs: used to emit runtime code (e.g. build-systems for your package, like cmake, go here)
    • depsBuildTarget: used to just don’t.

  • depsHost*: things used at runtime

    In other words, depsHostHost and depsHostTarget buildInputs are the same, but we prefer calling the runtime deps buildInputs for no reason other than historical precedent.

    But for the sake of strictness in this case, I will pretend that depsHostHost refers to buildInputs in the non-compiler case since, again, target is nonsensical.

  • depsTargetTarget: did you forget we don’t care about target?

To summarise: in a non-compiler package, we only use:

  • depsBuildBuild,
  • depsBuildHost nativeBuildInputs, and
  • depsHostHost buildInputs.

By this logic, to answer @AndersonTorres’s question:

extra-cmake-modules (ECM) is a (KDE???) package which contains instructions for augmenting cmake to find various commonly-used binaries and libraries at build-time. In other words, cmake is using ECM at build-time, for build-time, i.e. you do not want ECM in the runtime closure!

So strictly speaking, cmake goes in depsBuildBuild, ECM goes in nativeBuildInputs.
If cmake is also used to build the actual package, then cmake will also go in nativeBuildInputs.
This duplication is fine, because the two uses of cmake in these lists mean two different things.
Finally, whatever libraries you’re linking against, including those mentioned by ECM, will go in buildInputs.

Technically runtime binaries could also go in buildInputs, but AFAIK we don’t have any hooks to wrap the resulting package with binaries from buildInputs (we usually just use symlinkJoin or an explicit makeWrapper call), so it’d be kind of useless to put binaries in that list.

Oh yeah and uh, let me note the irony of mentioning a KDE package, because Qt cross is broken in nixpkgs lmao

Maybe I’ll write a splicing essay tomorrow…

9 Likes
22

If we strive for an always-on and ubiqutous cross-compilation, every layman maintainer should be able to easily write a package that works (provided upstream is not broken, of course) without a mandatory 2 weeks course on cross-compilology.

image
image976×732 142 KB

2 Likes
23

Oh, so I did make things worse in one respect with https://github.com/NixOS/nixpkgs/pull/329470…

Well, I guess since Qt cross is broken anyway it doesn’t really matter right now.

24

I think anyone who is cross-compiling will know the difference between buildPlatform and hostPlatform intuitively. So unless they are packaging a compiler, they really only have to ask themselves two questions:

  • Does this go in the run-time or (exclusively) build-time closure?
    • If run-time → buildInputs (probably.)
    • If build-time → does this emit code for run-time or (exclusively) build-time?
      • if run-time → nativeBuildInputs
      • if build-time → depsBuildBuild

And usually they might not even have to ask the second question, because most of the build-systems are emitting code for runtime, so guessing and putting it in nativeBuildInputs works fine most of the time.

Maybe, if we used consistent naming, this would become a simpler set of questions.
The first platform refers to the type of closure (Host vs Build).
The second platform refers to what it’s emitting code for (Host vs Build).
So, a new user might more easily remember the difference between depsHostHost, depsBuildHost, depsBuildBuild.

No need for target offsets unless you are building a compiler(-like) package, and that compiler has gcc-like limitations, or if you’re messing with stdenv bootstrapping (in which case you’ll suffer no matter what we call it).

But even if changing the names is unfeasible right now, I think the decision tree above suffices.
And if we arm people with that knowledge, we can get closer to flipping strictDeps to true by default :smiley:

(PS “provided upstream is not broken, of course” is carrying more weight than I’d hope. We do in fact see broken upstreams all the time, so…)

5 Likes
25
  1. What qualifies as a “layman maintainer” in a project like Nixpkgs?
  2. Two weeks? It is a very short time, I would say!
26

I think anyone who is cross-compiling will know the difference between buildPlatform and hostPlatform intuitively.

This is the problem: most maintainers are not interested in cross-compiling, therefore their packages are more often than not broken.
Nixpkgs should try to make it easy to have cross-compilation, even if the maintainer does not care about all these subtleties except for build vs runtime dependencies.
At least once something cross-compiles, it should be hard to break it.

Maybe, if we used consistent naming, this would become a simpler set of questions.

Maybe, but I don’t know if the GNU convention helps: I somehow always forget what these names mean right after I’ve checked the manual, which I do whenever I touch something slightly involved with cross-compilation.

What qualifies as a “layman maintainer” in a project like Nixpkgs?

The small time contributor that has just packaged a small tool/library (which doesn’t cross-compile and later becomes required by a larger project), or that has just fixed an issue by adding { libfoo = libfoo_2_1 } and doesn’t know about splicing.

27

I agree; if we got something working under cross, an action we can take today is to simply put strictDeps = true; in the expression :slight_smile: Then the native builds will have a more-similar environment to cross-compiled builds.

And if we can’t get cross working, put strictDeps = false; to explicitly flag it for further possible improvements.

The other thing we will want to enforce is actually disallowing any references to depsBuild* in the runtime closure (I can’t think of a violating example immediately, but at least the manual claims that we currently don’t enforce this.)

2 Likes
28

I can’t think of a scenario where depsHostHost would be used. You’d need some sort of binary that runs on the hostPlatform and produces code for the hostPlatform rather than the targetPlatform.

I guess in a cross compiler drv, the drvs for use in depsBuildBuild would be depsHostHost?

That’d be because I didn’t use what I intended to use. I had intended to use deps.hostTarget aka. buildInputs.

I don’t really see an alternative? You must make the distinction between these three platforms or else you couldn’t describe the build for a cross-compiler.

How do you think this would work?