Frustrations about splicing

Development Nixpkgs Architecture
2

Yes I think it’s not splicing per se that is cursed, it’s the way we implement scopes and callPackage, which hinders both cross-compilation and situations like “there’s foobar in python3Packages but I need to ask for foobar from the top-level pkgs”. Then there’s also the general issue of expressing more complex graphs than makeScope newScope chains when working out of tree

I wish I could do a more explicit/fine-grained dependency injection, sort of like

# package.nix
{
  dependencies.wayland-scanner.name = "wayland";
  dependencies.wayland-scanner.offsets = "buildHost";
  dependencies.wayland-host.name = "wayland";
  dependencies.wayland-host.offsets = "hostTarget";
  recipe = { stdenv, wayland-scanner, wayland-host, ... }: ...
}
2 Likes
3

We need those who know to explain it/try to document it in detail, and then review it and standardize our writing. I don’t think creating another new function that is generally inconsistent with the current callPackage assumptions is a good solution.

4 Likes
4

Where should this be explained? My hot take :fire: is we should remove the whole discussion of “machinery” from the nixpkgs manual, or somehow clearly separate it from the discussion of “how to use nixpkgs” which is what I believe the manual was intended to be.

5 Likes
5

@waffle8946 With https://github.com/NixOS/nixpkgs/pull/245243 I started implementing this: Use the Nixpkgs manual for stable user-facing documentation, while CONTRIBUTING.md and the various README.md files in the source code are for internal contributer-facing documentation. There’s still lots of work to do to fully implement this vision, but there’s already precedent :slight_smile:

6 Likes
6

I believe no explanation will convince of things like wafHook = waf.hook not allowing to change wafHook to waf.hook.

One of the selling points of a functional language is equational reasoning:

And it is hard to understand why Nix has no referential transparency in this case…

https://wiki.haskell.org/Referential_transparency

2 Likes
7

waf.hook would work if derivations inside derivations were spliced, but it would likely be expensive, alternatively add a function to splice a single drv by Artturin · Pull Request #267792 · NixOS/nixpkgs · GitHub could be used hook = splice callPackage ./hook.nix. splice would also fix the common issue of overrideAttrs removing the __spliced

This would be a good alternative to splicing Discourage/prohibit inheriting from package sets in callPackage · Issue #204303 · NixOS/nixpkgs · GitHub.
Another alternative would be to simply explicitly use the different package sets which gets rid of the magic, but that would complicate .override(a lib func could probaly simplify it). People want invisible and magical cross-compilation which works in most cases and non-cross users don’t want to think about cross.

8

In a certain sense we are doing this already.
I did a quick ripgrep and find

  • 800 matches for buildPackages\.
  • 1400 for buildPackages

People want invisible and magical cross-compilation which works in most cases and non-cross users don’t want to think about cross.

I believe breaking this expectation of “I want invisible and magic” would be a benefit in the long run.

1 Like
9

Most of these, I suspect, are just bad compromises which break overrides

10

I’ve been thinking about whether we need splicing at all. Couldn’t we just manually splice our packages as necessary?

Why don’t we just:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  nativeBuildInputs = with pkgsBuildHost; [
    meson
    ninja
  ];
  buildInputs = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

or even:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps =
    (with pkgsHostHost; [
      libfoo
    ]) ++ (with pkgsBuildHost; [
      meson
      ninja
    ]);

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

?

In order to override usage of a package, you’d then simply do:

package.override { pkgsBuildHost = pkgsBuildHost // { libfoo = ...; }; }

or use any other extension mechanism of your choice (i.e. overlays).

You could also build an override abstraction that would behave similar to the old .override with explicit spliced deps:

package.overridePkgs (prev: { libfoo = prev.libfoo.override...; })

It’d then apply that function to pkgsHostHost, pkgsBuildHost and any other pkgsXY.

2 Likes
11

Please note that combining the dependency attributes will break hook offsets.

1 Like
12

Ah, good point, didn’t know that. Combining the withs in a sane manner looks kinda ugly like that anyways and something like this would be better:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = with pkgsBuildHost; [
    meson
    ninja
  ];
  deps.hostHost = with pkgsHostHost; [
    libfoo
  ];

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

(We could even get fancy and do deps.build.host but I digress.)

This would also finally free us from the confusing legacy buildInputs and nativeBuildInputs names.

1 Like
13

We can be more explicit, in the sense of a function like

buildPackage = setFromPkgsHostHost : setFromPkgsBuildHost : . . .
14

While we’re at it, we could even improve the API a bit and do this:

{
  pkgsHostHost,
  pkgsBuildHost,
  lib,
}:

pkgsBuildHost.stdenv.mkDerivation {
  deps.buildHost = {
    inherit (pkgsBuildHost)
      meson
      ninja
      ;
  ];
  deps.hostHost = {
    inherit (pkgsHostHost) libfoo;
  };

  postInstall = ''
     ${lib.getExe pkgsBuildHost.tool} $out
  '';
}

as lists are notoriously hard to override when you attempt to do anything but append/prepend.

2 Likes
15

I don’t understand what that is supposed to achieve, could you elaborate?

16

See also: simpler, saner cross-compilation · Issue #227327 · NixOS/nixpkgs · GitHub.

I don’t think we can expose the buildHost type names to the world without a revolt. I don’t even fully understand all the combinations myself. @roberth’s attempt to give them palatable names seems promising. Python packages now use build-system, dependencies, etc., which seem directionally nicer.

1 Like
17

While I was elaborating, you provided a better idea:

18

Haha :smiley:

I can understand where you’re coming from but I have a different opinion on this matter: Consistent specific terms may not be as intuitive to a layman as well-chosen inconsistent unspecific terms but as long as they are clear and explained, a layman should be easily able to learn them. By learning the terms, people are exposed to the technology and theory behind the terms which IMO provides a much greater value than a term that is intuitively (mis-)understood.

buildInputs is a lot easier to intuitively understand than deps.hostHost; it’s the things that are put into the build, right?
It get a lot harder when you try to understand nativeBuildInputs intuitively though. IME even experienced Nix users/maintainers, while roughly knowing what it’s for and correctly using it (usually), struggle to explain it. Try it yourself, do you really know what happens when a package is put in nativeBuildInputs?

If you understood what deps.hostHost means, you will trivially be able to understand what deps.buildHost means. In fact, you will likely even have read about deps.buildHost while trying to understand deps.hostHost and understood them in concert.
If you then came across i.e. deps.buildTarget, you’d be much more likely to intuitively understand what that means or at least understand it after you’ve understood up the distinction between host and target. I think you’d also generally be far more likely to spot a cross-compilation bug armed with this knowledge.

For these reasons, as long as they are sensibly chosen and well explained, I always prefer consistent specific terms over intuitive inconsistent unspecific terms. (The masterclass are of course consistent specific intuitive terms but those are not always available.)

4 Likes
19

Can you explain to me why we don’t ~ever use depsHostHost because depsBuildBuild is preferred? I kind of get it, but I feel like you managed to pick one of the most confusing examples there…

In my opinion part of the problem is that “target” as part of the system sucks and is downstream of GCC mis‐design. If there was only build and host then the combinatorial explosion would be a lot more manageable. We’re probably stuck with it, though.

2 Likes
20

Example:
Why, with strictDeps, cmake goes to nativeBuildInputs and cmake-extra-modules goes to buildInputs?

21

A lot of this is actually in the nixpkgs manual, but I consider it really overwhelming due to too much prose and too many analogies. Let me try to summarise.

The three platforms are relevant when building a compiler:

  • buildPlatform = platform where the compiler is built
  • hostPlatform = platform where the compiler is run
  • targetPlatform = platform for which the compiler emits code, if the compiler has the limitation of only being able to emit code for a single platform

The word “compiler” here is relevant, because if you’re not building a compiler, then targetPlatform is a bit nonsensical here and we should really just care about the first two platforms. (s/compiler/package/g, in that case…) Moreover, if you can choose the compiler’s target at runtime, then targetPlatform becomes irrelevant.

So, I’ll discuss for now the case of a non-compiler package, where we only care about build and host, because that’s really what us mere mortals are generally dealing with.

In such a case:

  • depsBuild*: things that should only be available at build-time, i.e. never get into the runtime closure!

    • depsBuildBuild: used to emit more build-time code (e.g. some executables/libs used to build build-time tooling)
    • depsBuildHost nativeBuildInputs: used to emit runtime code (e.g. build-systems for your package, like cmake, go here)
    • depsBuildTarget: used to just don’t.

  • depsHost*: things used at runtime

    In other words, depsHostHost and depsHostTarget buildInputs are the same, but we prefer calling the runtime deps buildInputs for no reason other than historical precedent.

    But for the sake of strictness in this case, I will pretend that depsHostHost refers to buildInputs in the non-compiler case since, again, target is nonsensical.

  • depsTargetTarget: did you forget we don’t care about target?

To summarise: in a non-compiler package, we only use:

  • depsBuildBuild,
  • depsBuildHost nativeBuildInputs, and
  • depsHostHost buildInputs.

By this logic, to answer @AndersonTorres’s question:

extra-cmake-modules (ECM) is a (KDE???) package which contains instructions for augmenting cmake to find various commonly-used binaries and libraries at build-time. In other words, cmake is using ECM at build-time, for build-time, i.e. you do not want ECM in the runtime closure!

So strictly speaking, cmake goes in depsBuildBuild, ECM goes in nativeBuildInputs.
If cmake is also used to build the actual package, then cmake will also go in nativeBuildInputs.
This duplication is fine, because the two uses of cmake in these lists mean two different things.
Finally, whatever libraries you’re linking against, including those mentioned by ECM, will go in buildInputs.

Technically runtime binaries could also go in buildInputs, but AFAIK we don’t have any hooks to wrap the resulting package with binaries from buildInputs (we usually just use symlinkJoin or an explicit makeWrapper call), so it’d be kind of useless to put binaries in that list.

Oh yeah and uh, let me note the irony of mentioning a KDE package, because Qt cross is broken in nixpkgs lmao

Maybe I’ll write a splicing essay tomorrow…

9 Likes