Should we support encoding runtime dependencies without requiring a specific implementation?

Development
1

Right now the only way to specify runtime dependencies is to hard-code an implementation using something like makeWrapper. Although, there are a lot of cases I think it would be useful to encode that a specific file must exist or that a specific binary must exist in $PATH, without tying down the package to a specific implementation.

For example:

{ stdenv }

stdenv.mkDerivation {
  name = "example";

  ...

  meta = {
    runInputs = {
      paths = [ "/run/opengl-driver" ];
      executables = [ "wine" ];
    };
  };
}

Then nixos-rebuild could use this metadata to verify that the final resulting profile satisfies these dependencies.

1 Like
2

Can you link/cite a concrete example?

3

I think it would make a lot of sense!

It even applies much more broadly, you could e.g. delay choosing a libc implementation, or a docker/container runner, or a Python implementation, for an application. Conceptually youā€™d want to build a derivation against an API, then decide on an implementation of that API in a separate derivation. You can then only build the first derivation once, then have a (cheap) build for each combination with an implementation.

1 Like
4

Requiring something in $PATH etc is kinda annoying IMHO because it breaks expectations for things like nix run. However you can do a two-level, where the wrapper is in a separate derivation, so you donā€™t have to rebuild the ā€˜baseā€™ to change the ā€˜wrapperā€™. IIUC things like neovim and clang already use this approach.

5

xdg-utils is already a good concrete example of this as apps which open URLs assume xdg-open will be on the path, and they can call xdg-open https://example.org/ to open it. handlr was already linked as an example replacement.

Another example is replacing xdg-open with an implementation which uses dbus to ask the portal to launch apps like this https://github.com/LunNova/nixos-configs/blob/1c7264b860ca18c582d2cdf18b119c862b053fa7/packages/xdg-open-with-portal/default.nix Currently thereā€™s a mix of packages picking it up from path and wrapping it themselves.

I donā€™t know how often this applies in the general case, and I canā€™t remember other examples off the top of my head right now.

1 Like
6

Oh yep here are some examples where I think this would make sense:

7

Thatā€™s true for most cases, but I think it makes sense in certain circumstances. I feel like for nix run we could just give the same kind of error message pointing out that a runtime dependency is missing.

8

For certain common tools which one might want to replace, like xdg-open, it seems to me like nixpkgs config, combined with the two-level structure @dramforever suggested, is the right way to handle it. That way you can set in one place what you want to use for that, but you still keep the self-contained nature of a nix derivation closure.

1 Like
9

This sounds similar to npmā€™s concept of ā€œpeerā€ dependencies.

2 Likes
10

Yep, I think thatā€™s the ideal, but the problem comes when thereā€™s no good default, and it makes more sense to take what the user set on $PATH.

For example, I mentioned doing the exact same thing for texlab (that it should depend on a TeX distribution), but the other reviewers decided against it because they mentioned that many users may have a different TeX distribution installed, and expect that to be used instead:

https://github.com/NixOS/nixpkgs/pull/85700#issuecomment-617438601

For these situations Iā€™m suggesting that we have some other mechanism to determine when these dependencies are satisfied, instead of just leaving the dependency off altogether.

11

Another similar existing example is /run/opengl-drivers. Packages build against mesa, and then at runtime depend on whatever is symlinked in /run/opengl-drivers.

1 Like
12

This sounds similar to npmā€™s concept of ā€œpeerā€ dependencies .

Kind of, but more general, instead of requiring a specific nix dependency, I think we want to require a specific path or executable to exist in $PATH

13

Iā€™ve updated the description to give a better sense of how I think something like this would be implemented.

1 Like
14

For /run/opengl-drivers/lib specifically, there is a addOpenGLRunpath hook which will add the RUNPATH to all elf files:

  nativeBuildInputs = [ addOpenGLRunpath ];

and then:

objdump -x ./result/bin/<elf> | grep opengl
15

Yeah, but that hook canā€™t actually verify that /run/opengl-drivers actually exists in the final system profile during build time, it will just fail to load the library at runtime.

16

correct, on the consumer side thereā€™s GitHub - nix-community/nixGL: A wrapper tool for nix OpenGL application [maintainer=@guibou]. The problem is trying to add some impurity, without opening the floodgates of /usr/local/lib

17

Do you think it would help to have some formal way to describe these runtime impurities like what I mentioned in the main description though? Right now we assume these exist at runtime and fail at runtime if they donā€™t, but I feel like we could verify this earlier during profile generation.

18

The problem is c-style dynamic linking. Thereā€™s only so much we can do when an environment ā€œassumesā€ that libraries will be co-located with everything else. :frowning:

You might be able to do something with /etc/nix-ld.so.preload https://github.com/NixOS/nixpkgs/blob/8e36f0c4d18a55630954ff2206b1c05ec3fb8bb5/pkgs/development/libraries/glibc/dont-use-system-ld-so-preload.patch, if it only contained the paths of what was meant to be in /run/opengl-driver/lib.

This probably isnā€™t a good answer, but I donā€™t think there is one either; otherwise some crazy nix contributor would probably have already done it :slight_smile:

19

Part of the problem is that even nixos doesnā€™t actually know for sure whatā€™s in PATH at runtime. It comes from multiple sources, including home-manager and nix-env for user-installed stuff.

Part of the overall philosophy of the nix ecosystem is to use build-time resolution of resources wherever possible. This philosophy isnā€™t incidental. Itā€™s an important part of getting many of the benefits we get from it. PATH lookups are an exception to this philosophy, because you donā€™t want to reboot just to get a new package installed, but this doesnā€™t seem like it qualifies for an exception to me. If the current UX is bad, we need to make the build-time resolutionā€™s configuration process easier, not start trying to guarantee runtime resolution will work. It canā€™t even be done in environments like nix-env and nix profile anywayā€¦ not without a major overhaul to the architecture of nix itself, which I would view as quite detrimental.

20

With CA derivations, it might be possible to add a hardware drivers stub, and then people donā€™t have to rely on any runtime hacks to find the more specific hardware libraries.

The stub is just there so that the actual drivers only have a single downstream package. This should make it cheap for the contents to be replaced with something else without having to rebuild the actual immediate downstream dependencies.