Wouldn’t hydra have to compile every package that uses the stub with every possible driver to verify that they result in the same content addressable output?
Part of the problem is that even nixos doesn’t actually know for sure what’s in PATH at runtime.
No, but I think that looking through bin in the profile would work as a heuristic. We could assume that it will always be included in $PATH. Anything that’s dynamically added to $PATH won’t be accounted for, but by default $PATH only includes paths to profiles in NixOS.
Also, the problem is that there exist packages that will always depend on impure system resources, there’s no way we can avoid that. I think we may be able to avoid treating executables as “impure system resources”, but I don’t think we could ever get rid of impure required paths, and right now we have no way to encode those dependencies.
Except we don’t have the bin in the profile yet, when we’re considering adding a package to it…
Except we don’t have the
binin the profile yet, when we’re considering adding a package to it…
We don’t have to run the resolution check before creating the profile, we could run it afterwards.
Unless I misunderstand CA derivations, this check should only be for the stub. If we can somehow get the stub to assert equivalence across mesa and nvidia, then rebuilding all of the other downstream packages should be very cheap.
1 Like
Unless I misunderstand CA derivations, this check should only be for the stub. If we can somehow get the stub to assert equivalence across mesa and nvidia, then rebuilding all of the other downstream packages should be very cheap.
Ooh, ok I see what you mean. I thought you meant replacing the stub itself rather than replacing the inputs used to build the stub.
1 Like
I think we may be able to avoid treating executables as “impure system resources”
Oh wait no we can’t. Any executables that depend on Linux capabilities can’t be hard-coded. (eg. pkexec, sudo, ping, …)
I’ve had the sense through this conversation that the problems in this thread rhyme with some GitHub - abathur/resholve: a shell resolver? :) (find and resolve shell script dependencies) has with nailing down shell script dependencies. To be useful, setuid executables (including some really common ones) either need wrappers on NixOS or to come from the system on other platforms, but Nix doesn’t provide an ~oracle for discovering those paths at build time.
(I think this is a bit of a sharp corner for new Nix users broadly–look for examples of people trying to use pkgs.sudo–but you can read a little more about how it affects resholve in Nix: inability to resolve some ~wrapped / impure setuid executables such as `sudo` · Issue #29 · abathur/resholve · GitHub).
I would probably frame my problem the other way around (I don’t think resholve users broadly want the targets to be fungible), but the solution might be the same: a build-time oracle that can yield valid runtime paths without allowing those paths to be directly accessed from the build sandbox?
1 Like
Maybe I misunderstand the nature of this problem, but why can’t the dependencies be declared as inputs to the package much like inputs for flakes?
1 Like
Maybe I misunderstand the nature of this problem, but why can’t the dependencies be declared as inputs to the package much like
inputsfor flakes?
- A lot of packages are designed to support swapping dependencies at runtime, and forcing a dependency at build-time would remove this flexibility. (eg. winetricks can use any build of wine, and it should use the one in
$PATHby default, but will fail at runtime if wine isn’t in$PATH). - Adding a default fallback with
makeWrapper--suffix PATH :“works”, but increases the closure size, and includes an extra default dependency that you may not want. - There are certain executables that require privilege escalation through Linux capabilities. During a build, the capability flags are stripped during for security, and the only way to add them back is to use security wrappers in NixOS. We can only refer to these executables through
$PATHif we want to use the privileged features of that executable. - We have no way to describe that a package requires a specific path to exist at runtime (eg.
/etc/<package>.conf,/run/opengl-driver/lib, …) - Packages that don’t have a separate wrapper derivation will have to be rebuilt from source when given different inputs.
1 Like
I think that declaring a dependency for the package doesn’t have to preclude runtime access to alternatives. Rather, it just ensures that at least one will exist.
The difficult part may be identifying the API to the dependency (such as select items on $PATH) and which packages can satisfy that API. This will be different for every package.
When I want to use a different dependency, I’m going to install (realize) the new dependency. I might as well do so with a derivation associating the dependency with its dependent packages.
We do this now with various plugin-based tools such as vscode and python. How are winetricks and others are any different (conceptually)?
1 Like
The difficult part may be identifying the API to the dependency (such as select items on
$PATH) and which packages can satisfy that API. This will be different for every package.
Yeah, it would be very difficult to try to describe an API. We wouldn’t even be able to verify it even if we did. Right now I’m only suggesting that we encode a way to say an executable needs to exist in $PATH or a that path needs to exist. It’s not perfect, but I think it will help move a lot of runtime dependency errors to build time.
We do this now with various plugin-based tools such as vscode and python. How are
winetricksand others are any different (conceptually)?
For things like winetricks (depending on wine) & texlab (depending on a tex distribution) it’s more about ergonomics. By forcing a dependency at build time, you are now forced to use Nix to swap out dependencies, unless you allow overriding the default at runtime with suffix PATH :.
The problem is that it’s not obvious when a package has a different default from what you installed to $PATH, and you can be unnecessarily bloating your install if you’re not explicitly overriding the package to use the same build.
For winetricks & texlab, a better default is to use the “system installed package”. Which is why we don’t hardcode wine & a tex distribution right now, but if those aren’t installed, you’ll get runtime errors.
1 Like
In some sense the system derivation is that derivation today 
It’s just not very fine grained or well specified.
nixosConfigurations.lun-kosame-nixos.config.system.build.toplevel
edit: I guess the nixos module system right now has a lot of adhoc encoding of “this must be available at runtime” both by services enabling other services when turned on, and by options having assertions that check something is or isn’t enabled.
Encoding these assumptions in the NixOS modules of course works better for services, but not so well for programs you expect to use with nix run.
2 Likes
For certain packages, it could make sense introducing an environment variable into the wrapper that holds the path to the dependency. This may even be something that could be upstreamed. E.g., like some Wine-related tools already support executing $WINE instead of the wine executable from the PATH. The advantage I see in that is that it is more opt-in than introducing a system-wide path where stuff is picked up from implicitly. Some meta-attribute could document any run-time environment variables that are meaningful to the respective wrapper.
For build-time aspects like this, it may also be an option to have users override the derivation if a different default dependency is desired. Even in cases where one would like to switch dependencies at run-time, it seems okay to presume that at least one dependency should be pulled in when installing the package.
Is it really very helpful to require the existence of a path regardless of what might contained in it? We probably also would like to express its contents more formally. Of course, then the contents will wind up in the Nix store. Depending on how the run-time dependency gets picked up, this could be problematic. Such cases could be dealt with by working with upstream to support a more declarative way in which to wire up the process. If that doesn’t work, it’s possible that the only remaining way to solve the problem without sacrificing declarativity may be on the level NixOS rather than Nix.
This is a good pointer. Maybe we should consider more consistently splitting off wrappers into a secondary derivation. If this makes sense, it could be something supported by nixpkgs and the docs.
2 Likes