NixOS only packages

nyarly · June 27, 2018, 4:32pm

I repeatedly run into the case where I’m trying to install a package using nix-env, only to have it not work at all because it has to be installed within my configuration.nix.

The current case in point is zoom-us, which is a video conferencing app (afaict, like Skype). In common with other instances of this issue are that it uses the system’s video card, so it needs to build against the precise versions of the 3d drivers and libraries that are installed on the system.

My glancing blow with this issue makes it clear that it’s not a new discovery, and there are features of Nixpkgs designed to address them, to a degree. (e.g. there’s the unusual situation with how opengl libraries are linked to a non-store path).

What’s the right way to handle packages like zoom-us? Is the fact that they have to be a system-wide install something that can be fixed? Or can we institute a marker of some kind so that nix-env will bail with an informative error?

Mic92 · June 28, 2018, 9:58pm

OpenGL drivers is a special case that we cannot easily cover without kernel-bound distribution specific packages.

Maybe we could have a /run/opengl-driver/ in packages like GitHub - dezgeg/nix-fpm-multiuser: Multiuser distro packages for Nix with fpm to bridge the gap on non-nixos systems

zimbatm · June 29, 2018, 12:04pm

the OpenGL problem is quite difficult. @dezgeg has writtten a quite good analysis of it with a proposal on how to solve it over here:

github.com/NixOS/nixpkgs

Solve the libGL ABI problem

opened 02:23PM - 03 Nov 17 UTC

dezgeg

The problem =========== The design of libGL drivers is such that the userspa…ce part of the driver consists of a libGL.so that gets loaded in each process using OpenGL. That is, each driver vendor (Mesa, NVIDIA, AMD) ships their own libGL.so that we select dynamically (and impurely) by having NixOS set: ```` LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib ```` and having the NixOS module set the symlinks pointing to the proper packages depending on the system configuration. Now while the OpenGL ABI itself is stable, a major pain point for us that the impurity causes are conflicting library versions between any libraries that the driver itself _and_ the application depends on. Issue #16779 shows a manifestation of this problem: applications built on NixOS 16.03 would stop working on NixOS 16.09, because of a version conflict between libwayland.so used both by the application and Mesa: the application itself causes version X of libwayland.so being loaded to the process, but Mesa requires version Y of libwayland.so being loaded, thus the application cannot start up and fails with: Note that this problem is not inherently specific to NixOS -- the same problem is known to happen on other distros as well when the libstdc++ version provided by the Steam runtime conflicts with the libstdc++ that Mesa requires. A (potential) solution ====================== An attempt of solving this has been done in the libcapsule project (https://git.collabora.com/cgit/user/vivek/libcapsule.git/tree/README) by a Collabora employee. The approach taken there is to build a stub libGL.so that uses the little-known `dlmopen()` function to create a completely new symbol namespace for dynamic linking, and load the real libGL.so of the graphics driver there, and then redirect all exported symbols from the stub libGL.so to the entry points in the real libGL.so living in the segregated dynamic linker namespace. This is implemented via a clever hack of patching the PLT table of the stub libGL.so to point to the real libGL.so's entry points, so there is zero overhead for function calls to libGL! The problems in practice ------------------------ I attempted to package and use libcapsule during NixCon 2017, with not-so-great success (https://github.com/dezgeg/libcapsule, https://github.com/dezgeg/nixpkgs/tree/libcapsule). While approach taken by libcapsule seems theoretically sound, one problem seems to be that the proxied libGL driver needs to also provide exports for libX11.so among some other xcb libraries. I'm not totally sure why, but I'm guessing the X11 client driver keeps some per-process state on which GLX client-side library is associated with which X screen, so having two different libX11.so's in the main symbol namespace and inside the capsuled symbol namespace would break things. Now, that causes a problem because libraries like libXi (probably accidentally) allocate memory with `malloc()` from outside the capsule but free it with `XFree()`, which crashes because `XFree()` calls the `free()` inside the capsule, and those two glibc of course have their independent heaps. AFAICT, there's currently no way to have certain libraries loaded only once and shared by both the main `dlmopen()` namespace and the in-capsule `dlmopen()` namespace. A potential way to avoid that problem might be to try would be to use libcapsule between libglvnd and and the driver, which shouldn't require the hack of exporting symbols from libX11. Though what worries me a bit is whether having multiple glibcs loaded in will work either, given that there are some per-process and per-thread kernel APIs where the two glibcs might step on each others' feet. (`set_robust_list()` and `sbrk()` come to mind). But presumably the glibc people have given at least some thought to that, or the entire `dlmopen()` would become pretty much useless... cc: @vcunat, @abbradar

nyarly · June 29, 2018, 11:13pm

I sincerely hope I didn’t come off as trivializing the problem. I’ve thought about the issue exactly enough to realize that it’s very deep rabbit hole, and I’m impressed at the amount of effort that’s already been expended on it.

Mostly I wonder if it would be worth adding an annotation to expressions to the effect that “we know this won’t work unless you install them with the kernel, so don’t bother trying to nix-env for the moment”

(As an aside, libGL is one place, but BCC is another - because it compiles eBPF, it relies on details of the running kernel. I suspect there are more cases like BCC that aren’t as well surfaced, because there’s so many more programs that do heavyweight graphics.)

zimbatm · July 7, 2018, 1:18pm

Not at all.

Not at all. Nix shines because all the runtime dependencies can be pulled in the closure. And it breaks down when it’s can’t. It would be awesome to have a generic adapter that can be used to say “I need library XY with such and such ABI”. Unfortunately I don’t have enough C knowledge to know if it’s possible to for example extend the LD-preloader mechanism for that kind of things.