Design discussion about nixGL (OpenGL / Cuda / OpenCL wrapper for nix)

Hello.

Motivation for this post: OpenGL (And Cuda / OpenCL, …) applications from nix does not work on non nixOS system by default and may not work on nixOS. I wanted to make this point clear. It can be a real blocker for the adoption of nix. Said otherwise, no game, no web browser (at least WebGL and web browser with native OpenGL acceleration). No video editor. No blender, perhaps gimp, … can be used or distributed easily using nix.

I’m the author of https://github.com/guibou/nixGL which tries to mitigate this issue and I’m here to discuss the future of this tool. I’ll briefly explain the problem with OpenGL, the nixGL solution and limitations and then discuss how the tool can be improved.

I really need the community advice on that matter, and especially, if you know or want to discuss a better solution, please tell me.

OpenGL problem

OpenGL does not work out of the box with nix for a simple reason: the OpenGL driver cannot be part of the reproducible environment because it will depends on the target hardware (i.e. the hardware on which the program will be run).

There are a lot of different hardwares associated with different drivers, mostly Nvidia, AMD and Mesa. Note that most of the difficulties comes from proprietary drivers, and mesa is really simple to patch / setup. However the quality of the mesa driver (performance / features) is not enough, so we cannot simply ignore the proprietaries drivers.

Nix derivation are built without any knowledge of the future hardware and this will pose a problem at runtime. When run, an OpenGL application must link with an OpenGL library which is compatible with the hardware available on the target system. On most distribution, this library is part of the system and available inside /usr/lib or a similar directory.

The problem with nix is that a nix derivation won’t try (by default) to load the OpenGL driver which is available on the target system and will only load the OpenGL library which was used during the build of the derivation (and which is usually the mesa library). Most of the time this library is not compatible with the hardware of the target system and OpenGL initialization fails (with cryptic error message).

A solution may be to force the program to load the library provided by the system (using LD_LIBRARY_PATH for example). But it does not work most of the time because this library may have symbols or dependencies which are not compatible with the one used in the binary. I remember having conflicts with libcrypt and libc symbols.

More detail can be found in https://github.com/NixOS/nixpkgs/issues/9415 or on the nixGL github repository.

NixGL solution

I created NixGL to solve this issue. It uses LD_LIBRARY_PATH to load from nixpkgs a compatible OpenGL driver, which is a library compatible with the current hardware AND using the same libraries as the one used in the executable. nixGL is as simple as that (everything fits in a short nix file), plus a few wrappers to handle different cases such as hybrid systems or cuda.

NixOS

NixOS does not suffer initially from this issue because it globally installs an OpenGL library compatible with the current nixpkgs channel. But it may stop working if you are running an executable built from a different nixpkgs commit. Most of the time the incompatibilities are due to differences in glibc symbols.

I’m insisting on this point because it means that the current OpenGL situation in NixOS is still imperfect. For example, in my work, I have a client which uses a nixpkgs clone which is sometimes more recent, sometimes older than my current nixos channel, and I’m forced to use nixGL to run my client binaries on nixos. And my client is forced to use nixGL to run his binary on another distribution.

NixGL example

If you want to run binary foo using nvidia driver on a specific nixpkgs clone, you must setup in your environment buildInputs the following:

nixGL = (callPackage ./path/to/nixGL.nix {
     nixpkgs = yourCurrentNixpkgs;
     nvidiaVersion = "341.23";
     nvidiaHash = "checksum of the nvidia binary installer for 341.23";
}).nixGLNvidia

This will provide the nixGL wrapper in your environment, allowing you to call foo using nixGL foo.

As you see, users must pass the current nixpkgs clone (actually this may be improved using callPackage logic). But also the current nvidiaVersion and nvidiaHash.

The git repository comes with a script to automatically compute the hash of the nvidia installer, but this cannot be setup inside a reproducible environment.

NixGL limitations

NixGL works and solves the problem in a robust way. However it is a pain to use:

  • all nix binaries must be manually wrapped by a nixGL call. This can be done in the shell (when you execute the program) or done in a wrapper script, but see following points.
  • nixGL must be installed / updated when the nixpkgs commit OR the hardware used change. This means that nixGL cannot be part of a reproducible build environment, because it depends on the current hardware. So it is usually setup outside of the project and you must think about keeping it up to date with your hardware / nixpkgs requirements. I saw setups where users were saving their nixGL setup as part of their repository, but this setup had to be overrode manually on each different hardware.
  • At least for nvidia driver: the hash and version must be provided and the version must exactly match the one used on the target system. This is extremely painful for users. Mesa drivers does not suffer from this limitation because the mesa library seems to work with most (if any) installed kernel driver.
  • nixGL is not in nixpkgs, so users must fetch it from github. I receive a lot of messages asking for the integration in nixpkgs. I never took the time to propose a PR because I initially thought that nixGL will only be used by a minority of users and stay as a hack state.

Design ideas

This is the part I’d like you to help me write. I have some ideas:

Simple things

  • Propose a pull request so the nixGL script will be integrated in nixpkgs. It will be easier for users.
  • I’m thinking about integrating a list of all the nvidia driver + hash. This way, users won’t have to provide the hash of their nvidia binary. This will need more work from the maintainers, but will be more robust.

Driver versions

The biggest limitation (from my point of view) is the system dependent setting. I can transform nixGL to a script which detect (at runtime) which OpenGL driver is needed, but I don’t really know how to do that in a robust way. I can call the system glxinfo, but I need to be sure that it’s there.

This may also appear in nix configuration, somewhere. I can imagine a nixGL section in user nix configuration. However users will have to keep that one in sync with his current hardware value.

Nixpkgs versions

How can I handle different versions of nixpkgs? The current (and simplest) solution, is to install one nixGL per nixpkgs clone. This is a manual task done by the user for each of his OpenGL project. It is boring and error prone. I have no idea on how a globally installed nixGL tool can detect the nixpkgs clone used by the current binary.

However there is still an issue with a global nixgl which finds a way to autodetect the current nixpkgs clone. Just having a git commit hash for nixpkgs is not enough: users may had overrode the glibc in their overlay.

Build time wrapper script

I was also thinking about creating a wrapper (using makeWrapper) which, at nix build time, setups the nixGL wapper. This may be applied unconditionally on all derivation which uses openGL and the wrapper can point to the right nixpkgs clone (by serializing nixpkgs at build time, so all override are taken into account). But this poses other problem, people may want to switch their nixGL implementation at runtime (for example, to use hybrid graphic cards). I was thinking about a --nixGL-implementation command line flag which may be consumed by the wrapper.

Conclusion

I have a lot of ideas and the motivation to implement all of them. First tasks will be to sumbit a pull request for integration with nixpkgs and another with nvidia hash list. I’ll be interested by feedbacks on the idea / feasibility of storing the driver configuration inside nix.conf.

2 Likes

I think it would also make sense to have a home-manager module. Home-manager already provides a way to maintain session variables, so exporting LD_LIBRARY_PATH should be easy. As it maintains a single nix profile just as NixOS you should not have the problem of multiple nixpkgs versions as long everything is installed through home-manager. Your detection script might however require some impurity if you want to automatically detect the right version.

I did not thought about home-manager, that’s a great idea! That’s not “generic” enough (I won’t force users to install another tool to have OpenGL working), but it may be a really simple solution for home-manager users. I’ll have a look at this today.

However it only fixs the nixpkgs clone version issue if you assume that everything is installed using the same nixpkgs clone. One of the most annoying use case I’m facing is that I’m working on many professional projects which uses different nixpkgs clone and I really need one nixGL per project, else they won’t work.

The latter could be solved by having nixGL as part of nixpkgs so one can make sure it is compatible.

1 Like

Ive never had issues developing openGL projects with Nix, even when working on multiple machines. :man_shrugging:t3:

I’ll try to make an rss feed out of these pages:
https://www.nvidia.com/en-us/drivers/unix/linux-amd64-display-archive/
https://www.nvidia.com/en-us/drivers/unix/

The problem is they don’t include a version that was installed in my ubuntu before I upgraded the nvidia-driver, it was 440.33.

And that’s not the end of story. There are also 440.33.00 and 440.33.01 versions that are unlisted in the archive either. I should probably ask maintainers from ubuntu if they have their own archive, I couldn’t find anything like that by myself.

If I understand correctly, nixGL currently uses drivers which are in nixpkgs.
Would it also be possible to specify the location of an existing host system library and use that one?

Use case: I am experimenting with an aarch64 Nix installation on a Sailfish OS phone. These weird mobile graphics chips obviously work neither with Intel nor with Nvidia drivers, but have a weirdly patched libGLES* lying around.

That would actually be awesome - i am trying to use nix on the Nvidia Jetson Platform (also aarch64), which use special cuda drivers which you can not get easily.

2 Likes