Hello.
Motivation for this post: OpenGL (And Cuda / OpenCL, …) applications from nix does not work on non nixOS system by default and may not work on nixOS. I wanted to make this point clear. It can be a real blocker for the adoption of nix. Said otherwise, no game, no web browser (at least WebGL and web browser with native OpenGL acceleration). No video editor. No blender, perhaps gimp, … can be used or distributed easily using nix.
I’m the author of GitHub - nix-community/nixGL: A wrapper tool for nix OpenGL application [maintainer=@guibou] which tries to mitigate this issue and I’m here to discuss the future of this tool. I’ll briefly explain the problem with OpenGL, the nixGL solution and limitations and then discuss how the tool can be improved.
I really need the community advice on that matter, and especially, if you know or want to discuss a better solution, please tell me.
OpenGL problem
OpenGL does not work out of the box with nix for a simple reason: the OpenGL driver cannot be part of the reproducible environment because it will depends on the target hardware (i.e. the hardware on which the program will be run).
There are a lot of different hardwares associated with different drivers, mostly Nvidia, AMD and Mesa. Note that most of the difficulties comes from proprietary drivers, and mesa is really simple to patch / setup. However the quality of the mesa driver (performance / features) is not enough, so we cannot simply ignore the proprietaries drivers.
Nix derivation are built without any knowledge of the future hardware and this will pose a problem at runtime. When run, an OpenGL application must link with an OpenGL library which is compatible with the hardware available on the target system. On most distribution, this library is part of the system and available inside /usr/lib or a similar directory.
The problem with nix is that a nix derivation won’t try (by default) to load the OpenGL driver which is available on the target system and will only load the OpenGL library which was used during the build of the derivation (and which is usually the mesa library). Most of the time this library is not compatible with the hardware of the target system and OpenGL initialization fails (with cryptic error message).
A solution may be to force the program to load the library provided by the system (using LD_LIBRARY_PATH for example). But it does not work most of the time because this library may have symbols or dependencies which are not compatible with the one used in the binary. I remember having conflicts with libcrypt and libc symbols.
More detail can be found in libGL not working on non-NixOS (without setting up) · Issue #9415 · NixOS/nixpkgs · GitHub or on the nixGL github repository.
NixGL solution
I created NixGL to solve this issue. It uses LD_LIBRARY_PATH to load from nixpkgs a compatible OpenGL driver, which is a library compatible with the current hardware AND using the same libraries as the one used in the executable. nixGL is as simple as that (everything fits in a short nix file), plus a few wrappers to handle different cases such as hybrid systems or cuda.
NixOS
NixOS does not suffer initially from this issue because it globally installs an OpenGL library compatible with the current nixpkgs channel. But it may stop working if you are running an executable built from a different nixpkgs commit. Most of the time the incompatibilities are due to differences in glibc symbols.
I’m insisting on this point because it means that the current OpenGL situation in NixOS is still imperfect. For example, in my work, I have a client which uses a nixpkgs clone which is sometimes more recent, sometimes older than my current nixos channel, and I’m forced to use nixGL to run my client binaries on nixos. And my client is forced to use nixGL to run his binary on another distribution.
NixGL example
If you want to run binary foo using nvidia driver on a specific nixpkgs clone, you must setup in your environment buildInputs the following:
nixGL = (callPackage ./path/to/nixGL.nix {
nixpkgs = yourCurrentNixpkgs;
nvidiaVersion = "341.23";
nvidiaHash = "checksum of the nvidia binary installer for 341.23";
}).nixGLNvidia
This will provide the nixGL wrapper in your environment, allowing you to call foo using nixGL foo.
As you see, users must pass the current nixpkgs clone (actually this may be improved using callPackage logic). But also the current nvidiaVersion and nvidiaHash.
The git repository comes with a script to automatically compute the hash of the nvidia installer, but this cannot be setup inside a reproducible environment.
NixGL limitations
NixGL works and solves the problem in a robust way. However it is a pain to use:
- all nix binaries must be manually wrapped by a
nixGLcall. This can be done in the shell (when you execute the program) or done in a wrapper script, but see following points. -
nixGLmust be installed / updated when thenixpkgscommit OR the hardware used change. This means thatnixGLcannot be part of a reproducible build environment, because it depends on the current hardware. So it is usually setup outside of the project and you must think about keeping it up to date with your hardware / nixpkgs requirements. I saw setups where users were saving their nixGL setup as part of their repository, but this setup had to be overrode manually on each different hardware. - At least for nvidia driver: the hash and version must be provided and the version must exactly match the one used on the target system. This is extremely painful for users. Mesa drivers does not suffer from this limitation because the mesa library seems to work with most (if any) installed kernel driver.
- nixGL is not in nixpkgs, so users must fetch it from github. I receive a lot of messages asking for the integration in
nixpkgs. I never took the time to propose a PR because I initially thought thatnixGLwill only be used by a minority of users and stay as a hack state.
Design ideas
This is the part I’d like you to help me write. I have some ideas:
Simple things
- Propose a pull request so the nixGL script will be integrated in nixpkgs. It will be easier for users.
- I’m thinking about integrating a list of all the nvidia driver + hash. This way, users won’t have to provide the hash of their nvidia binary. This will need more work from the maintainers, but will be more robust.
Driver versions
The biggest limitation (from my point of view) is the system dependent setting. I can transform nixGL to a script which detect (at runtime) which OpenGL driver is needed, but I don’t really know how to do that in a robust way. I can call the system glxinfo, but I need to be sure that it’s there.
This may also appear in nix configuration, somewhere. I can imagine a nixGL section in user nix configuration. However users will have to keep that one in sync with his current hardware value.
Nixpkgs versions
How can I handle different versions of nixpkgs? The current (and simplest) solution, is to install one nixGL per nixpkgs clone. This is a manual task done by the user for each of his OpenGL project. It is boring and error prone. I have no idea on how a globally installed nixGL tool can detect the nixpkgs clone used by the current binary.
However there is still an issue with a global nixgl which finds a way to autodetect the current nixpkgs clone. Just having a git commit hash for nixpkgs is not enough: users may had overrode the glibc in their overlay.
Build time wrapper script
I was also thinking about creating a wrapper (using makeWrapper) which, at nix build time, setups the nixGL wapper. This may be applied unconditionally on all derivation which uses openGL and the wrapper can point to the right nixpkgs clone (by serializing nixpkgs at build time, so all override are taken into account). But this poses other problem, people may want to switch their nixGL implementation at runtime (for example, to use hybrid graphic cards). I was thinking about a --nixGL-implementation command line flag which may be consumed by the wrapper.
Conclusion
I have a lot of ideas and the motivation to implement all of them. First tasks will be to sumbit a pull request for integration with nixpkgs and another with nvidia hash list. I’ll be interested by feedbacks on the idea / feasibility of storing the driver configuration inside nix.conf.
