What's the proper way for software to download extra binaries on NixOS?

matklad · December 21, 2021, 3:25pm

When binary software is packaged for NixOS, we patch the binaries to use correct libraries in nixpkgs. Here’s an example for CLion, which includes a bunch of native helpers:

github.com

NixOS/nixpkgs/blob/ad6be174e114f8643f53af9d9982d7d18f999367/pkgs/applications/editors/jetbrains/default.nix#L47

    
      
                libdbusmenu
              ];
              dontAutoPatchelf = true;
              postFixup = (attrs.postFixup or "") + optionalString (stdenv.isLinux) ''
                (
                  cd $out/clion-${version}
                  # bundled cmake does not find libc
                  rm -rf bin/cmake/linux
                  ln -s ${cmake} bin/cmake/linux
          
          
        autoPatchelf $PWD/bin
          
          
        wrapProgram $out/bin/clion \
                    --set CL_JDK "${jdk}"
                )
              '';
            });
          
          
buildDataGrip = { name, version, src, license, description, wmClass, ... }:
            (mkJetBrainsProduct {
              inherit name version src wmClass jdk;

This seems like a reasonable thing to do. But what is the correct behavior for cases where the software itself downloads extra binaries from the internet? For example, CLion has a plugin system, and, IIRC, one on the things that the plugin can do is come packaged with some native binary, which the pluging shells out to from Java. What’s the proper way to support such software? Specifically, what could the authors of software who use “download binary from the internet and run it” do to make it friendlier to NixOS?

matklad · December 21, 2021, 3:28pm

Some extra example: this is what we do in rust-analyzer extenison for VS Cod to make rust-analyzer binary, downloaded at runtime, work:

github.com

rust-analyzer/rust-analyzer/blob/b65d9c3e62a55028236be84995d1ba43462f4a5c/editors/code/src/main.ts#L278-L319

    
      
          async function patchelf(dest: vscode.Uri): Promise<void> {
              await vscode.window.withProgress(
                  {
                      location: vscode.ProgressLocation.Notification,
                      title: "Patching rust-analyzer for NixOS"
                  },
                  async (progress, _) => {
                      const expression = `
                      {srcStr, pkgs ? import <nixpkgs> {}}:
                          pkgs.stdenv.mkDerivation {
                              name = "rust-analyzer";
                              src = /. + srcStr;
                              phases = [ "installPhase" "fixupPhase" ];
                              installPhase = "cp $src $out";
                              fixupPhase = ''
                              chmod 755 $out
                              patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" $out
                              '';
                          }
                      `;

This file has been truncated. show original

This doesn’t look particularly elegant, is there a more proper way here?

TLATER · December 21, 2021, 7:34pm

I’m not authorative on this, but in practice any runtime-downloading of things on NixOS goes completely against the philosophy of the platform. See also the horrors of packaging anything Gradle or npm related. That binaries always fail under these circumstances is not necessarily intended, but it’s also just not supported at all.

I think the optimal way of solving the problem is to have a derivation for the hidden runtime dependency, fetched from the real upstream, and to patch the application that uses it to use the binary from that derivation.

This puts the maintenance responsibility and trust back with NixOS, which is ideal. There are quite a few go and npm packages that do horrible things like downloading binaries that a project owner hosts on a personal site - having dependencies like that is a security (and stability) nightmare in my opinion.

If upstream is cooperative, an environment variable which specifies where to take the binary from is usually nicest. Nix derivations can then use makeWrapper to override the download feature and specify a proper, packaged binary to use. An “offline mode” that doesn’t even attempt to download things is also useful.

Not all upstreams are cooperative though (NixOS is still a tiny subset of users), and plugin systems like the clion one can make this kind of thing infeasible depending on how free-form they are.

ElvishJerricco · December 21, 2021, 9:29pm

Generally I think that programs that must operate this way (e.g. Steam) should just be run in a buildFHSUserEnv, but that programs should generally be made to avoid this altogether. Like for VS Code extensions that include binaries, I’d rather package the extension itself with Nix, and include a patchelf in the installPhase rather than having the extension invoke nix itself (I can already see several ways the rust-analyzer example can fail).