Grpc-tools in node programs tries to access internet while building

Help
1

I’m trying to package robotframework-browser GitHub - MarketSquare/robotframework-browser: Robot Framework Browser library powered by Playwright. but I get an error:

node_modules/color-support/bin.js: interpreter directive changed from "#!/usr/bin/env node" to "/nix/store/azpqgrv58akzqkj8fq8kb5dyc2jjk7zk-nodejs-24.2.0/bin/node"
node_modules/handlebars/bin/handlebars: interpreter directive changed from "#!/usr/bin/env node" to "/nix/store/azpqgrv58akzqkj8fq8kb5dyc2jjk7zk-nodejs-24.2.0/bin/node"
node_modules/handlebars/print-script: interpreter directive changed from "#! /usr/bin/env node" to "/nix/store/azpqgrv58akzqkj8fq8kb5dyc2jjk7zk-nodejs-24.2.0/bin/node"
npm warn Unknown env config "nodedir". This will stop working in the next major version of npm.
npm error code 1
npm error path /build/source/node_modules/grpc-tools
npm error command failed
npm error command sh -c node-pre-gyp install
npm error request to https://node-precompiled-binaries.grpc.io/grpc-tools/v1.13.0/linux-x64.tar.gz failed, reason: getaddrinfo EAI_AGAIN node-precompiled-binaries.grpc.io
npm error [info] it worked if it ends with ok
npm error [info] using node-pre-gyp@2.0.1
npm error [info] using node@22.16.0 | linux | x64 
npm error [info] check checked for "/build/source/node_modules/grpc-tools/bin/grpc_tools.node" (not found)
npm error [log] GET https://node-precompiled-binaries.grpc.io/grpc-tools/v1.13.0/linux-x64.tar.gz
npm error [error] install request to https://node-precompiled-binaries.grpc.io/grpc-tools/v1.13.0/linux-x64.tar.gz failed, reason: getaddrinfo EAI_AGAIN node-precompiled-binaries.grpc.io
npm error [error] install error
npm error [error] stack FetchError: request to https://node-precompiled-binaries.grpc.io/grpc-tools/v1.13.0/linux-x64.tar.gz failed, reason: getaddrinfo EAI_AGAIN node-precompiled-binaries.grpc.io
npm error     at ClientRequest.<anonymous> (/nix/store/ij7fva1mmk60f0bg4c4yl235ydbaiy8k-node-pre-gyp-2.0.1/lib/node_modules/@mapbox/node-pre-gyp/node_modules/node-fetch/lib/index.js:1501:11)
npm error     at ClientRequest.emit (node:events:518:28)
npm error     at emitErrorEvent (node:_http_client:104:11)
npm error     at TLSSocket.socketErrorListener (node:_http_client:518:5)
npm error     at TLSSocket.emit (node:events:518:28)
npm error     at emitErrorNT (node:internal/streams/destroy:170:8)
npm error     at emitErrorCloseNT (node:internal/streams/destroy:129:3)
npm error     at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
npm error [error] System Linux 6.12.34
npm error [error] command "/nix/store/pjr8jcds298brhwy1d3rmym9vayxhbfs-nodejs-22.16.0/bin/node" "/nix/store/ij7fva1mmk60f0bg4c4yl235ydbaiy8k-node-pre-gyp-2.0.1/lib/node_modules/@mapbox/node-pre-gyp/bin/node-pre-gyp" "install"
npm error [error] cwd /build/source/node_modules/grpc-tools
npm error [error] node -v v22.16.0
npm error [error] node-pre-gyp -v v2.0.1
npm error [error] not ok
npm error Log files were not written due to an error writing to the directory: /nix/store/zfjz0bvr7s2blggz7y4izndavsp4wg2f-npm-deps/_logs
npm error You can rerun the command with `--loglevel=verbose` to see the logs in your terminal

Any idea how to get rid of it?

My current derivation:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPypi,
  pythonOlder,
  robotframework,
  robotframework-pythonlibcore,
  robotframework-assertion-engine,
  grpcio,
  grpcio-tools,
  protobuf,
  overrides,
  click,
  seedir,
  wrapt,
  importNpmLock,
  npmHooks,
  nodejs_latest,
  fetchNpmDeps,
  fetchFromGitHub,
  node-pre-gyp,
  grpc-tools,
}:
# Maybe get inspiration from pkgs/by-name/fr/froide/package.nix
buildPythonPackage rec {
  version = "19.6.0";
  pname = "robotframework-browser";

  disabled = pythonOlder "3.9";

  nativeBuildInputs = [
    #importNpmLock.npmConfigHook
    npmHooks.npmConfigHook
    # npmHooks.npmInstallHook
    nodejs_latest
    node-pre-gyp
    grpc-tools
  ];
  npmDeps = fetchNpmDeps {
    #inherit src;
    src = fetchFromGitHub {
      owner = "MarketSquare";
      repo = "robotframework-browser";
      rev = "v${version}";
      hash = "sha256-kOifD+Fa1WCYbNZSRX89FlZio9oJ7Rh6aXQGVmVPXRI=";
    };
    hash = "sha256-paVNge7wlG+znGyc9wXHKYF4fpQUotREq/iddU1Pafo=";
  };
  # npmDeps = importNpmLock {
  #   # ? How to use the same path as src? So far it complains it is compressed
  #   npmRoot = ".";
  # };

  preBuild = ''
    echo "running npm i"
    #npm --loglevel verbose install
    #echo "after npm i"
    ls -al
  '';

  src = fetchFromGitHub {
    owner = "MarketSquare";
    repo = "robotframework-browser";
    rev = "v${version}";
    hash = "sha256-kOifD+Fa1WCYbNZSRX89FlZio9oJ7Rh6aXQGVmVPXRI=";
  };
  # src = fetchPypi {
  #   inherit version;
  #   pname = "robotframework_browser";
  #   sha256 = "sha256-7f0FeqoMaSqXhr4gofULMUV7DOjiqvQoc5CBn/WQOhQ=";
  # };

  # We fake a run of rfbrowser init
  patchPhase = ''
    runHook prePatch
    substituteInPlace ./Browser/playwright.py \
      --replace-fail '(installation_dir / "node_modules").is_dir()' 'True'
    runHook postPatch
    echo "BBBBB Done postPatch!"
  '';

  dependencies = [
    robotframework
    robotframework-pythonlibcore
    robotframework-assertion-engine
    grpcio
    grpcio-tools
    protobuf
    overrides
    click
    seedir
    wrapt
  ];

  postInstall = ''
  #runHook preInstall
  echo "AAAAAAAAAAAAAAaa"
  ls -al
  echo "dist"
  ls -al dist
  echo "fetch"
  echo ${npmDeps}
  ls ${npmDeps}
  #runHook postInstall
  '';

  meta = with lib; {
    description = "Robot Framework Browser library powered by Playwright. ";
    homepage = "https://robotframework-browser.org/";
    license = licenses.asl20;
    maintainers = with maintainers; [ tobiasBora ];
  };

}

(I also needed to trivially bump versions of grpcio-* programs)

2

Ok, so I finally understand what’s happening: grpc-tools is originally a C++ program, but it was packaged in nodejs with a trivial program that downloads pre-built C++ binaries. Nix does not like this, so they packaged grpc-tools themself starting from the C++ code directly. So my first trick was to remove grpc-tools from the package.json and include grpc-tools in the native build inputs. The second trick was to realize that nix forgot to also create the binary packages created by npm (these are trivial wrappers grpc-node/packages/grpc-tools/bin/protoc.js at 179dbfaeccc19bce786788a6b8e986990ab51329 · grpc/grpc-node · GitHub), so I manually added them like:

diff --git a/pkgs/by-name/gr/grpc-tools/package.nix b/pkgs/by-name/gr/grpc-tools/package.nix
index 5bf09b7f0b94..221727e5296a 100644
--- a/pkgs/by-name/gr/grpc-tools/package.nix
+++ b/pkgs/by-name/gr/grpc-tools/package.nix
@@ -25,6 +25,17 @@ stdenv.mkDerivation rec {
   installPhase = ''
     install -Dm755 -t $out/bin grpc_node_plugin
     install -Dm755 -t $out/bin deps/protobuf/protoc
+    # The node script creates two additional binaries that just forward their inputs to the above programs,
+    # but it seems unnecessary to install node etc just for this. So let's fake it using regular bash.
+    # https://github.com/grpc/grpc-node/blob/179dbfaeccc19bce786788a6b8e986990ab51329/packages/grpc-tools/package.json#L19-L21
+    cat >$out/bin/grpc_tools_node_protoc <<EOL
+    #/usr/bin/env bash
+    $out/bin/protoc --plugin=protoc-gen-grpc=$out/bin/grpc_node_plugin "\$@"
+    EOL
+    chmod +x $out/bin/grpc_tools_node_protoc
+    # grpc_tools_node_protoc_plugin seems to literally be the same as grpc_node_plugin
+    # except with a node wrapper
+    ln -s $out/bin/grpc_node_plugin $out/bin/grpc_tools_node_protoc_plugin
   '';

So I finally managed to get a fully working robotframework-browser packaged purely in nix:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPypi,
  pythonOlder,
  robotframework,
  robotframework-pythonlibcore,
  robotframework-assertion-engine,
  grpcio,
  protobuf,
  overrides,
  click,
  seedir,
  wrapt,
  npmHooks,
  nodejs_latest,
  fetchNpmDeps,
  fetchFromGitHub,
  node-pre-gyp,
  grpc-tools,
  grpcio-tools, # needed to generate --grpc_python_out + imported
  # For inv build
  invoke, # cmake-like for python
  mypy-protobuf,
  robotstatuschecker,
  pytest,
  beautifulsoup4,
  # To provide the binaries
  playwright-driver,
}:
let
  version = "19.6.0";
  pname = "robotframework-browser";
  src_orig = fetchFromGitHub {
    owner = "MarketSquare";
    repo = "robotframework-browser";
    rev = "v${version}";
    hash = "sha256-kOifD+Fa1WCYbNZSRX89FlZio9oJ7Rh6aXQGVmVPXRI=";
  };
  src = stdenv.mkDerivation {
    inherit pname;
    inherit version;
    src = src_orig;
    patches = [
      # Cf details on the above identical patch
      ./pin_playwright_to_nix_playwright_driver_version.diff
    ];
    buildPhase = "";
    installPhase = ''
      mkdir -p $out
      cp -Ra . $out
      cat $out/package.json
    '';
  };
in
buildPythonPackage rec {
  inherit pname;
  inherit version;

  disabled = pythonOlder "3.9";

  nativeBuildInputs = [
    #importNpmLock.npmConfigHook
    npmHooks.npmConfigHook
    # npmHooks.npmInstallHook
    nodejs_latest
    node-pre-gyp
    grpc-tools
    # For inv build
    invoke # cmake-like for python
    grpcio-tools # needed to generate --grpc_python_out
    mypy-protobuf
    robotstatuschecker
    pytest
    beautifulsoup4
  ];

  npmDeps = fetchNpmDeps {
    inherit src;
    hash = "sha256-5EYZ5t24aPWM12nGbBBu1nwb5QS27eWPCXT76JMIpoA="; # after patch
  };

  inherit src;

  # We fake a run of rfbrowser init
  # + grpc_tools_node_protoc already includes the plugin and is not runnable via npm
  # since nix directly builds it from C++ sources to avoid to use pre-build binaries
  # downloaded by npm
  patchPhase = ''
    runHook prePatch
    substituteInPlace ./Browser/playwright.py \
      --replace-fail '(installation_dir / "node_modules").is_dir()' 'True'
    substituteInPlace ./tasks.py \
      --replace-fail 'c.run("pip install -U pip")' 'return' \
      --replace-fail 'npm run grpc_tools_node_protoc' 'grpc_tools_node_protoc' \
      --replace-fail ' -- ' ' '
    substituteInPlace ./package.json \
      --replace-fail '"grpc-tools": "^1.13.0",' ' '
    sed -i '/--plugin=protoc-gen-grpc/d' ./tasks.py
    runHook postPatch
  '';

  dependencies = [
    robotframework
    robotframework-pythonlibcore
    robotframework-assertion-engine
    grpcio
    grpcio-tools
    protobuf
    overrides
    click
    seedir
    wrapt
  ];

  preBuild = ''
    inv build -d -e
  '';

  # makeWrapperArgs can't be used (not a script here but a library)
  postInstall = ''
    cp -r node_modules $out/lib
    cat >> $out/lib/python3.13/site-packages/Browser/__init__.py <<EOF
    import os
    if not "PLAYWRIGHT_BROWSERS_PATH" in os.environ:
        os.environ["PLAYWRIGHT_BROWSERS_PATH"] = "${playwright-driver.browsers}"
    if not "PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS" in os.environ:
        os.environ["PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS"] = "true"
    EOF
  '';

  meta = with lib; {
    description = "Robot Framework Browser library powered by Playwright. ";
    homepage = "https://robotframework-browser.org/";
    license = licenses.asl20;
    maintainers = with maintainers; [ tobiasBora ];
  };

}

Next step is to see if I can get a cleaner solution to synchronize the version of pkgs.playwright-driver.browsers and the playwright version used in robotframework… and do a PR eventually!