That may well be clobbering the PATH you are setting. It looks like you could probably test this theory by also setting export SPARK_ENV_LOADED=1 (probably not a real fix unless you also take over setting some of what is in bin/load-spark-env.sh.)
I guess this is just the version string baked in to the installPhase from the original derivation–so that part might be fine–but it is possible that there have been changes between 2.4.3 and 3.1.1, especially given the major version bump, that the existing package may not account for.
There may ultimately be some sort of incompatibility between 2.x and 3.x that will cause trouble, but I think there are still a few simple-ish things to try before coming to that conclusion.
Can you try:
removing export SPARK_ENV_LOADED=1 if you still have it set
removing PATH from your shellHook
in your override of spark, append procps to its buildInputs with something like: buildInputs = old.buildInputs ++ [ pkgs.procps ];
For a little context:
I think you’re just running into some wonkiness around a ~gap in the Nix ecosystem roughly caused by the fact that shell scripts aren’t compiled:
It’s easy for packagers to miss dependencies in shell scripts because Nix doesn’t have a process that’ll break/fail due to missing commands at build time.
When a package’s scripts contain bare command invocations, we either have to:
add (~leak) all of the script’s runtime dependencies to the user or system PATH
find some way to patch in a fixed PATH at build time
I suspect #2 explains why the nixpkgs derivation for spark builds a fixed PATH into conf/spark-env.sh, and #1 explains why ps is missing (there may also be more).
(I have been building resholve to address this ecosystem gap [i.e., resholve breaks a build when a dependency is missing, and rewrites bare invocations to absolute store paths], but the initial Nix API is focused on straightforward packages of shell scripts. The Nix integration needs more work before it can ~easily tackle cases like this.)