Are there rules to order of entries in PATH?

jakubgs · October 14, 2020, 12:12pm

Are there any concrete rules in what order things appear in PATH when provided in buildInputs or other arguments that modify PATH?

I made this simple derivation to test this:

{ pkgs ? import <nixpkgs> { } }:

pkgs.stdenv.mkDerivation {
  name = "nix-path-test";

  buildInputs = with pkgs; [ vim go ];

  phases = [ "buildPhase" ];
  buildPhase = ''
    echo $PATH | tr ':' '\n'
  '';
}

And the result I get is:

/nix/store/71n1xcigc00w3z7yc836jqcx9cb2dys8-patchelf-0.9/bin
/nix/store/m6h7zh8w6s52clnyskffj5lbkakqgywn-gcc-wrapper-9.2.0/bin
/nix/store/b3zsk4ihlpiimv3vff86bb5bxghgdzb9-gcc-9.2.0/bin
/nix/store/0k65d30z9xsixil10yw3bwajbdk4yskv-glibc-2.30-bin/bin
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin
/nix/store/n48b8n251dwwb04q7f3fwxdmirsakllz-binutils-wrapper-2.31.1/bin
/nix/store/hrkc2sf2883l16d5yq3zg0y339kfw4xv-binutils-2.31.1/bin
/nix/store/0k65d30z9xsixil10yw3bwajbdk4yskv-glibc-2.30-bin/bin
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin
/nix/store/qx3f6nrqgsb0y0sswy4zdw0sv86dwk9x-vim-8.2.0013/bin
/nix/store/fkvrggd8q3b543h6bal3clg9arzq6qvq-go-1.14.8/bin
/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin
/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/bin
/nix/store/dqq1bvpi3g0h4v05111b3i0ymqj4v5x1-diffutils-3.7/bin
/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/bin
/nix/store/b0vjq4r4sp9z4l2gbkc5dyyw5qfgyi3r-gnugrep-3.4/bin
/nix/store/c8balm59sxfkw9ik1fqbkadsvjqhmbx4-gawk-5.0.1/bin
/nix/store/g7dr83wnkx4gxa5ykcljc5jg04416z60-gnutar-1.32/bin
/nix/store/kkvgr3avpp7yd5hzmc4syh43jqj03sgb-gzip-1.10/bin
/nix/store/rw96psqzgyqrcd12qr6ivk9yiskjm3ab-bzip2-1.0.6.0.1-bin/bin
/nix/store/dp6y0n9cba79wwc54n1brg7xbjsq5hka-gnumake-4.2.1/bin
/nix/store/hrpvwkjz04s9i4nmli843hyw9z4pwhww-bash-4.4-p23/bin
/nix/store/xac1zfclx1xxgcd84vqb6hy3apl171n8-patch-2.7.6/bin
/nix/store/mm0w8jc58rn01c4kz2n9jvwd6bibcihs-xz-5.2.4-bin/bin

Which shows that both vim and go appear in the order they were specified in buildInputs, but they also appear in the middle of PATH, rather than in the beginning or at the end.

I assume the order of entries in PATH has to be deterministic, but what are some other rules that decide the order of things?

abathur · October 14, 2020, 5:50pm

I won’t claim to have looked closely enough to know, but in case no one else turns up with a definitive answer, in most cases this is governed by the implementation in https://github.com/NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh

jakubgs · October 14, 2020, 6:56pm

Thanks. That makes sense.

So I can see PATH being set from _PATH:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L603


      
          # temporary location and write a build report to $out.
          if [ -z "${prefix:-}" ]; then
              prefix="$out";
          fi
          
          
if [ "${useTempPrefix:-}" = 1 ]; then
              prefix="$NIX_BUILD_TOP/tmp_prefix";
          fi
          
          

          
PATH="${_PATH-}${_PATH:+${PATH:+:}}$PATH"
          HOST_PATH="${_HOST_PATH-}${_HOST_PATH:+${HOST_PATH:+:}}$HOST_PATH"
          if (( "${NIX_DEBUG:-0}" >= 1 )); then
              echo "final path: $PATH"
              echo "final host path: $HOST_PATH"
          fi
          
          
unset _PATH
          unset _HOST_PATH

Which appears to be configured here via addToSearchPath from within activatePackage:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L487


      
              fi
          
          
    # Only dependencies whose host platform is guaranteed to match the
              # build platform are included here. That would be `depsBuild*`,
              # and legacy `nativeBuildInputs`, in general. If we aren't cross
              # compiling, however, everything can be put on the PATH. To ease
              # the transition, we do include everything in thatcase.
              #
              # TODO(@Ericson2314): Don't special-case native compilation
              if [[ ( -z "${strictDeps-}" ||  "$hostOffset" -le -1 ) && -d "$pkg/bin" ]]; then
                  addToSearchPath _PATH "$pkg/bin"
              fi
          
          
    if [[ "$hostOffset" -eq 0 && -d "$pkg/bin" ]]; then
                  addToSearchPath _HOST_PATH "$pkg/bin"
              fi
          
          
    if [[ -f "$pkg/nix-support/setup-hook" ]]; then
                  source "$pkg/nix-support/setup-hook"
              fi
          }

Which is used here from _activatePkgs:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L510


      
              local -i hostOffset targetOffset
              local pkg
          
          
    for hostOffset in "${allPlatOffsets[@]}"; do
                  local pkgsVar="${pkgAccumVarVars[$hostOffset + 1]}"
                  for targetOffset in "${allPlatOffsets[@]}"; do
                      (( "$hostOffset" <= "$targetOffset" )) || continue
                      local pkgsRef="${pkgsVar}[$targetOffset - $hostOffset]"
                      local pkgsSlice="${!pkgsRef}[@]"
                      for pkg in ${!pkgsSlice+"${!pkgsSlice}"}; do
                          activatePackage "$pkg" "$hostOffset" "$targetOffset"
                      done
                  done
              done
          }
          
          
# Run the package setup hooks and build _PATH
          _activatePkgs
          
          
# Set the relevant environment variables to point to the build inputs
          # found above.

And packages seem to be coming from pkgAccumVarVars which is defined here:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L287


      
          
          
# shellcheck disable=SC2034
          declare -a pkgsBuildBuild pkgsBuildHost pkgsBuildTarget
          declare -a pkgsHostHost pkgsHostTarget
          declare -a pkgsTargetTarget
          
          
declare -a pkgBuildAccumVars=(pkgsBuildBuild pkgsBuildHost pkgsBuildTarget)
          declare -a pkgHostAccumVars=(pkgsHostHost pkgsHostTarget)
          declare -a pkgTargetAccumVars=(pkgsTargetTarget)
          
          
declare -a pkgAccumVarVars=(pkgBuildAccumVars pkgHostAccumVars pkgTargetAccumVars)
          
          

          
# Hooks
          
          
declare -a envBuildBuildHooks envBuildHostHooks envBuildTargetHooks
          declare -a envHostHostHooks envHostTargetHooks
          declare -a envTargetTargetHooks
          
          
declare -a pkgBuildHookVars=(envBuildBuildHook envBuildHostHook envBuildTargetHook)
          declare -a pkgHostHookVars=(envHostHostHook envHostTargetHook)

But then the trail gets very confusing due to some high level Bash magic, but I can see that the variables like buildInputs appear to be collected via findInputs:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L452-L454


      
          for pkg in $buildInputs $propagatedBuildInputs ; do
              findInputs "$pkg"  0  1
          done

Which in turn is another function full of Bash dark magic:

github.com

NixOS/nixpkgs/blob/6bae92a3eea8c56999d9c76417809a81e0fedfa6/pkgs/stdenv/generic/setup.sh#L341


      
              propagatedTargetDepFiles
          )
          
          
# Platform offsets: build = -1, host = 0, target = 1
          declare -a allPlatOffsets=(-1 0 1)
          
          

          
# Mutually-recursively find all build inputs. See the dependency section of the
          # stdenv chapter of the Nixpkgs manual for the specification this algorithm
          # implements.
          findInputs() {
              local -r pkg="$1"
              local -ri hostOffset="$2"
              local -ri targetOffset="$3"
          
          
    # Sanity check
              (( "$hostOffset" <= "$targetOffset" )) || exit -1
          
          
    local varVar="${pkgAccumVarVars[$hostOffset + 1]}"
              local varRef="$varVar[\$targetOffset - \$hostOffset]"
              local var="${!varRef}"

Quite the rabbit hole…

jonringer · October 14, 2020, 7:28pm

it’s a bit of a rabbit hole to support cross compilation, in which only packages which can be executed by the host should be available on the PATH.

Is there something you are trying to solve by ordering the PATH? Generally in nixpkgs, you just need one occurrence of a package.

jakubgs · October 14, 2020, 7:50pm

Not really, I’m just trying to understand how things work under the hood. Thanks for explaining.