Nix-shell buildInputs (ordering?) issue

Help
1

Hi all,

I am using the direnv/nix-shell combination for a multitude of projects with a lot of success.

I have this project that behaves weirdly though - I have a dependency on azure-cli as well as some Python 3 packages, so I’ve added both as buildInputs:

{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell {
  buildInputs = with pkgs; [
    azure-cli
    (python3.withPackages (p: with p; [ docopt ]))
  ];
}

The issue is - it looks like the order of the elements in buildInputs matters. With the shell.nix above the docopt Python module is not part of the resulting derivation:

❯ python3 -c 'import docopt'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'docopt'

If I switch the order (i.e., if I list azure-cli after python3.withPackages (...)) the module is there.

(In reality my shell.nix is a bit more complex - it has more dependencies and a shellHook - but I was trying to isolate the issue and that’s the shell.nix I was able to reproduce it with. The “best” part is that this workaround doesn’t work with my actual shell.nix.)

Any help will be greatly appreciated! Thanks!

❯ nixos-version
20.09.3857.b2a189a8618 (Nightingale)
❯ nix --version
nix (Nix) 2.3.10
❯ cat .envrc
use_nix
2

AFAIK ordering issues are usually a hint that there’s more than one copy of a package in your environment and that you’re using the “wrong” one.

I suspect azure-cli is also pulling in a python3 that it has a runtime dependency on, configured with its own set of dependent packages.

type -a python3 should help clarify what is in the environment.

3

Thanks @abathur!

Indeed this seems to be the case.

I’m wondering if this is a bug in the azure-cli package?

I’d (maybe naively) think that there should be no reason for azure-cli to need the Python interpreter propagated in the PATH.

4

I don’t think it’s a bug in the package, at least technically. Maybe @jonringer can answer that more definitively.

My rough understanding is that the nix+python ecosystem has some (historical?) quirks wrt to propagation and dependency leaking, and that this is more of a ~systemic issue than a package bug. There’s ongoing WIP that I think aims to address this issue in WIP: Python: wrap and patch using `requiredPythonModules` instead of `propagatedBuildInputs` by FRidh · Pull Request #102613 · NixOS/nixpkgs · GitHub (and maybe others).

2 Likes
5

I’m not able to reproduce this on master. azure-cli should be using buildPythonApplication, so it should be setting it’s own PYTHONPATH before being invoked.

Likely there’s some interaction between your existing python3 interpreter and the one in the shell (probably different versions). using the --pure flag should determine if it’s from the outside scope

$ nix-shell shell.nix --pure --run 'python3 -c "import docopt"'
3 Likes
6

@jonringer, thanks, there’s no Python interpreter outside my nix-shell’s though.

As you say - azure-cli seems wrapped and indeed the wrapper is setting the PYTHONPATH, but the interpreter seems to also leak into the PATH:

❯ type -a python3
bash: type: python3: not found
❯ cat shell.nix
{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell { buildInputs = with pkgs; [ azure-cli ]; }
❯ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz nix-shell shell.nix --pure --run 'type -a python3'
python3 is /nix/store/q6gfck5czr67090pwm53xrdyhpg6bx67-python3-3.8.9/bin/python3

With 20.09 nixpkgs azure-cli runs with Python 3.7 and python3 is 3.8. On master both use 3.8, which I think is the reason why the issue is not reproducible:

❯ cat shell.nix
{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell {
  buildInputs = with pkgs; [
    azure-cli
    (python3.withPackages (p: with p; [ docopt ]))
  ];
}
❯ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/20.09.tar.gz nix-shell shell.nix --pure --run 'type -a python3'
python3 is /nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/bin/python3
python3 is /nix/store/m2l3wxz9amzw4khgsrszcpzrj2cid33h-python3-3.8.5-env/bin/python3

I guess the question is (for me at least) - why is azure-cli using a different version from python3.

7

Around branchoff time, aiohttp didn’t support python3.8. So azure-cli was pinned to 3.7. This is no longer the case, and on master it using python3.8

$ nix-build -A azure-cli
/nix/store/fb74pkzh96yi5vx5al8gdg9d49ymrwwa-python3.8-azure-cli-2.20.0

It just wasn’t backported

Yea, I looked a bit more into this, seems to be related to aws-sam-cli, docker-compose: add to python3Packages by kini · Pull Request #121716 · NixOS/nixpkgs · GitHub. And the root cause seems to be the behavior propagatedBuildInputs with nix-shell which is the culprit. The PR that @abathur linked is one potential solution to this, it would just be a very large paradigm shift in nix+python packaging.

Edit: English is hard, even for native speakers

1 Like
8

I did find two alternatives:

  • set the PYTHONPATH explicitly as part of the shellhook
  • Use something besides nix-shell like nix shell (… I swear I’m not trolling)

ShellHook route:

# shell.nix
{ pkgs ? import <nixpkgs> { } }:

with pkgs;
let
  myPythonEnv = python3.withPackages (p: with p; [ docopt ]);
in pkgs.mkShell {
  nativeBuildInputs = [
    azure-cli
    myPythonEnv
  ];
  
  shellHook = ''
    export PYTHONPATH=${myPythonEnv}/${myPythonEnv.sitePackages}
  '';
}

nix shell route:

# env.nix
{ pkgs ? import <nixpkgs> { } }:

with pkgs; [
  azure-cli
  (python3.withPackages (p: with p; [ docopt ]))
]

Then create the shell with

nix shell -f env.nix

Unfortunately this probably doesn’t allow for any shellHooks

1 Like
9

For the record, I ended up wrapping az like this:

azure-cli' = writeShellScriptBin "az" "exec ${azure-cli}/bin/az \"$@\"";