Python module not found when combining `pytest` and `subprocess`

jacg · July 3, 2020, 9:54am

TLDR

The chain pytest → subprocess.check_output → python program fails to import a non-standard module.

In cases which succeed, numpy is found in '/nix/store/<SHA>-python3-3.8.3-env/lib/python3.8/site-packages', which is missing from sys.path in the pytest-subprocess combination.

I have tried to reduce it to a minimal case, described in detail below.

Minimal components

Environment definition

The environment is defined in hmm.nix:

{ py-version ? "37" }:
let
  commit-id = "f1a79c86358c5464c64b4fad00fca07a10e62a74";
  nixpkgs-url = "https://github.com/nixos/nixpkgs/archive/${commit-id}.tar.gz";
  pkgs = import (builtins.fetchTarball { url = nixpkgs-url; }) {};
  python = builtins.getAttr ("python" + py-version) pkgs;
  pypkgs = python.pkgs;
  command = pkgs.writeShellScriptBin;

  pythonWithPackages = python.withPackages (ps: [
      ps.numpy
      ps.pytest
    ]);

in

pkgs.mkShell {
  buildInputs = [ pythonWithPackages ];
}

A simple program

The program simply imports a non-standard module (numpy) and reports its location. It is stored in hmm.py:

#!/usr/bin/env python

import pprint, sys
pprint.pprint(sys.path)
import numpy
print(numpy)

The failing case: `pytest` calling the program via `subprocess`

The hmm.py program is called via subprocess from pytest. This is done in test_call().
For comparison, test_import_numpy is a copy ‘n’ paste of the source of hmm.py, bypassing subprocess.

This file is called callhmm_test.py:

from subprocess import STDOUT, check_output, CalledProcessError
from pprint import pprint

def test_call():
    command = './hmm.py'
    try:
        print(check_output(command, shell = True, stderr=STDOUT).decode('UTF-8'))
    except CalledProcessError as e:
        # Ensure that stdout and stderr are visible when test fails
        print(e.stdout.decode())
        raise

def test_import_numpy():
    import pprint, sys
    pprint.pprint(sys.path)
    import numpy

Observations

Environment activation

The environment is entered thus:

nix-shell hmm.nix --argstr py-version 38

Two variations succeed with identical results

Running hmm.py either directly, or via the callhmm_test.test_call function

./hmm.py
python -c 'from callhmm_test import test_call; test_call()'

succeeds with the identical results:

['/tmp/hmm',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python38.zip',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/lib-dynload',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/site-packages',
 '/nix/store/jpi895v1pnnz7xi8hjmb1q9gxk9agh3w-python3-3.8.3-env/lib/python3.8/site-packages']
<module 'numpy' from '/nix/store/jpi895v1pnnz7xi8hjmb1q9gxk9agh3w-python3-3.8.3-env/lib/python3.8/site-packages/numpy/__init__.py'>

`pytest` without `subprocess` also succeeds

The contents of hmm.py copy-pasted into a pytest test, also succeed. More stuff appears in sys.path than in the two previous cases, but exactly the same version of numpy is found:

$ pytest -k numpy -s
=========================================== test session starts ============================================
platform linux -- Python 3.8.3, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/hmm
collected 2 items / 1 deselected / 1 selected                                                              

callhmm_test.py 
['/tmp/hmm',
 '/nix/store/17bk1sg0bs3csir4vilykmiv9y2ff67m-python3.8-pytest-5.3.5/bin',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python38.zip',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/lib-dynload',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/site-packages',
 '/nix/store/jpi895v1pnnz7xi8hjmb1q9gxk9agh3w-python3-3.8.3-env/lib/python3.8/site-packages',
 '/nix/store/17bk1sg0bs3csir4vilykmiv9y2ff67m-python3.8-pytest-5.3.5/lib/python3.8/site-packages',
 '/nix/store/3hzmvwgwc3z6zl07g66hccr5y18993dz-python3.8-attrs-19.3.0/lib/python3.8/site-packages',
 '/nix/store/pqrr5yypqzjcxbiv8paw3blq1fdkdcnw-python3.8-py-1.8.1/lib/python3.8/site-packages',
 '/nix/store/4z66ha9p1njr78jigrv5n48clq4b0ma0-python3.8-setuptools-44.0.0/lib/python3.8/site-packages',
 '/nix/store/zsn0wxnxkjqs95v5r3q84yskfp7nbgg1-python3.8-six-1.14.0/lib/python3.8/site-packages',
 '/nix/store/77k93xyy9mka94g2s65wrl11ph4fxdv9-python3.8-pluggy-0.13.1/lib/python3.8/site-packages',
 '/nix/store/8zbv2cwgbbr0apn1jfqkikjhs03fdwkc-python3.8-importlib-metadata-1.3.0/lib/python3.8/site-packages',
 '/nix/store/am1rhq8017sqpkwnkalm5101vx309cvf-python3.8-zipp-0.6.0/lib/python3.8/site-packages',
 '/nix/store/lwvq1zkkzmnc526djfzgh6rabblrqbrc-python3.8-more-itertools-8.0.2/lib/python3.8/site-packages',
 '/nix/store/dhywrx95z1fjnkq9f6mia0dbp1bk380a-python3.8-atomicwrites-1.3.0/lib/python3.8/site-packages',
 '/nix/store/pbnr69p0xqiv8mkqgimrv1b7fh0rihxw-python3.8-wcwidth-0.1.7/lib/python3.8/site-packages',
 '/nix/store/9z50rdyklzh7q2zxk24kllmnz73qg5z4-python3.8-packaging-20.1/lib/python3.8/site-packages',
 '/nix/store/6nqglqpsnhaqzmj1mk59xixi18hw1pmd-python3.8-pyparsing-2.4.6/lib/python3.8/site-packages']
<module 'numpy' from '/nix/store/jpi895v1pnnz7xi8hjmb1q9gxk9agh3w-python3-3.8.3-env/lib/python3.8/site-packages/numpy/__init__.py'>
.

===================================== 1 passed, 1 deselected in 0.11s ======================================

`pytest` and `subprocess` together result in failure to find `numpy`

If pytest is used to execute callhmm_test.test_call, then numpy is not found:

$ pytest --tb=short -k call
=========================================== test session starts ============================================
platform linux -- Python 3.8.3, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/hmm
collected 2 items                                                                                          

callhmm_test.py F.                                                                                   [100%]

================================================= FAILURES =================================================
________________________________________________ test_call _________________________________________________
callhmm_test.py:7: in test_call
    print(check_output(command, shell = True, stderr=STDOUT).decode('UTF-8'))
/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/subprocess.py:411: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/subprocess.py:512: in run
    raise CalledProcessError(retcode, process.args,
E   subprocess.CalledProcessError: Command './hmm.py' returned non-zero exit status 1.
------------------------------------------- Captured stdout call -------------------------------------------
['/tmp/hmm',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python38.zip',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/lib-dynload',
 '/nix/store/9ik411n4c66cynrgcs2ym0ds7ckph8pp-python3-3.8.3/lib/python3.8/site-packages']
Traceback (most recent call last):
  File "./hmm.py", line 5, in <module>
    import numpy
ModuleNotFoundError: No module named 'numpy'

======================================= 1 failed, 1 passed in 0.23s ========================================

Crucially '/nix/store/jpi895v1pnnz7xi8hjmb1q9gxk9agh3w-python3-3.8.3-env/lib/python3.8/site-packages', where numpy was found in the other cases, is missing from sys.path.