To be clear, in most of the above I was mainly referring to container instances executing ansible playbooks (manual via podman vs. docker-executor by a gitlab-runner (where I have limited control on the container execution); these now behave the same for this context. But indeed I can reproduce it in the devShell.
The ansible error shows the full store path of the “wrong” interpreter, but it leaves out crucial information about how it actually found it. The peculiar thing is, it finds the correct py-env to get the pexpect module, but then tries to execute it with the wrong interpreter (even though it suggests it’s just trying /bin/python3.10 etc., which is there in the container and the “correct” one).
TASK [Testing remote connection using expect] ****************************************************************************************************
task path: /home/jeroen/devel/g1tlab/r/deploy/rctestenv.rcregistration/.submodules/rcregistration/_test-interpreter.ansible.yml:20
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/tmp `"&& mkdir "` echo /var/tmp/ansible-tmp-1687935594.4501653-394-12230152464923
5 `" && echo ansible-tmp-1687935594.4501653-394-122301524649235="` echo /var/tmp/ansible-tmp-1687935594.4501653-394-122301524649235 `" ) && sleep
0'
<pisatellite> Attempting python interpreter discovery
<localhost> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'python3.11'"'"'; command -v '"'"'python3.10'"'"'; command -v '"'"'p
ython3.9'"'"'; command -v '"'"'python3.8'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; com
mand -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'/usr/bin/py
thon'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<localhost> EXEC /bin/sh -c '/nix/store/lwzzgbnj41d657lpxczk6l5f7d5zcnj1-python3-3.10.11/bin/python3.10 && sleep 0'
<pisatellite> Python interpreter discovery fallback (unable to get Linux distribution/version info)
Using module file /nix/store/9ab2488gcawn919c5rpqd7hgp0dy153r-python3-3.10.11-env/lib/python3.10/site-packages/ansible/modules/expect.py
<localhost> PUT /home/jeroen/devel/g1tlab/r/deploy/rctestenv.rcregistration/.submodules/ansible-defs/tmp/ansible-local-371oarlchpw/tmpjh1urw0j TO
/nix/store/12gimb0hnqgisvzwk1nf30rlgz49l2yd-fake-nss/var/tmp/ansible-tmp-1687935594.4501653-394-122301524649235/AnsiballZ_expect.py
<localhost> EXEC /bin/sh -c 'chmod u+x /var/tmp/ansible-tmp-1687935594.4501653-394-122301524649235/ /var/tmp/ansible-tmp-1687935594.4501653-394-12
2301524649235/AnsiballZ_expect.py && sleep 0'
<localhost> EXEC /bin/sh -c '/nix/store/lwzzgbnj41d657lpxczk6l5f7d5zcnj1-python3-3.10.11/bin/python3.10 /var/tmp/ansible-tmp-1687935594.4501653-39
4-122301524649235/AnsiballZ_expect.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /var/tmp/ansible-tmp-1687935594.4501653-394-122301524649235/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "/var/tmp/ansible_ansible.builtin.expect_payload_ka5qy6cq/ansible_ansible.builtin.expect_payload.zip/ansible/modules/expect.py", line 117,
in <module>
ModuleNotFoundError: No module named 'pexpect'
[WARNING]: Platform linux on host pisatellite is using the discovered Python interpreter at
/nix/store/lwzzgbnj41d657lpxczk6l5f7d5zcnj1-python3-3.10.11/bin/python3.10, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.15/reference_appendices/interpreter_discovery.html for more information.
results from doing (in the container):
ansible-playbook -vvv _test-interpreter.ansible.yml
(the script executes a simple ansible.builtin.expect task)
BUT: if I do in the container:
sh-5.2# readlink $(command -v python3.10)
/nix/store/9ab2488gcawn919c5rpqd7hgp0dy153r-python3-3.10.11-env/bin/python3.10
which is the correct python environment.
One thing I failed to mention before: the test script is like this, notably it forces most commands being executed on the ansible controller while executing a barebones ssh session with connection data from hosts.yaml (which is what I need for my use case since some targets are semi-embedded and some don’t have python or pub-key auth.)
- hosts: pisatellite
name: Test whether the correct python interpreter is used by using expect
tasks:
- name: Testing remote connection using expect
ansible.builtin.expect:
timeout: 60
command: |
sh -c '\
echo "We ($(hostname -s)) tell you to report your hostname..." | \
ssh {{ ssh_custom_args }} -p{{ ansible_ssh_port }} {{ ansible_user }}@{{ ansible_host }} "\
cat -;
echo Hello from $(hostname -s);
"'
responses:
(.*)password: "{{ ansible_ssh_pass }}"
delegate_to: localhost
connection: local
BTW, circling back to the binary wrapper topic: as I now understand (and checked) all binaries in my /nix/store/9ab2488gcawn919c5rpqd7hgp0dy153r-python3-3.10.11-env/bin/ path are binary wrappers, each weighing 15K, which seems awfully big considering they just need to set a few env vars?
.