How can I build Ansible with custom Python?

jakubgs · April 8, 2021, 8:17pm

I have an issue with Ansible due to how it runs a pure Nix Python which lacks the pyyaml module that normally would be installed on any other system, and for that reason a local_action calling a script which does import yaml fails with ModuleNotFoundError.

I tried this:

{
  # HACK: Fix for Ansible not having PyYaml available.
  python38Yaml = pkgs.python38.withPackages (ps: [ ps.pyyaml ]);
  ansible = python38Yaml.pkgs.toPythonApplication pkgs.python38Packages.ansible;
}

But still no cigar. The Python used is still not the one with pyyaml:

 > whereis ansible
ansible: /nix/store/wfz1w32g8vzn6if8nkq25s5xz4cs43am-user-environment/bin/ansible

 > readlink /nix/store/wfz1w32g8vzn6if8nkq25s5xz4cs43am-user-environment/bin/ansible
/nix/store/kcas64xvnrp4xw7cxk7jirha71zgw7p0-python3.8-ansible-2.9.12/bin/ansible

 > grep python3-3.8.8 /nix/store/kcas64xvnrp4xw7cxk7jirha71zgw7p0-python3.8-ansible-2.9.12/bin/ansible
export PATH='/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/bin:/nix/store/kcas64xvnrp4xw7cxk7jirha71zgw7p0-python3.8-ansible-2.9.12/bin:/nix/store/sbiym6y0nmyabnh6mz4xzy26l0fhyqy7-python3.8-setuptools-47.3.1/bin:/nix/store/s2v289fh1w71wkhvbhk2rsflyfaadm1d-python3.8-netaddr-0.8.0/bin:/nix/store/m4a6myh64yabq1rwd1crqslfvz0wv3rr-python3.8-jmespath-0.10.0/bin:/nix/store/anjfg0j7d8g1i9zkmsbwvfs7znc1y4p2-python3.8-chardet-3.0.4/bin'${PATH:+':'}$PATH

I can see it using /nix/store/...-python3-3.8.8/bin, but what I expected was my /nix/store/...-python3-3.8.8-env/bin which includes pyyaml.

jakubgs · April 9, 2021, 2:23pm

I’m not sure, but I’m starting to think this is not doable, because of this:
https://github.com/NixOS/nixpkgs/blob/583389b67b0405d708e8f11c1f3f3a3da5317fcc/pkgs/development/interpreters/python/wrapper.nix#L22
Where python.executable is just a string that is the appended to a bin path.

jonringer · April 9, 2021, 4:39pm

ansible is also just a python package, so could can do something like

[09:37:00] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ nix-build -E 'with import ./. {}; python38.withPackages (ps: with ps; [ pyyaml ansible ])'
/nix/store/yfayr1qh026rs2qq17wnq5if4j677pzw-python3-3.8.8-env
[09:37:04] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ ./result/bin/python -c 'import yaml'
[09:37:07] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ ./result/bin/python -c 'import ansible'

ansible also gets exported in bin.

$ ./result/bin/ansible --version
ansible 2.9.12
  config file = None
  configured module search path = ['/home/jon/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /nix/store/yfayr1qh026rs2qq17wnq5if4j677pzw-python3-3.8.8-env/lib/python3.8/site-packages/ansible
  executable location = /nix/store/fynpwnvgmhykrjgvfkfwif9qn63gmh7i-python3.8-ansible-2.9.12/bin/ansible
  python version = 3.8.8 (default, Feb 19 2021, 11:04:50) [GCC 10.2.0]

This is more done for alternative python interpreters (e.g. Pypy)

EDIT:
One issue with just using the python package is that you no longer have access to the other ansible versions. So you will have a less “stable” environment

jakubgs · April 9, 2021, 4:50pm

Thanks man. That’s neat, but it doesn’t work with what I’m trying to do:

`test.yml`

- gather_facts: false
  hosts: localhost              
  tasks:          
    - local_action: command python3 -c 'import yaml'

 > /nix/store/yfayr1qh026rs2qq17wnq5if4j677pzw-python3-3.8.8-env/bin/python -c 'import yaml'
 > /nix/store/yfayr1qh026rs2qq17wnq5if4j677pzw-python3-3.8.8-env/bin/ansible-playbook test.yml

PLAY [localhost] **********************************************************************************************************************************

TASK [command] ************************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": [
        "python3",
        "-c",
        "import yaml"
    ],
    "delta": "0:00:00.058295",
    "end": "2021-04-09 18:49:52.893597",
    "rc": 1,
    "start": "2021-04-09 18:49:52.835302"
}

STDERR:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'yaml'


MSG:

non-zero return code

PLAY RECAP ****************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

jakubgs · April 9, 2021, 4:51pm

My understanding is that when i try to use local_action(or any other way of running things locally) Ansible uses only it’s own interpreter without any additional packages to run the script.

jakubgs · April 9, 2021, 4:52pm

I might add that this way of running local python scripts totally works on any other OS, like Ubuntu, and python picks up yaml module just fine. It’s a weird edge case.

jonringer · April 9, 2021, 4:56pm

Solution: don’t use ansible

jk

Yea, the only explanation is that ansible is not using YOUR python interpreter then, as yaml should be in site-packages.

You can also use the env var PYTHONPATH to add it

export PYTHONPATH="${python3}/${python3.sitePackages}"

jakubgs · April 9, 2021, 5:01pm

I wish I could. Too much infra already using it, and I’d love to port everything to NixOS/NixOps, but there’s just too much work with new stuff to spend time porting everything else =].

That’s cool, but I’m not working on a derivation. Maybe I can explain what’s happening:

I have a python script that I want to run a the beginning of a playbook to check versions of Ansible roles against the requirements.txt file because Ansible or Ansible galaxy doesn’t actually check that(stupid, I know…). And I have a simple script that does that:

github.com

status-im/infra-nimbus/blob/03457d9cfe559843206b443953925fbebdc9e5a6/ansible/galaxycheck.py

#help("modules")
import sys
print(sys.executable)
import yaml
import subprocess
from os import path, environ


# Where Ansible looks for installed roles.
ANSIBLE_ROLES_PATH = path.join(environ['HOME'], '.ansible/roles')

class Role:
    def __init__(self, name, version):
        self.name = name
        self.version = version

    @property
    def path(self):
        return path.join(ANSIBLE_ROLES_PATH, self.name)

This file has been truncated. show original

But to run it locally I need yaml modules to parse requirements.txt(which is actually yaml despite the .txt extension…). Since I’m on NixOS and install Ansible via NixOS it uses the interpreter directly without any packages and is missing yaml, which makes the script fail.

It’s not a huge issue, and I guess I’ll just rewrite the script to parse requirements.txt “by hand”, but this is annoying :D.

Thanks anyway for your help. Much appreciated.

jonringer · April 9, 2021, 5:05pm

You don’t have to, just write it in a shell.nix or as part of your devShell if you’re using flakes.

When you activate the dev shell, you can add my line to the shellHook and you should be able to use it.

For most projects, I have a shell.nix/flake.nix + direnv active, so just changing to the project’s directory gives me a “working” environment

jakubgs · April 9, 2021, 5:08pm

Okay, I tried it with;

{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell {
  name = "infra-nimbus-shell";
  buildInputs = [ pkgs.ansible ];
  shellHook = ''
    export PYTHONPATH="${pkgs.python3}/${pkgs.python3.sitePackages}"
  '';
}

But it looks like Ansible just doesn’t inherit the $PYTHONPATH because it just fails with ModuleNotFoundError. And I checked that PYTHONPATH is set when I call ansible-playbook.

I think this is hopeless and the simplest way is to just not use yaml module.

Thanks again!

jonringer · April 9, 2021, 5:11pm

Someone who is more familiar with ansible may be able to help you.

I’ve replaced anything that could be solved with ansible with nix, so I don’t have much experience with it

jakubgs · April 9, 2021, 5:12pm

My next client is definitely getting an all-NixOS solution :D. I’m so done with Ansible.

jakubgs · April 9, 2021, 5:14pm

And indeed, changing my test playbook to this:

- gather_facts: false
  hosts: localhost
  tasks:
    - command: python3 -c 'import yaml'
      environment:
        PYTHONPATH: '/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python3.8/site-packages'

Does work. So the issue is that it uses the interpreter alone and does not inherit or provide a PYTHONPATH.

jakubgs · April 9, 2021, 5:26pm

I used this:

- gather_facts: false
  hosts: localhost
  tasks:
    - local_action: shell python3 -c 'import sys; print(":".join(sys.path))'
      register: pythonpath
    - debug: msg='{{ pythonpath.stdout | replace(":", "\n") }}'

And this is the output:

/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python38.zip
/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python3.8
/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python3.8/lib-dynload
/nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python3.8/site-packages

Which appears to not even be the python that includes Ansible:

 > ls -l /nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/bin 
total 28
lrwxrwxrwx 1 root root     8 Jan  1  1970 2to3 -> 2to3-3.8
-r-xr-xr-x 1 root root   148 Jan  1  1970 2to3-3.8
lrwxrwxrwx 1 root root     7 Jan  1  1970 idle -> idle3.8
lrwxrwxrwx 1 root root     7 Jan  1  1970 idle3 -> idle3.8
-r-xr-xr-x 1 root root   146 Jan  1  1970 idle3.8
lrwxrwxrwx 1 root root     8 Jan  1  1970 pydoc -> pydoc3.8
lrwxrwxrwx 1 root root     8 Jan  1  1970 pydoc3 -> pydoc3.8
-r-xr-xr-x 1 root root   131 Jan  1  1970 pydoc3.8
lrwxrwxrwx 1 root root     9 Jan  1  1970 python -> python3.8
lrwxrwxrwx 1 root root     9 Jan  1  1970 python3 -> python3.8
-r-xr-xr-x 1 root root 16208 Jan  1  1970 python3.8
-r-xr-xr-x 1 root root  3291 Jan  1  1970 python3.8-config
lrwxrwxrwx 1 root root    16 Jan  1  1970 python3-config -> python3.8-config
lrwxrwxrwx 1 root root    16 Jan  1  1970 python-config -> python3.8-config

It’s just bare Python without any site-packages:

 > ls -l /nix/store/qy5z9gcld7dljm4i5hj3z8a9l6p37y81-python3-3.8.8/lib/python3.8/site-packages
total 4                                                                                       
dr-xr-xr-x 2 root root    5 Jan  1  1970 __pycache__
-r--r--r-- 1 root root  119 Jan  1  1970 README.txt
-r--r--r-- 1 root root 1659 Jan  1  1970 sitecustomize.py

jakubgs · April 9, 2021, 5:34pm

Looks like this is a known issue to Homebrew users:

Ansible modifies PYTHONPATH which causes Python commands to fail · Issue #9956 · ansible/ansible · GitHub
Ansible formula modifies PYTHONPATH and causes Python scripts within playbooks to fail · Issue #35647 · Homebrew/legacy-homebrew · GitHub

jakubgs · April 9, 2021, 5:46pm

I found a way to pass PYTHONPATH to Ansible based on this StackOverflow answer:

- gather_facts: false
  hosts: localhost              
  tasks:          
    - command: python3 -c 'import yaml'
      environment:
        PYTHONPATH: '{{ ansible_env.PYTHONPATH }}'

By using ansible_env I can access the environment variables of the host from which Ansible is being run, and hence set it for the task. Not pretty, but it does work.

arianvp · April 9, 2021, 6:37pm

This is the solution i came up with at work:

github.com/wireapp/wire-server-deploy

Fix ansible module dependencies

wireapp:develop ← wireapp:fix-python-path-for-good

opened 09:09AM - 01 Apr 21 UTC

arianvp

+18 -16

What we were doing before was way too complicated. Ansible itself doesn't have …any dependency on boto to function. The _target host_ needs to have boto installed. In our case the target host is 'localhost'. Ansible will by default pick /usr/bin/python3 as the python interpreter ogit n target hosts but of course we can't rely on that having boto installed. Instead, we configure ansible_python_interpreter to point to the environment we built with nix; which contains a python that has boto installed. This way boto can be found and the playbooks should succeed!

jakubgs · April 9, 2021, 6:52pm

Oh you’re setting ansible_python_interpreter. Very cool, thanks!

Johannes · August 11, 2023, 6:07am

I don’t think this is the nixos-way to do it. I stumble accross the same Problem.
IMHO - The problem lays in the provided ansible-base package. Which uses python as an reference.

/

default.nix

I’m to new to nixos to realy understand how ansible is build here - but IMHO there should be a way to enhance ansible-core by a setup-hook by the packages you need to have ansible to run.
Python - NixOS Wiki
Or am i wrong?
But i’m unsure on how to archive this.
Setting the python interpreter in the ansible-files seems really wrong?!