Creating a nix-shell for python with ansible / molecule installed

Help
1

Hey everyone!

I am trying to create a nix-shell for development purposes with ansible, and also for the learning experience I guess :slight_smile: I am planning to do testing of ansible roles and playbooks using molecule, which comes as a python package.

Molecule is, afaik, not part of nixpkgs at the moment, so after a bit of reasearch I decided to get it via buildPythonPackage. The problem I have run into is that in order to build molecule, I also need a very specific version of PyYAML as a dependency, namely one that is below version 6.0 but equal to or above version 5.1. This version is also not part of nixpkgs.

So one of the propagatedBuildInputs has to be overwritten somehow by fetching the proper version of PyYAML from pypi. And potentially other packages as well. But I have no idea how to do that properly.

Any help is appreciated!

Here is the current code of my little nix-shell experiment so far:

{ pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz) {} }:

let
  molecule = pkgs.python39Packages.buildPythonPackage rec {
    pname = "molecule";
    version = "3.5.2";

    src = pkgs.python39Packages.fetchPypi {
        inherit pname version;
        sha256 = "yCrwmeXAmY1+sWo79l7VpO3Zfjgk+5OM4BvwZKRs4Mo=";
    };

    propagatedBuildInputs = with pkgs.python39Packages; [ 
        ansible 
        ansible-core 
        ansible-lint
        gevent
        geoip2
        setuptools
        setuptools-scm
        libselinux
        pyyaml
        click-help-colors
        pkgs.selinux-python
    ];

    doCheck = false;
  };

  ansiblePython = pkgs.python39.buildEnv.override {
    extraLibs = [ molecule ];
  };
in

pkgs.mkShell {
  buildInputs = [ ansiblePython ];
}
2

I did manage to get one step ahead, just to encounter another error I do not comprehend.

My code now looks like this, which appears to override pyyaml 6.0 with 5.4.1:

let
  myPythonPackages = pkgs.python39Packages.override {
    overrides = self: super: {
      pyyaml = super.pyyaml.overrideAttrs (oldAttrs: {
        src = pkgs.python39Packages.fetchPypi {
          pname = "PyYAML";
          version = "5.4.1";
          sha256 = "YHd0y7oocyv6gCtUuqdIQhX1MJkQVbtWLvvtWy8gpF4=";
          # sha256 = "aPtRnBQwb+yXIKKltFvJ8MjRuccq30XDe67fzZScNaI=";
        };
      });
    };
  };

  molecule = myPythonPackages.buildPythonPackage rec {
    pname = "molecule";
    version = "3.5.2";

    src = myPythonPackages.fetchPypi {
        inherit pname version;
        sha256 = "yCrwmeXAmY1+sWo79l7VpO3Zfjgk+5OM4BvwZKRs4Mo=";
    };

    propagatedBuildInputs = with myPythonPackages; [
        # pip
        # wheel
        # ansible 
        # ansible-core 
        # ansible-lint
        # gevent
        # geoip2
        setuptools
        setuptools-scm
        # libselinux
        pyyaml
        click-help-colors
        pkgs.selinux-python
    ];

    doCheck = false;
  };

  ansiblePython = pkgs.python39.buildEnv.override {
    extraLibs = [ molecule ];
  };
in

pkgs.mkShell {

  buildInputs = [ 
    ansiblePython
  ];
}

Problem is, I now get this error:

Successfully installed PyYAML-5.4.1
/build/PyYAML-5.4.1
Finished executing pipInstallPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/b359a4ddhswimd3qrpxn7wzr73d57mil-python3.9-PyYAML-6.0
shrinking /nix/store/b359a4ddhswimd3qrpxn7wzr73d57mil-python3.9-PyYAML-6.0/lib/python3.9/site-packages/yaml/_yaml.cpython-39-x86_64-linux-gnu.so
strip is /nix/store/npm4g1zsj5yzygf6bq46pbi9fqhxisha-gcc-wrapper-10.3.0/bin/strip
stripping (with command strip and flags -S) in /nix/store/b359a4ddhswimd3qrpxn7wzr73d57mil-python3.9-PyYAML-6.0/lib 
patching script interpreter paths in /nix/store/b359a4ddhswimd3qrpxn7wzr73d57mil-python3.9-PyYAML-6.0
checking for references to /build/ in /nix/store/b359a4ddhswimd3qrpxn7wzr73d57mil-python3.9-PyYAML-6.0...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
running install tests
Traceback (most recent call last):
  File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/build/PyYAML-5.4.1/tests/lib/test_all.py", line 2, in <module>
    import sys, yaml, test_appliance
  File "/build/PyYAML-5.4.1/tests/lib/test_appliance.py", line 70
    except Exception, exc:
                    ^
SyntaxError: invalid syntax
error: builder for '/nix/store/0j60mxqw2hsb8j10xcigi5l6dacbh75k-python3.9-PyYAML-6.0.drv' failed with exit code 1;
       last 10 log lines:
       >   File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
       >     return _run_code(code, main_globals, None,
       >   File "/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/lib/python3.9/runpy.py", line 87, in _run_code
       >     exec(code, run_globals)
       >   File "/build/PyYAML-5.4.1/tests/lib/test_all.py", line 2, in <module>
       >     import sys, yaml, test_appliance
       >   File "/build/PyYAML-5.4.1/tests/lib/test_appliance.py", line 70
       >     except Exception, exc:
       >                     ^
       > SyntaxError: invalid syntax
       For full logs, run 'nix log /nix/store/0j60mxqw2hsb8j10xcigi5l6dacbh75k-python3.9-PyYAML-6.0.drv'.
error: 1 dependencies of derivation '/nix/store/xqha5jhsmsh17lwm8f3rc8q7myw4kjb4-python3-3.9.9-env.drv' failed to build

Anyone any ideas what is going on here?

It would be a shame if we couldn’t get molecule for Ansible development up and running on NixOS in a nix-shell…

btw. two other things: Getting a nix-shell just with pyyaml 5.4.1 works without flaws. And looking at the full log didn’t really tell me much else, but if someone is interested I will of course paste it here as well.

3

I did manage to get molecule up and running using an environment built with mach-nix. Sad thing is that I have no idea how mach-nix is doing its thing in this case. I would still appreciate any hint what I was missing in my own little attempt.

4

If anyone else sees this. I’ve used a mach-nix for a while but it stopped working so I use a FHS environment now. I’ve published it in a github repo, but it’s basically:

$ cat shell.nix 
{ pkgs ? import <nixpkgs> {} }:
(pkgs.buildFHSUserEnv {
  name = "molecule";
  targetPkgs = pkgs: (with pkgs; [
    python39
    python39Packages.pip
    python39Packages.virtualenv
  ]);
  runScript = "bash";
}).env

which is used with nix-shell, followed by . ./activate.sh:

$ cat activate.sh 
#!/usr/bin/env bash

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
1 Like
5

Since I’ll be doing more Ansible shortly, I will take a look at your setup, thanks @Melkor333