Can't detect cryptography even though its installed

[Subito]

SUMMARY

Running acme_certificate, I get the following error:

fatal: [remote_host]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (cryptography) on remote_host's Python /usr/local/bin/python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

ISSUE TYPE

• Bug Report

COMPONENT NAME

acme_certificate

ANSIBLE VERSION

ansible 2.10.17
  config file = /Users/svbito/src/nkhosting/ansible/ansible.cfg
  configured module search path = ['/Users/svbito/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/svbito/.virtualenvs/ansible29/lib/python3.9/site-packages/ansible
  executable location = /Users/svbito/.virtualenvs/ansible29/bin/ansible
  python version = 3.9.15 (main, Oct 11 2022, 22:27:25) [Clang 14.0.0 (clang-1400.0.29.102)]

COLLECTION VERSION

# /Users/svbito/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.7.0

OS / ENVIRONMENT

Target OS: FreeBSD 13.1

STEPS TO REPRODUCE

- name: Let the challenge be validated and retrieve the cert and intermediate certificate
  community.crypto.acme_certificate:
    account_key_src: /usr/local/etc/ssl/letsencrypt-account.key
    account_email: [email protected]
    src: /usr/local/etc/ssl/{{domain}}.csr
    cert: /usr/local/etc/ssl/{{domain}}.crt
    fullchain: /usr/local/etc/ssl/{{domain}}_fullchain.crt
    chain: /usr/local/etc/ssl/{{domain}}_chain.crt
    challenge: dns-01
    acme_directory: https://acme-v02.api.letsencrypt.org/directory
    remaining_days: 30
    acme_version: 2
    data: "{{ challenge }}"
    select_crypto_backend: cryptography
    select_chain:
      - test_certificates: last
        issuer:
          CN: ISRG Root X1

EXPECTED RESULTS

The cert gets fetched

ACTUAL RESULTS

fatal: [remote_host]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (cryptography) on rb6.kommune.net's Python /usr/local/bin/python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

I ran ansible remote_host -m community.general.python_requirements_info -a 'dependencies=cryptography' and got the following:

remote_host | SUCCESS => {
    "changed": false,
    "mismatched": {},
    "not_found": [],
    "python": "/usr/local/bin/python",
    "python_system_path": [
        "/tmp/ansible_community.general.python_requirements_info_payload_8w6yi_79/ansible_community.general.python_requirements_info_payload.zip",
        "/usr/local/lib/python39.zip",
        "/usr/local/lib/python3.9",
        "/usr/local/lib/python3.9/lib-dynload",
        "/usr/local/lib/python3.9/site-packages"
    ],
    "python_version": "3.9.14 (main, Oct  7 2022, 01:19:45) \n[Clang 13.0.0 ([email protected]:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a",
    "valid": {
        "cryptography": {
            "desired": null,
            "installed": "38.0.1"
        }
    }
}

If I use the openssl backend, everything works as expected

[felixfontein]

Is cryptography installed on the remote host, for the Python and user that is used by ansible?

[webknjaz]

I've taken a prompt look at this.
It seems like python_requirements_info makes checks for "has cryptography at all" while acme_certificate uses "has modern cryptography". But that "modern cryptography" flag is being reset on any exceptions in the check code (like import or runtime errors). This is probably the source of such behavior: https://github.com/ansible-collections/community.crypto/blob/1f4840b/plugins/module_utils/acme/backend_cryptography.py#L67-L68.
Too bad that the actual exception isn't being recorded there so that the traceback could be included in the failure result. Here's what I like to do:

...

try:
    import lib_name
except ImportError as import_exc:
    LIB_NAME_IMPORT_ERROR = import_exc
    from traceback import format_exception
else:
    LIB_NAME_IMPORT_ERROR = None

...

def do_runtime_interaction(..., ansible_module, ...):
    ...
    if LIB_NAME_IMPORT_ERROR is not None:  # or isinstance(LIB_NAME_IMPORT_ERROR, BaseException)
        ansible_module.fail_json(
            msg=missing_required_lib('lib_name'),
            traceback=format_exception(LIB_NAME_IMPORT_ERROR),
        )
    ...

...

@felixfontein here's an idea for a better problem reporting idiom ^

[Subito]

Is cryptography installed on the remote host, for the Python and user that is used by ansible?

yes. its always run as root. But digging a bit further into this, it seems that the problem disappears again if I downgrade cryptography. Maybe the problem lies over there and not in the detection mechanism. Although its a bit confusing that python -c 'import cryptography' works in ansibles python interpreter while module fails with this error message.

[webknjaz]

Maybe the problem lies over there and not in the detection mechanism.

Sounds like it's both: the detection mechanism calls cryptography.hazmat.backends.default_backend() (it does not just run imports) which might fail with some sort of runtime error which is then caught by an overly broad except block that doesn't inspect what's actually happened and just sets a "nope, there's no usable cryptography" flag.
Try interactively repeating all the steps from that huge try-except block at https://github.com/ansible-collections/community.crypto/blob/1f4840b/plugins/module_utils/acme/backend_cryptography.py#L52-L66, using the problematic cryptography version to see if I'm right.

[felixfontein]

I created #519 to improve the error handling.

[Subito]

Sounds like it's both: the detection mechanism calls cryptography.hazmat.backends.default_backend() (it does not just run imports) which might fail with some sort of runtime error which is then caught by an overly broad except block that doesn't inspect what's actually happened and just sets a "nope, there's no usable cryptography" flag.
Try interactively repeating all the steps from that huge try-except block at https://github.com/ansible-collections/community.crypto/blob/1f4840b/plugins/module_utils/acme/backend_cryptography.py#L52-L66, using the problematic cryptography version to see if I'm right.

Seems like some of the imports do in fact break with the new cryptography version: ImportError: cannot import name 'x509' from 'cryptography.hazmat.bindings._rust' (unknown location). That might be a packaging issue on the OS side, I will investigate and report back. Thanks a lot for your help!

[felixfontein]

This sounds more like a packaging problem or a broken cryptography installation (the Rust bindinds might be missing). The modules work fine with 38.0.1, as some of the CI runs show (for example https://dev.azure.com/ansible/community.crypto/_build/results?buildId=57094&view=logs&j=5f3a012a-81d6-5fc5-a5af-5b322882411e&t=57903074-f940-517a-2092-4b13b1d0eaf0&l=582).

[webknjaz]

That might be a packaging issue on the OS side, I will investigate and report back.

cryptography migrated from C to Rust in v3.4. This means that if it's built from sdist (hence compiled from source), it does need a functional Rust compiler of a compatible version (https://github.com/pyca/cryptography/blob/0f2d830/setup.py#L63). I suppose it also requires OpenSSL bindings.
There are some notes on the build deps at https://cryptography.io/en/latest/installation/.
They also ship manylinux wheels that bundle all non-typical dynamic libs per PEP 600. These wheels are picked up by a sufficiently modern pip automatically. Unfortunately, there's no similar standard for BSD (manylinux targets glibc envs, plus there's also a separate PEP for musllinux). Which ultimately means that you can't use the easy way of installing cryptography.
Also note that during the installation, pip first builds a platform wheel for the current environment (your BSD + CPython version combo essentially). And when you install it again in a similar env under the same user (the cache dir is located under the user's home), it won't rebuild it but just unarchive that cached wheel. This means that if for some reason you end up (successfully) building a broken cryptography wheel, it'll get reused despite fixing your build env. Watch out for that.

That might be a packaging issue on the OS side

Only if you rely on OS packaging and not pip. Have you tried using a virtualenv with pip-managed packages in it?

the Rust bindinds might be missing

There's no such thing, it's only used at build time. But OpenSSL is indeed loaded through CFFI bindings.

The modules work fine with 38.0.1, as some of the CI runs show

Ansible Core CI and the related ansible-test infra has an integrated dumb-pypi index with a bunch of prebuilt platform-specific wheels for projects that have slow and/or complicated build process. cryptography is one of them. Here's a matrix of wheels that we build for the supported FreeBSD versions: https://github.com/ansible/spare-tire/blob/b560583/wheel_matrix.yml#L18-L45.

They are available from this index: https://spare-tire.testing.ansible.com/simple/cryptography/. @Subito you can pip install from there for testing. Also, you can download your matching wheel and inspect/compare its contents using the wheel unpack command.

[felixfontein]

the Rust bindinds might be missing

There's no such thing, it's only used at build time.

The built Rust libraries still need to be loaded ar runtime, as otherwise large parts of cryptography that have been written in Rust wouldn't work anymore.

[felixfontein]

community.crypto 2.7.1 is out which should improve error reporting in this case.

Closing this issue since the underlying problem is unrelated to this collection, and the new error reporting should point that out better :) Thanks everyone!

[webknjaz]

The built Rust libraries still need to be loaded ar runtime, as otherwise large parts of cryptography that have been written in Rust wouldn't work anymore.

Looks like you're confusing the terminology, then. The Rust itself is a compiler. It produces shared objects (*.so) that contain something executable and loadable from within Python. When that happens, there's no interaction with Rust whatsoever. These shared objects are technically called Python C-extensions (because they use Python C-API).
So Rust is used at build time. But the artifacts produced by it are loaded in runtime. You're probably talking about the latter.