Implement reading provider information from packages/sources

[potiuk]

This PR implements discovering and readin provider information from
packages (using entry_points) and - if found - from local
provider yaml files for the built-in airflow providers,
when they are found in the airflow.provider packages.
The provider.yaml files - if found - take precedence over the
package-provided ones.

Add displaying provider information in CLI

Closes: #12470

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Hey @ashb @mik-laj -> this is the "base" version of the ProvidersManager - updated to include both options:

• running directly from the sources (in which case provider information is read directly from provider.yaml file) - dev
• run-time version that reads the information from package entry_points - production

It also includes CLI that you can query the providers installed.

Reading from packages is much slower than from file, but it is bearable (~2-3 seconds on my machine). most of it is resolving and checking package dependencies, not reading the content. Also - currently - two packages (snowflake and amazon) fail to register because they have "ContextualVersionConflict" which I hope we will eventually resolve (#10854, #10854)

You can also try it out yourself - there is a new 'providers' cli command displaying the installed providers:

Works both - in dev env (with local sources) and prod env (with packages)

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

Hey @ashb. It's green - this one is "pure" provider discovery - without connections/extra links - it just contains the discovery and restructured provider package tests to show better what's happening.

Right now what happens:

1. for backport providers I install airflow 1.10.12 (soon 1.10.13) install all the provider.whl and provider.tgz packages and check if all classes from airlfow.providers are importable

2. for regular providers, I prepare now "airflow" wheel package and install it from there (not from sources directly) and install all the providers from packages (whl or .tgz - I run two checks), I check if all providers are importable and then I run the new airflow providers list command to get all the providers and see if we see the expected number of providers as output.

This does not work now because we have import version conflict problems, but once we solve it, it should work - we have 59 out of 60 now - so for now I just skip exit in the number check.

See here: https://github.com/apache/airflow/runs/1438201616?check_suite_focus=true#step:9:4436

I will continue on rebasing the others and we can discuss tomorrow how close we can get to support the cases that you have in mind (with plugins) not only the providers. I think we can find some good future-proof solutions now, I simply do not have the capacity to also implement plugins at the same time, but I think if we implement it correctly, we can have a follow up PR implementing the "plugin" case you mentioned as well, without some heavy modifications.

[ashb]

I think this is all we'd have to change to remove the yaml.safe_load from in the generated provieer_info.

It won't be the prettiest, but we can fix that by running black on it after generating the template.

I tested this and this is what the provider info for amazon package looks okay, if horrible without black formatting.

[ashb]

I think this is all we'd have to change to remove the yaml.safe_load from in the generated provieer_info.
It won't be the prettiest, but we can fix that by running black on it after generating the template.

I tested this and this is what the provider info for amazon package looks okay, if horrible without black formatting.

Formatting with black is quite straight forward:

+    import black
     with open(get_provider_file_path, "wt") as get_provider_file:
-        get_provider_file.write(get_provider_content)
+        get_provider_file.write(black.format_str(get_provider_content, mode=black.FileMode()))
 

(In black >= 0.20.8b0, that has been renamed to just Mode, but the version in my container was older than that.)

It now looks like:

def get_provider_info():
    {
        "package-name": "apache-airflow-providers-amazon",
        "name": "Amazon",
        "description": "Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).\n",
        "versions": ["1.0.0b2"],
        "integrations": [
            {

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

I will be running some more tests but I think everything addressed @ashb/@mik-laj

[potiuk]

It look like this one is good to go. Some quarantined tests failed. I am re-running it.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

@ashb -> I would love to merge that one, so that I can rebase and finish the other two :)

[ashb]

I hadn't been looking cos of all the failing tests. Is it ready now?

[potiuk]

I hadn't been looking cos of all the failing tests. Is it ready now?

Yep. Rebased it and but last time it passed all tests.

[potiuk]

gentle reminder @ash :)

[ashb]

One tiny change -- preemptively approving.

[kaxil]

needs rebasing on Master but LGTM otherwise