Use Pip 21.* to install airflow officially

[potiuk]

The PIP 20.2.4 was so far the only officially supported installation
mechanism for Airflow as there were some problems with conflicting
dependencies (which were ignored by previous versio of PIP).

This change attempts to solve this by removing a [gcp] extra
from apache-beam which turns out to be the major source of
the problem - as it contains requirements to the old version of
google client libraries (but apparently only used for tests).

The "apache-beam" provider migh however need the [gcp] extra
for other components so in order to not break the backwards
compatibility, another approach is used.

Instead of adding [gcp] as extra in the apache-beam extra,
the apache.beam provider's [google] extra is extended with
'apache-beam[gcp]' additional requirement so that whenever the
provider is installed, the apache-beam with [gcp] extra is installed
as well.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Hey @ashb and @kaxil -> If we get this one merged, I think we might want to release a few providers (Google, Azure) where we had some small issues with warning (I am looking at an Azure warning in a moment) + Apache.Beam operator with the updated extra and we might aim to release 2.0.3 rather quickly - with mostly "pip 21 full support".

[github-actions]

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.

[potiuk]

It's green. Looks like, when we merge, we have a full PIP 21 support for Airflow @kaxil @ashb

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

I'd love that one to be merged, to finally get PIP 21 support (and keep the deps under control).

[uranusjr]

Just a heads-up, pip 21.1.1 will likely be released next week to fix an import issue for 3.6.0 and 3.6.1.

[potiuk]

Just a heads-up, pip 21.1.1 will likely be released next week to fix an import issue for 3.6.0 and 3.6.1.

Yep. I think I will add a couple of "automated" warnings (a'la dependabot). that will make us aware of the need to upgrade when newer versions of PIP/Pylint and few other tools that we are pinning. Dependabot is not good for us, because we are continuously updating automatically, and keep open dependencies, but those few dependencies we have pinned might make perfect sense to get "warnings" about.