Skip to content

Commit

Permalink
Use Pip 21.* to install airflow officially (apache#15513)
Browse files Browse the repository at this point in the history 
* Use Pip 21.* to install airflow officially

The PIP 20.2.4 was so far the only officially supported installation
mechanism for Airflow as there were some problems with conflicting
dependencies (which were ignored by previous versio of PIP).

This change attempts to solve this by removing a [gcp] extra
from `apache-beam` which turns out to be the major source of
the problem - as it contains requirements to the old version of
google client libraries (but apparently only used for tests).

The "apache-beam" provider migh however need the [gcp] extra
for other components so in order to not break the backwards
compatibility, another approach is used.

Instead of adding [gcp] as extra in the apache-beam extra,
the apache.beam provider's [google] extra is extended with
'apache-beam[gcp]' additional requirement so that whenever the
provider is installed, the apache-beam with [gcp] extra is installed
as well.

* Update airflow/providers/apache/beam/CHANGELOG.rst

Co-authored-by: Tzu-ping Chung <[email protected]>

* Update airflow/providers/apache/beam/CHANGELOG.rst

Co-authored-by: Tzu-ping Chung <[email protected]>

* Update airflow/providers/google/CHANGELOG.rst

Co-authored-by: Tzu-ping Chung <[email protected]>

* Update airflow/providers/google/CHANGELOG.rst

Co-authored-by: Tzu-ping Chung <[email protected]>

Co-authored-by: Tzu-ping Chung <[email protected]>
(cherry picked from commit e229f35)
  • Loading branch information
@potiuk
potiuk committed May 6, 2021
1 parent 2ee00ab commit 4423ea7
Show file tree
Hide file tree
Showing 28 changed files with 48 additions and 173 deletions.
20 changes: 2 additions & 18 deletions CONTRIBUTING.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -553,15 +553,7 @@ Airflow dependencies

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down Expand Up @@ -788,15 +780,7 @@ Pinned constraint files

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
10 changes: 1 addition & 9 deletions CONTRIBUTORS_QUICK_START.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -167,15 +167,7 @@ Setup Airflow with Breeze and PyCharm

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ ARG AIRFLOW_GID="50000"

ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"

ARG AIRFLOW_PIP_VERSION=20.2.4
ARG AIRFLOW_PIP_VERSION=21.1

# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR="on"
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.ci
View file
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
ARG INSTALL_FROM_PYPI="true"
ARG AIRFLOW_PIP_VERSION=20.2.4
ARG AIRFLOW_PIP_VERSION=21.1
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
Expand Down
12 changes: 2 additions & 10 deletions IMAGES.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -172,15 +172,7 @@ This will build the image using command similar to:
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down Expand Up @@ -632,7 +624,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_RUNTIME_APT_ENV`` | | Additional env variables defined |
| | | when installing runtime deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PIP_VERSION`` | ``20.2.4`` | PIP version used. |
| ``AIRFLOW_PIP_VERSION`` | ``21.1`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
Expand Down
8 changes: 0 additions & 8 deletions INSTALL
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,6 @@ java -jar apache-rat.jar -E ./.rat-excludes -d .
python3 -m venv PATH_TO_YOUR_VENV
source PATH_TO_YOUR_VENV/bin/activate

NOTE!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.

# [required] building and installing by pip (preferred)
pip install .

Expand Down
20 changes: 2 additions & 18 deletions LOCAL_VIRTUALENV.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,7 @@ Extra Packages

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down Expand Up @@ -137,15 +129,7 @@ To create and initialize the local virtualenv:

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
9 changes: 1 addition & 8 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -149,14 +149,7 @@ correct Airflow tag/version/branch and Python versions in the URL.

NOTE!!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow reliably, you might need to either downgrade
pip to version 20.2.4 `pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3,
you might need to add option] `--use-deprecated legacy-resolver` to your pip install command.
While `pip 20.3.3` solved most of the `teething` problems of 20.3, this note will remain here until we
set `pip 20.3` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only `pip` installation is currently officially supported.
Only `pip` installation is currently officially supported.

While they are some successes with using other tools like [poetry](https://python-poetry.org) or
[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as
Expand Down
7 changes: 0 additions & 7 deletions UPDATING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1727,13 +1727,6 @@ you should use `pip install apache-airflow[apache.atlas]`.

NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.


If you want to install integration for Microsoft Azure, then instead of

```
Expand Down
4 changes: 4 additions & 0 deletions airflow/provider.yaml.schema.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,10 @@
"items": {
"type": "string"
}
},
"additional-extras": {
"type": "object",
"description": "Additional extras that the provider should have"
}
},
"additionalProperties": false,
Expand Down
8 changes: 0 additions & 8 deletions airflow/providers/apache/beam/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,6 @@ are in `airflow.providers.apache.beam` python package.

## Installation

NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.

You can install this package on top of an existing airflow 2.* installation via
`pip install apache-airflow-providers-apache-beam`

Expand Down
2 changes: 1 addition & 1 deletion airflow/providers/apache/hive/transfers/mssql_to_hive.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# pylint: disable=no-member
"""This module contains operator to move data from MSSQL to Hive."""

from collections import OrderedDict
Expand Down
1 change: 1 addition & 0 deletions airflow/providers/microsoft/mssql/hooks/mssql.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# pylint: disable=no-member
"""Microsoft SQLServer hook module"""

import pymssql
Expand Down
9 changes: 0 additions & 9 deletions dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,6 @@ are in ``{{FULL_PACKAGE_NAME}}`` python package.
Installation
------------

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


You can install this package on top of an existing airflow 2.* installation via
``pip install {{PACKAGE_PIP_NAME}}``
{%- if PIP_REQUIREMENTS %}
Expand Down
8 changes: 0 additions & 8 deletions dev/provider_packages/PROVIDER_README_TEMPLATE.rst.jinja2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,6 @@ in the `documentation <https://airflow.apache.org/docs/{{ PACKAGE_PIP_NAME }}/{{
Installation
============

NOTE!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.

You can install this package on top of an existing airflow 2.* installation via
``pip install {{PACKAGE_PIP_NAME}}``

Expand Down
2 changes: 1 addition & 1 deletion dev/provider_packages/SETUP_TEMPLATE.py.jinja2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ from setuptools import find_namespace_packages, setup

logger = logging.getLogger(__name__)

version = '{{ RELEASE_NO_LEADING_ZEROS }}{{ VERSION_SUFFIX }}'
version = '{{ RELEASE_NO_LEADING_ZEROS }}'

my_dir = dirname(__file__)

Expand Down
24 changes: 22 additions & 2 deletions dev/provider_packages/prepare_provider_packages.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,14 @@ def get_package_extras(provider_package_id: str) -> Dict[str, List[str]]:
if cross_provider_dependencies.get(provider_package_id)
else {}
)
provider_yaml_dict = get_provider_yaml(provider_package_id)
additional_extras = provider_yaml_dict.get('additional-extras')
if additional_extras:
for key in additional_extras:
if key in extras_dict:
extras_dict[key].append(additional_extras[key])
else:
extras_dict[key] = additional_extras[key]
return extras_dict


Expand Down Expand Up @@ -1295,11 +1303,11 @@ def convert_to_provider_info(provider_info: Dict[str, Any]) -> Dict[str, Any]:
This method converts the full provider.yaml schema into the limited version needed at runtime.
"""
updated_provider_info = deepcopy(provider_info)
expression = jsonpath_ng.parse("[hooks,operators,integrations,sensors,transfers]")
expression = jsonpath_ng.parse("[hooks,operators,integrations,sensors,transfers,additional-extras]")
return expression.filter(lambda x: True, updated_provider_info)


def get_provider_info_from_provider_yaml(provider_package_id: str) -> Dict[str, Any]:
def get_provider_yaml(provider_package_id: str) -> Dict[str, Any]:
"""
Retrieves provider info from the provider yaml file. The provider yaml file contains more information
than provider_info that is used at runtime. This method converts the full provider yaml file into
Expand All @@ -1312,6 +1320,18 @@ def get_provider_info_from_provider_yaml(provider_package_id: str) -> Dict[str,
raise Exception(f"The provider.yaml file is missing: {provider_yaml_file_name}")
with open(provider_yaml_file_name) as provider_file:
provider_yaml_dict = yaml.load(provider_file, SafeLoader) # noqa
return provider_yaml_dict


def get_provider_info_from_provider_yaml(provider_package_id: str) -> Dict[str, Any]:
"""
Retrieves provider info from the provider yaml file. The provider yaml file contains more information
than provider_info that is used at runtime. This method converts the full provider yaml file into
stripped-down provider info and validates it against deprecated 2.0.0 schema and runtime schema.
:param provider_package_id: package id to retrieve provider.yaml from
:return: provider_info dictionary
"""
provider_yaml_dict = get_provider_yaml(provider_package_id=provider_package_id)
provider_info = convert_to_provider_info(provider_yaml_dict)
validate_provider_info_with_2_0_0_schema(provider_info)
validate_provider_info_with_runtime_schema(provider_info)
Expand Down
7 changes: 1 addition & 6 deletions docs/apache-airflow/installation.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,7 @@ tested with Python 3.6, 3.7, and 3.8, but does not yet support Python 3.9.
Installation tools
''''''''''''''''''

The official way of installing Airflow is with the ``pip`` tool.
There was a recent (November 2020) change in resolver, so currently only 20.2.4 version is officially
supported, although you might have a success with 20.3.3+ version (to be confirmed if all initial
issues from ``pip`` 20.3.0 release have been fixed in 20.3.3). In order to install Airflow you need to
either downgrade pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
Only ``pip`` installation is currently officially supported.

While there are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
10 changes: 1 addition & 9 deletions docs/apache-airflow/start/local.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,7 @@ This quick start guide will help you bootstrap a Airflow standalone instance on

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
10 changes: 1 addition & 9 deletions docs/apache-airflow/upgrade-check.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,7 @@ time of writing: 1.10.15) and then to download this package and run the script a

.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.
Only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
Expand Down
Loading

0 comments on commit 4423ea7

Please sign in to comment.