Skip to content

Commit

Permalink
[AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted f…
Browse files Browse the repository at this point in the history
…rom UPDATING.md (#6240)
  • Loading branch information
ryw authored and feluelle committed Oct 3, 2019
1 parent b35972f commit 426d3a9
Showing 1 changed file with 23 additions and 18 deletions.
41 changes: 23 additions & 18 deletions UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ assists users migrating to a new version.
**Table of contents**

- [Airflow Master](#airflow-master)
- [Airflow 1.10.5](#airflow-1105)
- [Airflow 1.10.4](#airflow-1104)
- [Airflow 1.10.3](#airflow-1103)
- [Airflow 1.10.2](#airflow-1102)
Expand Down Expand Up @@ -382,24 +383,24 @@ PR: [#5990](https://github.com/apache/airflow/pull/5990)
FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has `*`, `?`, or `[` in it then you should replace these with `[*]`, `[?]`, and `[[]`.

### Change dag loading duration metric name
Change DAG file loading duration metric from
`dag.loading-duration.<dag_id>` to `dag.loading-duration.<dag_file>`. This is to
Change DAG file loading duration metric from
`dag.loading-duration.<dag_id>` to `dag.loading-duration.<dag_file>`. This is to
better handle the case when a DAG file has multiple DAGs.

### Changes to ImapHook, ImapAttachmentSensor and ImapAttachmentToS3Operator

ImapHook:
* The order of arguments has changed for `has_mail_attachment`,
* The order of arguments has changed for `has_mail_attachment`,
`retrieve_mail_attachments` and `download_mail_attachments`.
* A new `mail_filter` argument has been added to each of those.

ImapAttachmentSensor:
* The order of arguments has changed for `__init__`.
* A new `mail_filter` argument has been added to `__init__`.
* A new `mail_filter` argument has been added to `__init__`.

ImapAttachmentToS3Operator:
* The order of arguments has changed for `__init__`.
* A new `imap_mail_filter` argument has been added to `__init__`.
* A new `imap_mail_filter` argument has been added to `__init__`.

### Changes to `SubDagOperator`

Expand All @@ -423,15 +424,15 @@ you should write `@GoogleCloudBaseHook.provide_gcp_credential_file`

### Changes to S3Hook

Note: The order of arguments has changed for `check_for_prefix`.
Note: The order of arguments has changed for `check_for_prefix`.
The `bucket_name` is now optional. It falls back to the `connection schema` attribute.

### Changes to Google Transfer Operator
To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator`
To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator`
has been renamed to `request_filter`.

### Changes in Google Cloud Transfer Hook
To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and
To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and
`GCPTransferServiceHook.list_transfer_operations` has been renamed to `request_filter`.

### Export MySQL timestamps as UTC
Expand Down Expand Up @@ -462,7 +463,7 @@ Hence, the default value for `master_disk_size` in DataprocClusterCreateOperator

### Changes to SalesforceHook

* renamed `sign_in` function to `get_conn`
* renamed `sign_in` function to `get_conn`

### HTTPHook verify default value changed from False to True.

Expand All @@ -473,8 +474,8 @@ This can be overwriten by using the extra_options param as `{'verify': False}`.

* The following parameters have been replaced in all the methods in GCSHook:
* `bucket` is changed to `bucket_name`
* `object` is changed to `object_name`
* `object` is changed to `object_name`

* The `maxResults` parameter in `GoogleCloudStorageHook.list` has been renamed to `max_results` for consistency.

### Changes to CloudantHook
Expand Down Expand Up @@ -606,10 +607,14 @@ The `do_xcom_push` flag (a switch to push the result of an operator to xcom or n
See [AIRFLOW-3249](https://jira.apache.org/jira/browse/AIRFLOW-3249) to check if your operator was affected.

### Changes to Dataproc related Operators
The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from
The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from
`dataproc_xxxx_properties` and `dataproc_xxx_jars` to `dataproc_properties`
and `dataproc_jars`respectively.
Arguments for dataproc_properties dataproc_jars
and `dataproc_jars`respectively.
Arguments for dataproc_properties dataproc_jars

## Airflow 1.10.5

No breaking changes.

## Airflow 1.10.4

Expand All @@ -626,12 +631,12 @@ If you have a specific task that still requires Python 2 then you can use the Py

### Changes to GoogleCloudStorageHook

* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
* as a part of this replacement, the `multipart` & `num_retries` parameters for `GoogleCloudStorageHook.upload` method have been deprecated.

The client library uses multipart upload automatically if the object/blob size is more than 8 MB - [source code](https://github.com/googleapis/google-cloud-python/blob/11c543ce7dd1d804688163bc7895cf592feb445f/storage/google/cloud/storage/blob.py#L989-L997). The client also handles retries automatically

* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.
* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.

Updating to `google-cloud-storage >= 1.16` changes the signature of the upstream `client.get_bucket()` method from `get_bucket(bucket_name: str)` to `get_bucket(bucket_or_name: Union[str, Bucket])`. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (`GoogleCloudStorageHook().get_conn().get_bucket(...)` or similar) will need to be updated.

Expand Down Expand Up @@ -896,7 +901,7 @@ then you need to change it like this
@property
def is_active(self):
return self.active

### Support autodetected schemas to GoogleCloudStorageToBigQueryOperator

GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required.
Expand All @@ -908,7 +913,7 @@ define a schema_fields:
gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
...
schema_fields={...})

or define a schema_object:

gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
Expand Down

0 comments on commit 426d3a9

Please sign in to comment.