Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md #6240

Merged
merged 2 commits into from
Oct 3, 2019
Merged

[AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md #6240

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
75630cf
Fix whitespace
ryw Oct 2, 2019
fb0fe46
Make it clear that 1.10.5 wasn't accidently omitted
ryw Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 23 additions & 18 deletions UPDATING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ assists users migrating to a new version.
**Table of contents**

- [Airflow Master](#airflow-master)
- [Airflow 1.10.5](#airflow-1105)
- [Airflow 1.10.4](#airflow-1104)
- [Airflow 1.10.3](#airflow-1103)
- [Airflow 1.10.2](#airflow-1102)
Expand All @@ -43,24 +44,24 @@ assists users migrating to a new version.
FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has `*`, `?`, or `[` in it then you should replace these with `[*]`, `[?]`, and `[[]`.

### Change dag loading duration metric name
Change DAG file loading duration metric from
`dag.loading-duration.<dag_id>` to `dag.loading-duration.<dag_file>`. This is to
Change DAG file loading duration metric from
`dag.loading-duration.<dag_id>` to `dag.loading-duration.<dag_file>`. This is to
better handle the case when a DAG file has multiple DAGs.

### Changes to ImapHook, ImapAttachmentSensor and ImapAttachmentToS3Operator

ImapHook:
* The order of arguments has changed for `has_mail_attachment`,
* The order of arguments has changed for `has_mail_attachment`,
`retrieve_mail_attachments` and `download_mail_attachments`.
* A new `mail_filter` argument has been added to each of those.

ImapAttachmentSensor:
* The order of arguments has changed for `__init__`.
* A new `mail_filter` argument has been added to `__init__`.
* A new `mail_filter` argument has been added to `__init__`.

ImapAttachmentToS3Operator:
* The order of arguments has changed for `__init__`.
* A new `imap_mail_filter` argument has been added to `__init__`.
* A new `imap_mail_filter` argument has been added to `__init__`.

### Changes to `SubDagOperator`

Expand All @@ -84,15 +85,15 @@ you should write `@GoogleCloudBaseHook.provide_gcp_credential_file`

### Changes to S3Hook

Note: The order of arguments has changed for `check_for_prefix`.
Note: The order of arguments has changed for `check_for_prefix`.
The `bucket_name` is now optional. It falls back to the `connection schema` attribute.

### Changes to Google Transfer Operator
To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator`
To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator`
has been renamed to `request_filter`.

### Changes in Google Cloud Transfer Hook
To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and
To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and
`GCPTransferServiceHook.list_transfer_operations` has been renamed to `request_filter`.

### Export MySQL timestamps as UTC
Expand Down Expand Up @@ -123,7 +124,7 @@ Hence, the default value for `master_disk_size` in DataprocClusterCreateOperator

### Changes to SalesforceHook

* renamed `sign_in` function to `get_conn`
* renamed `sign_in` function to `get_conn`

### HTTPHook verify default value changed from False to True.

Expand All @@ -134,8 +135,8 @@ This can be overwriten by using the extra_options param as `{'verify': False}`.

* The following parameters have been replaced in all the methods in GCSHook:
* `bucket` is changed to `bucket_name`
* `object` is changed to `object_name`
* `object` is changed to `object_name`

* The `maxResults` parameter in `GoogleCloudStorageHook.list` has been renamed to `max_results` for consistency.

### Changes to CloudantHook
Expand Down Expand Up @@ -267,10 +268,14 @@ The `do_xcom_push` flag (a switch to push the result of an operator to xcom or n
See [AIRFLOW-3249](https://jira.apache.org/jira/browse/AIRFLOW-3249) to check if your operator was affected.

### Changes to Dataproc related Operators
The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from
The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from
`dataproc_xxxx_properties` and `dataproc_xxx_jars` to `dataproc_properties`
and `dataproc_jars`respectively.
Arguments for dataproc_properties dataproc_jars
and `dataproc_jars`respectively.
Arguments for dataproc_properties dataproc_jars

## Airflow 1.10.5

No breaking changes.

## Airflow 1.10.4

Expand All @@ -287,12 +292,12 @@ If you have a specific task that still requires Python 2 then you can use the Py

### Changes to GoogleCloudStorageHook

* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
* as a part of this replacement, the `multipart` & `num_retries` parameters for `GoogleCloudStorageHook.upload` method have been deprecated.

The client library uses multipart upload automatically if the object/blob size is more than 8 MB - [source code](https://github.com/googleapis/google-cloud-python/blob/11c543ce7dd1d804688163bc7895cf592feb445f/storage/google/cloud/storage/blob.py#L989-L997). The client also handles retries automatically

* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.
* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.

Updating to `google-cloud-storage >= 1.16` changes the signature of the upstream `client.get_bucket()` method from `get_bucket(bucket_name: str)` to `get_bucket(bucket_or_name: Union[str, Bucket])`. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (`GoogleCloudStorageHook().get_conn().get_bucket(...)` or similar) will need to be updated.

Expand Down Expand Up @@ -557,7 +562,7 @@ then you need to change it like this
@property
def is_active(self):
return self.active

### Support autodetected schemas to GoogleCloudStorageToBigQueryOperator

GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required.
Expand All @@ -569,7 +574,7 @@ define a schema_fields:
gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
...
schema_fields={...})

or define a schema_object:

gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
Expand Down