diff --git a/UPDATING.md b/UPDATING.md index a2e98ae93ca37b..3eb61f5104773c 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -26,6 +26,7 @@ assists users migrating to a new version. **Table of contents** - [Airflow Master](#airflow-master) +- [Airflow 1.10.5](#airflow-1105) - [Airflow 1.10.4](#airflow-1104) - [Airflow 1.10.3](#airflow-1103) - [Airflow 1.10.2](#airflow-1102) @@ -382,24 +383,24 @@ PR: [#5990](https://github.com/apache/airflow/pull/5990) FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has `*`, `?`, or `[` in it then you should replace these with `[*]`, `[?]`, and `[[]`. ### Change dag loading duration metric name -Change DAG file loading duration metric from -`dag.loading-duration.` to `dag.loading-duration.`. This is to +Change DAG file loading duration metric from +`dag.loading-duration.` to `dag.loading-duration.`. This is to better handle the case when a DAG file has multiple DAGs. ### Changes to ImapHook, ImapAttachmentSensor and ImapAttachmentToS3Operator ImapHook: -* The order of arguments has changed for `has_mail_attachment`, +* The order of arguments has changed for `has_mail_attachment`, `retrieve_mail_attachments` and `download_mail_attachments`. * A new `mail_filter` argument has been added to each of those. ImapAttachmentSensor: * The order of arguments has changed for `__init__`. -* A new `mail_filter` argument has been added to `__init__`. +* A new `mail_filter` argument has been added to `__init__`. ImapAttachmentToS3Operator: * The order of arguments has changed for `__init__`. -* A new `imap_mail_filter` argument has been added to `__init__`. +* A new `imap_mail_filter` argument has been added to `__init__`. ### Changes to `SubDagOperator` @@ -423,15 +424,15 @@ you should write `@GoogleCloudBaseHook.provide_gcp_credential_file` ### Changes to S3Hook -Note: The order of arguments has changed for `check_for_prefix`. +Note: The order of arguments has changed for `check_for_prefix`. The `bucket_name` is now optional. It falls back to the `connection schema` attribute. ### Changes to Google Transfer Operator -To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator` +To obtain pylint compatibility the `filter ` argument in `GcpTransferServiceOperationsListOperator` has been renamed to `request_filter`. ### Changes in Google Cloud Transfer Hook - To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and + To obtain pylint compatibility the `filter` argument in `GCPTransferServiceHook.list_transfer_job` and `GCPTransferServiceHook.list_transfer_operations` has been renamed to `request_filter`. ### Export MySQL timestamps as UTC @@ -462,7 +463,7 @@ Hence, the default value for `master_disk_size` in DataprocClusterCreateOperator ### Changes to SalesforceHook -* renamed `sign_in` function to `get_conn` +* renamed `sign_in` function to `get_conn` ### HTTPHook verify default value changed from False to True. @@ -473,8 +474,8 @@ This can be overwriten by using the extra_options param as `{'verify': False}`. * The following parameters have been replaced in all the methods in GCSHook: * `bucket` is changed to `bucket_name` - * `object` is changed to `object_name` - + * `object` is changed to `object_name` + * The `maxResults` parameter in `GoogleCloudStorageHook.list` has been renamed to `max_results` for consistency. ### Changes to CloudantHook @@ -606,10 +607,14 @@ The `do_xcom_push` flag (a switch to push the result of an operator to xcom or n See [AIRFLOW-3249](https://jira.apache.org/jira/browse/AIRFLOW-3249) to check if your operator was affected. ### Changes to Dataproc related Operators -The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from +The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from `dataproc_xxxx_properties` and `dataproc_xxx_jars` to `dataproc_properties` -and `dataproc_jars`respectively. -Arguments for dataproc_properties dataproc_jars +and `dataproc_jars`respectively. +Arguments for dataproc_properties dataproc_jars + +## Airflow 1.10.5 + +No breaking changes. ## Airflow 1.10.4 @@ -626,12 +631,12 @@ If you have a specific task that still requires Python 2 then you can use the Py ### Changes to GoogleCloudStorageHook -* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054) +* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054) * as a part of this replacement, the `multipart` & `num_retries` parameters for `GoogleCloudStorageHook.upload` method have been deprecated. The client library uses multipart upload automatically if the object/blob size is more than 8 MB - [source code](https://github.com/googleapis/google-cloud-python/blob/11c543ce7dd1d804688163bc7895cf592feb445f/storage/google/cloud/storage/blob.py#L989-L997). The client also handles retries automatically -* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`. +* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`. Updating to `google-cloud-storage >= 1.16` changes the signature of the upstream `client.get_bucket()` method from `get_bucket(bucket_name: str)` to `get_bucket(bucket_or_name: Union[str, Bucket])`. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (`GoogleCloudStorageHook().get_conn().get_bucket(...)` or similar) will need to be updated. @@ -896,7 +901,7 @@ then you need to change it like this @property def is_active(self): return self.active - + ### Support autodetected schemas to GoogleCloudStorageToBigQueryOperator GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required. @@ -908,7 +913,7 @@ define a schema_fields: gcs_to_bq.GoogleCloudStorageToBigQueryOperator( ... schema_fields={...}) - + or define a schema_object: gcs_to_bq.GoogleCloudStorageToBigQueryOperator(