[DSRE-9] - Upgrade Airflow wtmo to 1.10.15 bridge release prior to 2.0

[haroldwoo]

Why 1.10.15 initially?

https://airflow.apache.org/docs/apache-airflow/2.1.0/upgrading-to-2.html#step-2-upgrade-to-airflow-1-10-15-a-k-a-our-bridge-release

Changelogs of potential note to review (from 1.10.13 -> 1.10.15):

• Delete the pod launched by KubernetesPodOperator when task is cleared. apache/airflow#10666
• KubernetesPodOperator template fix apache/airflow#10963
• Fix issues with Gantt View apache/airflow#12419
• Fix pod_mutation_hook for 1.10.13 apache/airflow#10850
• Add XCom.deserialize_value to Airflow 1.10.13 apache/airflow#12328
• Mount airflow.cfg to pod_template_file apache/airflow#12311
• All k8s object must comply with JSON Schema apache/airflow#12003
• [add] custom airflow env into pod template apache/airflow#11480
• airflow test only works for tasks in 1.10, not whole dags apache/airflow#11191
• Security upgrade lodash from 4.17.19 to 4.17.20 apache/airflow#11095
• Mask Password in Log table when using the CLI apache/airflow#11468
• Deprecate adding Operators and Sensors via plugins apache/airflow#12069
• Deprecate importing Hooks from plugin-created module apache/airflow#12133
• BugFix: Tasks with depends_on_past or task_concurrency are stuck apache/airflow#12663
• [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY apache/airflow#3651
• [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac apache/airflow#3729
• [AIRFLOW-2886] Secure Flask SECRET_KEY apache/airflow#3738
• Rename [scheduler] max_threads to [scheduler] parsing_processes apache/airflow#12605
• Add metric for scheduling delay between first run task & expected start time apache/airflow#9544
• Add new-style 2.0 command names for Airflow 1.10.x apache/airflow#12725
• Ensure webserver isn't running with old config apache/airflow#12747
• Add Kubernetes cleanup-pods CLI command for Helm Chart apache/airflow#11802
• KubernetesExecutor should accept images from executor_config apache/airflow#13074
• Scheduler should acknowledge active runs properly (in 1.10.x) apache/airflow#13803
• [kubernetes_generate_dag_yaml] - Fix dag yaml generate func in version 1.10.14 apache/airflow#13816
• Make --yes work for task clear command apache/airflow#14188
• Fix merging of secrets and configmaps apache/airflow#14090
• Fix webserver exiting when gunicorn master crashes. Closes #13469 apache/airflow#13470
• [AIRFLOW-4438] Add Gzip compression to S3_hook  apache/airflow#8571
• Add "already checked" to failed pods in K8sPodOperator apache/airflow#11368
• Add generate_yaml command to easily test KubernetesExecutor before deploying pods apache/airflow#10677

full list (https://airflow.apache.org/docs/apache-airflow/stable/changelog.html#airflow-1-10-15-2021-03-17)

TODO:

• resolve pip dependencies
• review changelogs
• resolve - Broken DAG: [/app/dags/mozfun.py] cannot import name '_check_google_client_version' from 'pandas_gbq.gbq' (/usr/local/lib/python3.7/site-packages/pandas_gbq/gbq.py)
• Test containers
• Diff 1.10.15 and 1.10.12 code for dags/operators/gcp_container_operator, and dataproc operators for moz_dataproc_operator usage. Also s3_to_gcs_transfer_operator

Test jobs

• GKEPodOperator
• Dataproc (taar_lite standalone, use {{yesterday_ds}} as date)

Test plugins

• backfill plugin
• Mozilla docs

Addl note:
In 1.10.15, Adding Operators, Hooks and Sensors via Airflow Plugins is deprecated
The ability to import Operators, Hooks and Sensors via the plugin mechanism has been deprecated and will raise warnings in Airflow 1.10.13 and will be removed completely in Airflow 2.0.
Check https://airflow.apache.org/docs/1.10.13/howto/custom-operator.html to see how you can create and import Custom Hooks, Operators and Sensors.
In the plugins folder these may be effected?

• log_email_backend.py
• celery_visibility_timeout_fix.py

[acmiyaguchi]

Looks okay to me, since this is primarily a release with backported functionality. Have you run airflow upgrade_check?

[haroldwoo]

Not yet. The upgrade_check is for 1.10 to 2.0