Airflow Upgrade with Cherry-Picks [only] from 1.10.4+twtr #64

ayushSethi22 · 2021-01-05T11:24:48Z

Make sure you have checked all steps below.

JIRA

My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a JIRA issue.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

Passes flake8

* CP Contains fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * reformat * flake8 fix

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) Commit already available in 1.10.12

CP contains [EWT-16]: Airflow fix for manual trigger during version upgrade (twitter-forks#13) * [EWT-16]: Airflow fix for manual trigger during version upgrade

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) CP of f757a54 Commit already available in 1.10.12

…-forks#16)

[CP] Contains [AIRFLOW-5597] Linkify urls in task instance log CP of f757a54 Commit already available in 1.10.12

CP contains [TWTTR] Fix for rendering code on UI (twitter-forks#34)

CP contains [TWTR] CP from 1.10+twtr (twitter-forks#35) * 99ee040: CP from 1.10+twtr * 2e01c24: CP from 1.10.4 ([TWTR][AIRFLOW-4939] Fixup use of fallback kwarg in conf.getint) * 00cb4ae: [TWTR][AIRFLOW-XXXX] Cherry-pick d4a83bc and bump version (twitter-forks#21) * CP 51b1aee: Relax version requiremets (twitter-forks#24) * CP 67a4d1c: [CX-16266] Change with reference to 1a4c164 commit in open source (twitter-forks#25) * CP 54bd095: [TWTR][CX-17516] Queue tasks already being handled by the executor (twitter-forks#26) * CP 87fcc1c: [TWTR][CX-17516] Requeue tasks in the queued state (twitter-forks#27) * CP 98a1ca9: [AIRFLOW-6625] Explicitly log using utf-8 encoding (apache#7247) (twitter-forks#31) * fixing models.py and jobs.py file fix after CP * fixing typo and version bump Co-authored-by: Vishesh Jain <[email protected]>

CP Contains Experiment API path fix (twitter-forks#37) Co-authored-by: Vishesh Jain <[email protected]>

CP Contains Export scheduler env variable into worker pods. (twitter-forks#38)

Cp Contains [EWT-115][EWT-118] Initialise dag var to None and fix for DagModel.fileloc (missed in EWT-16) (twitter-forks#39) Co-authored-by: Vishesh Jain <[email protected]> Co-authored-by: aoen <[email protected]>

[CX-16591] Fix regex to work with impersonated clusters like airflow_scheduler_ddavydov (twitter-forks#42)

[CP][EWT-128] Fetch task logs from worker pods (19ac45a) (twitter-forks#43)

[CP][AIRFLOW-6561][EWT-290]: Adding priority class and default resource for worker pod. (twitter-forks#47)

[CP][EWT-302]Patch Pool.DEFAULT_POOL_NAME in BaseOperator (apache#8587) (twitter-forks#49) Open source commit id: b37ce29 Co-authored-by: Vishesh Jain <[email protected]>

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (twitter-forks#52) CP of 2d5b8a5

…itter-forks#51) Update the dataflow URL regex as per AIRFLOW-9323 Co-authored-by: Rajat Srivastava <[email protected]>

EWT-370: Use python3 to launch the dataflow job. (twitter-forks#53)

[EWT-450] fixing sla miss triggering duplicate alerts every minute (twitter-forks#56) Co-authored-by: Vishesh Jain <[email protected]>

[CP] Handle IntegrityErrors for trigger dagruns & add Stacktrace when DagFileProcessorManager gets killed (twitter-forks#57) CP of faaf179 - from master CP of 2102122 - from 1.10.12

[TWTR][EWT-472] Add lifecycle support while launching worker pods (twitter-forks#59)

[TWTTR] Don't enqueue tasks again if already queued for K8sExecutor(twitter-forks#60) Basically reverting commit 87fcc1c and making changes specifically into the Celery Executor class only.

[CP][TWTR][EWT-377] Fix DagBag bug when a Dag has invalid schedule_interval (twitter-forks#61) CP of 5605d10 & apache#11462

[TWTR][EWT-350] Reverting the last commit partially (twitter-forks#62)

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) CP of f757a54 Commit already available in 1.10.12

…rib files as they have been moved to airflow

…ontrib kubernetes_executor

airflow/executors/kubernetes_executor.py

vshshjn7 · 2021-01-05T12:33:51Z

airflow/models/dag.py

-            dag = self.get_dag()
-        return dag
+
+    def get_dag(self):


looks like get_dag logic has changed.
This was in 1.10.4 (https://github.com/apache/airflow/blob/1.10.4/airflow/models/dag.py#L1508)

Can you remove this change?
Here, you just need to replace fileloc with get_local_fileloc(), rest will be same

vshshjn7 · 2021-01-05T12:36:13Z

airflow/models/taskinstance.py

@@ -1004,6 +1004,7 @@ def signal_handler(signum, frame):
            self.state = State.SUCCESS
        except AirflowSkipException as e:
            # Recording SKIP
+            # This change is in reference to [AIRFLOW-5653][CX-16266]


do we need this statement?

This commit was done by me in OS Airflow 1.10.6 so it need not be cherry picked. That should remove this comment as well.

vshshjn7 · 2021-01-05T12:43:41Z

airflow/models/dag.py

@@ -1767,6 +1772,24 @@ class DagModel(Base):
    def __repr__(self):
        return "<DAG: {self.dag_id}>".format(self=self)

+    def get_local_fileloc(self):


Make sure that we use get_local_fileloc instead of fileloc

Checked for rest of the instances, looks good

msumit · 2021-01-06T07:00:13Z

airflow/config_templates/default_airflow.cfg

-# Should be supplied in the format: ``key = value``
+# Should be supplied in the format: key = value
+
+[kubernetes_worker_resources]


Do we still need to add this section? Isn't it supposed to be available in 1.10.14 by default?

msumit · 2021-01-06T07:00:54Z

airflow/config_templates/default_airflow.cfg

@@ -595,7 +595,7 @@ pool = prefork

 # The number of seconds to wait before timing out ``send_task_to_executor`` or
 # ``fetch_celery_task_state`` operations.
-operation_timeout = 2
+operation_timeout = 10


Can we add this override in the airflow.cfg we've in source/airflow?

rajatsri28 · 2021-01-06T07:18:39Z

airflow/models/taskinstance.py

@@ -1004,6 +1004,7 @@ def signal_handler(signum, frame):
            self.state = State.SUCCESS
        except AirflowSkipException as e:
            # Recording SKIP
+            # This change is in reference to [AIRFLOW-5653][CX-16266]


This commit was done by me in OS Airflow 1.10.6 so it need not be cherry picked. That should remove this comment as well.

airflow/models/baseoperator.py

abhishekbafna

Hi, I do not see this change in PR.

abhishekbafna@4ac9a6d

.arcconfig

airflow/models/baseoperator.py

abhishekbafna · 2021-01-06T08:44:22Z

airflow/version.py

@@ -18,4 +18,6 @@
 # under the License.
 #

-version = '1.10.14'
+version = '1.10.14+test1'


Instead of test, it should be twtr or something.

…ments

…comments

…on set from test to twtr

README_TWITTER.md

vshshjn7 · 2021-01-07T06:45:58Z

UPDATING.md

+```
+Fix for this, https://github.com/apache/airflow/pull/8587
+
+## Airflow 1.10.4


this line can be removed

airflow/migrations/versions/d38e04c12aa2_add_serialized_dag_table.py

…comments

Ayush Sethi and others added 28 commits December 15, 2020 01:32

EWT-569 : Initial Commit for migrations

ed564bd

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 76fe7ac from 1.10.4

587b99c

* CP Contains fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * reformat * flake8 fix

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00

8fafd51

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) Commit already available in 1.10.12

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00

816f7c9

CP contains [EWT-16]: Airflow fix for manual trigger during version upgrade (twitter-forks#13) * [EWT-16]: Airflow fix for manual trigger during version upgrade

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 91d2b00

2bb4059

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) CP of f757a54 Commit already available in 1.10.12

CP(55bb579) [AIRFLOW-5597] Linkify urls in task instance log (twitter…

458d299

…-forks#16)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 94cdcf6

2a5f70d

[CP] Contains [AIRFLOW-5597] Linkify urls in task instance log CP of f757a54 Commit already available in 1.10.12

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 4ce8d4c from 1.10.4

78dafd6

CP contains [TWTTR] Fix for rendering code on UI (twitter-forks#34)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : f7050fb

cadbf66

CP Contains Experiment API path fix (twitter-forks#37) Co-authored-by: Vishesh Jain <[email protected]>

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 8a689af from 1.10.4

234156f

CP Contains Export scheduler env variable into worker pods. (twitter-forks#38)

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick 5875a15 from 1.10.4

6ed530d

Cp Contains [EWT-115][EWT-118] Initialise dag var to None and fix for DagModel.fileloc (missed in EWT-16) (twitter-forks#39) Co-authored-by: Vishesh Jain <[email protected]> Co-authored-by: aoen <[email protected]>

[EWT-569] Airflow Upgrade to 1.10.14, Cherry-Pick a68e2b3 from 1.10.4

119c428

[CX-16591] Fix regex to work with impersonated clusters like airflow_scheduler_ddavydov (twitter-forks#42)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : e9642c2

f5350ea

[CP][EWT-128] Fetch task logs from worker pods (19ac45a) (twitter-forks#43)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : d5d0a07

c54061a

[CP][AIRFLOW-6561][EWT-290]: Adding priority class and default resource for worker pod. (twitter-forks#47)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 9b58c88

1698a7a

[CP][EWT-302]Patch Pool.DEFAULT_POOL_NAME in BaseOperator (apache#8587) (twitter-forks#49) Open source commit id: b37ce29 Co-authored-by: Vishesh Jain <[email protected]>

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 7b52a71

bd62ed0

[CP][AIRFLOW-3121] Define closed property on StreamLogWriter (apache#3955) (twitter-forks#52) CP of 2d5b8a5

[EWT-361] Fix broken regex pattern for extracting dataflow job id (tw…

7c053df

…itter-forks#51) Update the dataflow URL regex as per AIRFLOW-9323 Co-authored-by: Rajat Srivastava <[email protected]>

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 4b5b977

93e2678

EWT-370: Use python3 to launch the dataflow job. (twitter-forks#53)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 596e24f

af5a6be

[EWT-450] fixing sla miss triggering duplicate alerts every minute (twitter-forks#56) Co-authored-by: Vishesh Jain <[email protected]>

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : b3d7fb4

3849402

[CP] Handle IntegrityErrors for trigger dagruns & add Stacktrace when DagFileProcessorManager gets killed (twitter-forks#57) CP of faaf179 - from master CP of 2102122 - from 1.10.12

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : bac4acd

27f136b

[TWTR][EWT-472] Add lifecycle support while launching worker pods (twitter-forks#59)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 6162402

565b6e2

[TWTTR] Don't enqueue tasks again if already queued for K8sExecutor(twitter-forks#60) Basically reverting commit 87fcc1c and making changes specifically into the Celery Executor class only.

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 1991419

4ea1c3c

[CP][TWTR][EWT-377] Fix DagBag bug when a Dag has invalid schedule_interval (twitter-forks#61) CP of 5605d10 & apache#11462

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : 48be0f9

840dc0c

[TWTR][EWT-350] Reverting the last commit partially (twitter-forks#62)

[EWT-569] Airflow Upgrade to 1.10.14 [CP] from 1.10.4+twtr : d8c473e

2fb2c2a

[CP][EWT-548][AIRFLOW-6527] Make send_task_to_executor timeout configurable (twitter-forks#63) CP of f757a54 Commit already available in 1.10.12

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Removing cont…

3f589ac

…rib files as they have been moved to airflow

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Adding back c…

ddd9d09

…ontrib kubernetes_executor

vshshjn7 reviewed Jan 5, 2021

View reviewed changes

airflow/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

vshshjn7 reviewed Jan 5, 2021

View reviewed changes

msumit reviewed Jan 6, 2021

View reviewed changes

rajatsri28 reviewed Jan 6, 2021

View reviewed changes

abhishekbafna reviewed Jan 6, 2021

View reviewed changes

Ayush Sethi added 3 commits January 7, 2021 11:18

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Resolving com…

cf95e34

…ments

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Resolving PR …

502007c

…comments

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Airflow Versi…

8f2290e

…on set from test to twtr

vshshjn7 reviewed Jan 7, 2021

View reviewed changes

README_TWITTER.md Outdated Show resolved Hide resolved

vshshjn7 reviewed Jan 7, 2021

View reviewed changes

airflow/migrations/versions/d38e04c12aa2_add_serialized_dag_table.py Outdated Show resolved Hide resolved

[EWT-569] Airflow Upgrade to 1.10.14 from 1.10.4+twtr : Resolving PR …

34c6696

…comments

msumit approved these changes Jan 8, 2021

View reviewed changes

msumit merged commit d85a9a3 into twitter-forks:1.10.14+twtr Jan 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow Upgrade with Cherry-Picks [only] from 1.10.4+twtr #64

Airflow Upgrade with Cherry-Picks [only] from 1.10.4+twtr #64

ayushSethi22 commented Jan 5, 2021

vshshjn7 Jan 5, 2021 •

edited

Loading

vshshjn7 Jan 5, 2021

rajatsri28 Jan 6, 2021

vshshjn7 Jan 5, 2021

ayushSethi22 Jan 7, 2021

msumit Jan 6, 2021

msumit Jan 6, 2021

rajatsri28 Jan 6, 2021

abhishekbafna left a comment

abhishekbafna Jan 6, 2021

vshshjn7 Jan 7, 2021

ayushSethi22 Jan 7, 2021

Airflow Upgrade with Cherry-Picks [only] from 1.10.4+twtr #64

Airflow Upgrade with Cherry-Picks [only] from 1.10.4+twtr #64

Conversation

ayushSethi22 commented Jan 5, 2021

JIRA

Description

Tests

Commits

Documentation

Code Quality

vshshjn7 Jan 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhishekbafna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vshshjn7 Jan 5, 2021 •

edited

Loading