[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability #67

kaxil · 2019-10-23T16:35:21Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5088
- https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler

Description

Here are some details about my PR, including screenshots of any UI changes:
*Backport of [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability apache/airflow#5743 for v1-10- branches **
Based on [AIRFLOW-5088][AIP-24] Add DAG serialization using JSON apache/airflow#5701, this PR implements functionalities including writing serialized DAGs to DB in scheduler, reading DAGs from DB in webserver, controlled by [core] dagcached

The goal is to decouple webserver from the DAG folder, instead it reads everything from database.

Rendering template by functions is an exception, in that case it needs to re-import DAG, because functions are stringified in serialized DAG.

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

Passes flake8

It implements the method proposed in AIP-24 to serialize DAG. It will be used in DAG persistency in DB to solve webserver scalability issue. (cherry picked from commit 2bd1a51ec75f680a6e6e2101bd948a78421a644a)

…scalability (apache#5743)

(cherry picked from commit 7ce34b2a959fc1f8322836f38f474a831e4901a1)

(cherry picked from commit c068c67c48d294a589b58be0d0ad8b657c361a77) (cherry picked from commit 04fbf2beac57dcf26b118ebbe5a2bf175ce08af8)

…matching date in DAG (cherry picked from commit be412522cb95a19a51b2f208ae8ebea76e8b667a)

(cherry picked from commit d030b10bec9cd0e468f36e97e131d497d5a43fc6)

- Code Cleanup for JSON columns - Test code to allow old mysql & sqlite versions (cherry picked from commit 1db8044f9d29edf25f2b8ad4cd21c496c243534a)

(cherry picked from commit d14497ff28d123d45d626019cabcbd977c5de79d)

This was a valid type for schedule_interval already, so we should continue supporting it (cherry picked from commit ec9d705f1a90790bdcb099196269c77d3cc3d53c) (cherry picked from commit 9805b4a183b87976dc33ae80c7e6a209849ba5d7) (cherry picked from commit f00d9237cd9224571e43bda67ad4dddfb009c402)

(cherry picked from commit 92d442d33dd8c81ea73026405d3978d133140807) (cherry picked from commit 7d371d329613c48deef0d8a812c817f2013db8f9)

(cherry picked from commit 549c1f9cd9ab0bfeac4f75fa713cbaae842a6e82)

(cherry picked from commit d0ce27e3f3b6046016800855ad2e57fa67d8b57f) (cherry picked from commit 50a60b6a026e6d6249f069944be86560d87a67ca)

(cherry picked from commit 5a84ca517cef0dacff23f57b360a554b461b5034)

(cherry picked from commit 712ff47cbada7373eaa4fa92bb9220a453c445ae) (cherry picked from commit 52a0e9e39dc006501eb9d8ac0881357900548cf7)

To save start-up time (and memory) this changes the DabBag to not be populated by the webserver on start up - and when a specific dag is asked for it will be loaded on-demand from the SerializedDAG table. Co-Authored-By: Ash Berlin-Taylor <[email protected]>

(cherry picked from commit e91ad24b006823eadd6f3e21fc7cc5c8dd57b0d1)

(cherry picked from commit 3f08d2f986364315c3e43bde3524f12d069392ae)

…ache#6302) (cherry-picked from 1de210b)

ExtraOperatorLinks are supported if Plugins are registered for them (cherry picked from commit 9cb6e28) (cherry picked from commit 72c75860ecfcd1930f1dedc7a0c713f122ea51a5)

(cherry picked from commit e840616) (cherry picked from commit 6d01d8e5bac1b6e829b9da6fc50c1a4b6d23bcaf)

(cherry picked from commit 8a07aee3e5cf133c45ee4ae26aad6104c84502ab)

…ache#5874) cherry-picked from apache@8f6ca53

) (cherry picked from commit 3140c45) (cherry picked from commit df65f8e)

(cherry picked from commit 99a5c2e)

This was incorrectly removed while cherry-picking and resolving conflicts

Zhou Fang and others added 23 commits October 23, 2019 16:27

[AIRFLOW-5088][AIP-24] Add DAG serialization using JSON (apache#5701)

d984217

It implements the method proposed in AIP-24 to serialize DAG. It will be used in DAG persistency in DB to solve webserver scalability issue. (cherry picked from commit 2bd1a51ec75f680a6e6e2101bd948a78421a644a)

[AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver …

a3cfc3e

…scalability (apache#5743)

Make _primitive_types Py2 & Py3 compatible

d16df59

(cherry picked from commit 7ce34b2a959fc1f8322836f38f474a831e4901a1)

Fix issue with different class for Pendulum Timezones

098ea82

(cherry picked from commit c068c67c48d294a589b58be0d0ad8b657c361a77) (cherry picked from commit 04fbf2beac57dcf26b118ebbe5a2bf175ce08af8)

Update timezone class & Do not serialize dates in tasks if they have …

7d53a03

…matching date in DAG (cherry picked from commit be412522cb95a19a51b2f208ae8ebea76e8b667a)

Change type of data column to JSON & Add metric for dagbag size

9b628a2

(cherry picked from commit d030b10bec9cd0e468f36e97e131d497d5a43fc6)

Code Cleanup for JSON columns

690e002

- Code Cleanup for JSON columns - Test code to allow old mysql & sqlite versions (cherry picked from commit 1db8044f9d29edf25f2b8ad4cd21c496c243534a)

Add Debug info

e8003c0

(cherry picked from commit d14497ff28d123d45d626019cabcbd977c5de79d)

Reduce Sizing of SerializedDAG

04af56f

Add specific test for schedule_interval serialization

ed722f2

Delete non-existent Dags

6b70e54

(cherry picked from commit 92d442d33dd8c81ea73026405d3978d133140807) (cherry picked from commit 7d371d329613c48deef0d8a812c817f2013db8f9)

Remove comment

742e8ae

(cherry picked from commit 549c1f9cd9ab0bfeac4f75fa713cbaae842a6e82)

fix bugs that date/time/IntEnum are not supported in serialization.

6710b96

(cherry picked from commit d0ce27e3f3b6046016800855ad2e57fa67d8b57f) (cherry picked from commit 50a60b6a026e6d6249f069944be86560d87a67ca)

Deactivate DAGs instead of deleting if their DAG file is deleted

f7165a9

(cherry picked from commit 5a84ca517cef0dacff23f57b360a554b461b5034)

Fix CI

4728d2f

(cherry picked from commit 712ff47cbada7373eaa4fa92bb9220a453c445ae) (cherry picked from commit 52a0e9e39dc006501eb9d8ac0881357900548cf7)

Fix flake8

9060783

(cherry picked from commit e91ad24b006823eadd6f3e21fc7cc5c8dd57b0d1)

Add default args to decorated_fields

02f85b3

(cherry picked from commit 3f08d2f986364315c3e43bde3524f12d069392ae)

[AIRFLOW-5636] Allow adding or overriding existing Operator Links (ap…

bf94658

…ache#6302) (cherry-picked from 1de210b)

Add support for OperatorLinks

f61ca29

ExtraOperatorLinks are supported if Plugins are registered for them (cherry picked from commit 9cb6e28) (cherry picked from commit 72c75860ecfcd1930f1dedc7a0c713f122ea51a5)

Cleanup

83572c6

(cherry picked from commit e840616) (cherry picked from commit 6d01d8e5bac1b6e829b9da6fc50c1a4b6d23bcaf)

Move serialization directory out of dags folder

6470e2a

(cherry picked from commit 8a07aee3e5cf133c45ee4ae26aad6104c84502ab)

kaxil requested review from ashb and schnie October 23, 2019 16:35

kaxil and others added 5 commits October 23, 2019 19:21

Update path

5d60f5f

[AIRFLOW-5268] Apply same DAG naming conventions as in literature (ap…

0e0037c

…ache#5874) cherry-picked from apache@8f6ca53

[AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (apache#6102

cbd41b4

) (cherry picked from commit 3140c45) (cherry picked from commit df65f8e)

[AIRFLOW-5481] Allow Deleting Renamed DAGs (apache#6101)

0429902

(cherry picked from commit 99a5c2e)

Fix bad merge_conflict resolution

23ea1bc

This was incorrectly removed while cherry-picking and resolving conflicts

kaxil added 3 commits October 24, 2019 17:19

Add test for relativedelta

7deb805

Fix Import

cde0c0a

Backport for Py2

1085f2f

kaxil force-pushed the pr_5992 branch from 1e466c8 to 1085f2f Compare October 24, 2019 21:49

schnie approved these changes Oct 24, 2019

View reviewed changes

schnie merged commit b2cdaa2 into v1-10-5 Oct 24, 2019

schnie deleted the pr_5992 branch October 24, 2019 23:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability #67

[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability #67

kaxil commented Oct 23, 2019

[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability #67

[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability #67

Conversation

kaxil commented Oct 23, 2019

Jira

Description

Tests

Commits

Documentation

Code Quality