[AIRFLOW-5088][AIP-24][BackPort] Persisting serialized DAG in DB for webserver scalability

[kaxil]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5088
  • https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler

Description

• Here are some details about my PR, including screenshots of any UI changes:
  *Backport of [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability #5743 for v1-10- branches **
  Based on [AIRFLOW-5088][AIP-24] Add DAG serialization using JSON #5701, this PR implements functionalities including writing serialized DAGs to DB in scheduler, reading DAGs from DB in webserver, controlled by [core] dagcached

The goal is to decouple webserver from the DAG folder, instead it reads everything from database.

Rendering template by functions is an exception, in that case it needs to re-import DAG, because functions are stringified in serialized DAG.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

cc @coufon

[codecov-io]

Codecov Report

Merging #5992 into v1-10-test will increase coverage by 2.57%.
The diff coverage is 79.27%.

@@              Coverage Diff               @@
##           v1-10-test    #5992      +/-   ##
==============================================
+ Coverage       76.79%   79.36%   +2.57%     
==============================================
  Files             511      518       +7     
  Lines           34734    35238     +504     
==============================================
+ Hits            26673    27966    +1293     
+ Misses           8061     7272     -789

Impacted Files	Coverage Δ
airflow/serialization/__init__.py	100% <100%> (ø)
airflow/models/baseoperator.py	94.77% <100%> (+0.28%)	⬆️
airflow/settings.py	85.71% <100%> (+0.15%)	⬆️
airflow/models/__init__.py	100% <100%> (ø)	⬆️
airflow/serialization/enums.py	100% <100%> (ø)
airflow/utils/dag_processing.py	58.69% <16.66%> (-0.25%)	⬇️
airflow/www/views.py	45.01% <27.27%> (-0.1%)	⬇️
airflow/models/dagbag.py	84.95% <34.37%> (-7.21%)	⬇️
airflow/www/utils.py	82.65% <60%> (-0.69%)	⬇️
airflow/models/dag.py	92.07% <80%> (-0.91%)	⬇️
... and 61 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ea227f...afa60a5. Read the comment docs.

[kaxil]

ToDo:

• Graph View & Tree View are broken currently because [AIRFLOW-5268] Apply same DAG naming conventions as in literature #5874 has not been merged to v1-10-* branch yet (The Graph View & Tree View uses _upstream_tasks to create an edge between tasks), so either we need to include _upstream_tasks attribute to the serialized JSON or we need to backport that PR in v1-10-* branches. - Backported that PR in 9b5fa1abb3970213c0ba68a0b212fc6dfdbad4db

[potiuk]

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[tooptoop4]

cfg points to broken link https://airflow.apache.org/howto/enable-dag-serialization.html

[kaxil]

Thanks, will update it.

…

On Tue, Dec 24, 2019, 16:09 tooptoop4 ***@***.***> wrote: cfg points to broken link https://airflow.apache.org/howto/enable-dag-serialization.html — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#5992?email_source=notifications&email_token=ACDHIJVO3KTRORZ5F2W7I2LQ2IX2LA5CNFSM4ITE67U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTMPEY#issuecomment-568772499>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDHIJRIBSDQATOXBM3CJ3LQ2IX2LANCNFSM4ITE67UQ> .

[kaxil]

Correct Link in case you are looking for it is: https://airflow.apache.org/docs/1.10.7/dag-serialization.html

…

On Tue, Dec 24, 2019 at 4:10 PM Kaxil Naik ***@***.***> wrote: Thanks, will update it. On Tue, Dec 24, 2019, 16:09 tooptoop4 ***@***.***> wrote: > cfg points to broken link > https://airflow.apache.org/howto/enable-dag-serialization.html > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <#5992?email_source=notifications&email_token=ACDHIJVO3KTRORZ5F2W7I2LQ2IX2LA5CNFSM4ITE67U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTMPEY#issuecomment-568772499>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACDHIJRIBSDQATOXBM3CJ3LQ2IX2LANCNFSM4ITE67UQ> > . >

[tooptoop4]

for a .py that generates multi dynamic dags how can parsing the dag be avoided? @kaxil