[AIRFLOW-6871] optimize tree view for large DAGs #7492

houqp · 2020-02-21T21:54:29Z

This change reduces page size by more than 10X and reduces page load time by 3-5X. As a result, the tree view can now load large DAGs that were causing 5XX errors before.

Another example: one of our DAGs' tree view had a page size of 200MB and took 1 minute to load. With this patch, it now loads within 24s with a page size of 17MB.

List of optimizations applied to the view handler:

only seralize used task instance attributes to json instead of the
whole ORM object
encode task instance attributes as array instead of dict
encode datetime in unix timestamp instead of iso formmat string
push task instance attribute construction into client side JS
remove redundant task instance attributes
simplify reduce_nodes() logic, remove unnecessary if statements
seralize JSON as string to be used with JSON.parse on the client side
to speed up browser JS parse time
remove spaces in seralized JSON string to reduce payload size

Issue link: AIRFLOW-6871

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

codecov-io · 2020-02-22T05:34:25Z

Codecov Report

Merging #7492 into master will increase coverage by 0.08%.
The diff coverage is 84.21%.

@@            Coverage Diff             @@
##           master    #7492      +/-   ##
==========================================
+ Coverage   86.76%   86.85%   +0.08%     
==========================================
  Files         896      896              
  Lines       42649    42663      +14     
==========================================
+ Hits        37005    37055      +50     
+ Misses       5644     5608      -36

Impacted Files	Coverage Δ
airflow/www/views.py	`76.19% <84.21%> (-0.05%)`	⬇️
airflow/jobs/scheduler_job.py	`90.07% <0%> (+0.43%)`	⬆️
airflow/utils/sqlalchemy.py	`84.93% <0%> (+1.36%)`	⬆️
airflow/hooks/dbapi_hook.py	`91.73% <0%> (+1.65%)`	⬆️
airflow/providers/postgres/hooks/postgres.py	`94.36% <0%> (+16.9%)`	⬆️
...roviders/google/cloud/operators/postgres_to_gcs.py	`85.29% <0%> (+32.35%)`	⬆️
airflow/providers/postgres/operators/postgres.py	`100% <0%> (+50%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3111406...adf0c6d. Read the comment docs.

airflow/www/templates/airflow/tree.html

airflow/www/views.py

ashb · 2020-02-22T09:23:32Z

Can you expand on "seralize JSON as string to be used with JSON.parse on the client side
to speed up browser JS parse time"? I don't see how doing more can make it faster

houqp · 2020-02-23T23:15:27Z

@ashb a quick analysis can be found at: https://developpaper.com/json-parse-is-faster-than-object-literal-syntax/.

This optimization only makes sense for very large JSON payload, which is the case for tree.html. For the large DAGs that we have, it reduces page load time by couple hundred milliseconds.

airflow/www/templates/airflow/tree.html

airflow/www/views.py

houqp · 2020-02-24T22:55:16Z

@ashb all builds are passing now, ready for another round of review.

ashb · 2020-02-25T22:12:31Z

airflow/www/views.py

+            if task.depends_on_past:
+                node['depends_on_past'] = task.depends_on_past
+            if task.start_date:
+                # round to seconds to reduce payload size


How much does it reduce it by? Is stripping of all ms noticable? (Could we perhaps limit to 3 or 6 sig. fig.?)

not much for task node since we don't have too many of them, that's why i didn't add the rounding in the first place here. It did make a big difference for task instance node since we have lots of them, IIRC, probably around 10-20% size reduction.

I can change it to round to 3 sig. fig. everywhere to see what the performance implication would be.

@ashb round to 3 sig. fig. increases the overall payload size by 15%. The question now is do we care about millisecond accuracy for task start/end time enough to take this 15% performance hit?

So far, I found second granularity has been good enough for us, but I might be missing other use-cases.

Probably not needed.

Is it worth making it a config option do you think?

This code path has a very hot function call loop that's very sensitive to if statements. For the large DAG that we have, adding one extra if statement increases the response time by more than 400ms. That's why simplify reduce_nodes() logic, remove unnecessary if statements is in the optimization list :)

That and based on the understanding that we are rewriting Airflow web into a proper SPA, I think it's best not to introduce a config for this change. I would prefer us giving round to second a try and come back to add more sig. fig. or add a config later on if any real use-case comes up. It's better to not engineer solutions when we don't have a good use-case in mind.

If you are really concerned about the precision, we can perhaps change it to round to 1 sig. fig. for a 7% performance hit. I can't really think of a case where knowing 0.01 second difference of runtime is important.

Good reasoning. I'll copy some/most of this in to the commit message for future-proofing.

Leaving this unresolved as a reminder to myself.

airflow/www/templates/airflow/tree.html

airflow/www/views.py

This change reduces page size by more than 10X and reduces page load time by 3-5X. As a result, the tree view can now load large DAGs that were causing 5XX error without the patch. List of optimizations applied to the view handler: * only seralize used task instance attributes to json instead of the whole ORM object * encode task instance attributes as array instead of dict * encode datetime in unix timestamp instead of iso formmat string * push task instance attribute construction into client side JS * remove redundant task instance attributes * simplify reduce_nodes() logic, remove unnecessary if statements * seralize JSON as string to be used with JSON.parse on the client side to speed up browser JS parse time * remove spaces in seralized JSON string to reduce payload size

houqp · 2020-02-28T16:55:35Z

@ashb updated if statement and CI is passing now :)

This change reduces page size by more than 10X and reduces page load time by 3-5X. As a result, the tree view can now load large DAGs that were causing 5XX error without the patch. List of optimizations applied to the view handler: * only seralize used task instance attributes to json instead of the whole ORM object * encode task instance attributes as array instead of dict * encode datetime in unix timestamp instead of iso formmat string * push task instance attribute construction into client side JS * remove redundant task instance attributes * simplify reduce_nodes() logic, remove unnecessary if statements * seralize JSON as string to be used with JSON.parse on the client side to speed up browser JS parse time * remove spaces in seralized JSON string to reduce payload size

This change reduces page size by more than 10X and reduces page load time by 3-5X. As a result, the tree view can now load large DAGs that were causing 5XX error without the patch. List of optimizations applied to the view handler: * only seralize used task instance attributes to json instead of the whole ORM object * encode task instance attributes as array instead of dict * encode datetime in unix timestamp instead of iso formmat string * push task instance attribute construction into client side JS * remove redundant task instance attributes * simplify reduce_nodes() logic, remove unnecessary if statements * seralize JSON as string to be used with JSON.parse on the client side to speed up browser JS parse time * remove spaces in seralized JSON string to reduce payload size Co-Authored-By: QP Hou <[email protected]> (cherry-picked from c1c2d6a)

boring-cyborg bot added the area:webserver Webserver related Issues label Feb 21, 2020

houqp requested a review from ashb February 21, 2020 21:54

houqp force-pushed the tree_optimized branch 6 times, most recently from 83548cb to 41c65e1 Compare February 22, 2020 04:51

robinedwards reviewed Feb 22, 2020

View reviewed changes

airflow/www/templates/airflow/tree.html Outdated Show resolved Hide resolved

houqp force-pushed the tree_optimized branch from 41c65e1 to 84998e0 Compare February 22, 2020 08:20

ashb requested changes Feb 22, 2020

View reviewed changes

airflow/www/templates/airflow/tree.html Outdated Show resolved Hide resolved

airflow/www/templates/airflow/tree.html Outdated Show resolved Hide resolved

airflow/www/views.py Outdated Show resolved Hide resolved

houqp force-pushed the tree_optimized branch from 84998e0 to 0e4e965 Compare February 23, 2020 23:11

houqp requested a review from ashb February 23, 2020 23:15

houqp force-pushed the tree_optimized branch 3 times, most recently from ebcd4e5 to 2697fda Compare February 24, 2020 00:48

ashb reviewed Feb 24, 2020

View reviewed changes

airflow/www/templates/airflow/tree.html Outdated Show resolved Hide resolved

ashb reviewed Feb 24, 2020

View reviewed changes

airflow/www/views.py Outdated Show resolved Hide resolved

ashb reviewed Feb 24, 2020

View reviewed changes

airflow/www/views.py Show resolved Hide resolved

ashb reviewed Feb 24, 2020

View reviewed changes

airflow/www/views.py Outdated Show resolved Hide resolved

houqp force-pushed the tree_optimized branch from 2697fda to 9912faa Compare February 24, 2020 16:05

houqp requested a review from ashb February 24, 2020 16:10

houqp force-pushed the tree_optimized branch 2 times, most recently from 035855f to 67766d2 Compare February 24, 2020 20:42

ashb reviewed Feb 25, 2020

View reviewed changes

houqp force-pushed the tree_optimized branch from 67766d2 to c47964a Compare February 25, 2020 22:14

ashb reviewed Feb 25, 2020

View reviewed changes

airflow/www/templates/airflow/tree.html Outdated Show resolved Hide resolved

houqp requested a review from ashb February 25, 2020 22:44

houqp force-pushed the tree_optimized branch from c47964a to adf0c6d Compare February 25, 2020 22:59

ashb reviewed Feb 27, 2020

View reviewed changes

airflow/www/views.py Outdated Show resolved Hide resolved

houqp force-pushed the tree_optimized branch 4 times, most recently from 916dc8e to cb933e3 Compare February 28, 2020 07:33

houqp force-pushed the tree_optimized branch from cb933e3 to f57038e Compare February 28, 2020 16:00

houqp requested a review from ashb February 28, 2020 16:55

ashb approved these changes Feb 28, 2020

View reviewed changes

ashb merged commit c1c2d6a into apache:master Feb 28, 2020

houqp deleted the tree_optimized branch February 28, 2020 23:01

mik-laj mentioned this pull request Mar 20, 2020

webgui cannot display all tasks #7788

Closed

mik-laj added the area:performance label Apr 9, 2020

qzheng1989 mentioned this pull request Apr 22, 2020

Json Parse Error in Tree View #8523

Closed

ashb mentioned this pull request Jan 7, 2021

Durations less than 1 second are not visible in the TI tooltops #13537

Merged

ashb mentioned this pull request Apr 7, 2022

Remove extraneous json parsing in Calendar and Grid #22835

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-6871] optimize tree view for large DAGs #7492

[AIRFLOW-6871] optimize tree view for large DAGs #7492

houqp commented Feb 21, 2020 •

edited

Loading

codecov-io commented Feb 22, 2020 •

edited

Loading

ashb commented Feb 22, 2020

houqp commented Feb 23, 2020

houqp commented Feb 24, 2020

ashb Feb 25, 2020

houqp Feb 25, 2020

houqp Feb 25, 2020

ashb Feb 25, 2020

houqp Feb 27, 2020

ashb Feb 27, 2020

ashb Feb 27, 2020

houqp commented Feb 28, 2020

[AIRFLOW-6871] optimize tree view for large DAGs #7492

[AIRFLOW-6871] optimize tree view for large DAGs #7492

Conversation

houqp commented Feb 21, 2020 • edited Loading

codecov-io commented Feb 22, 2020 • edited Loading

Codecov Report

ashb commented Feb 22, 2020

houqp commented Feb 23, 2020

houqp commented Feb 24, 2020

ashb Feb 25, 2020

Choose a reason for hiding this comment

houqp Feb 25, 2020

Choose a reason for hiding this comment

houqp Feb 25, 2020

Choose a reason for hiding this comment

ashb Feb 25, 2020

Choose a reason for hiding this comment

houqp Feb 27, 2020

Choose a reason for hiding this comment

ashb Feb 27, 2020

Choose a reason for hiding this comment

ashb Feb 27, 2020

Choose a reason for hiding this comment

houqp commented Feb 28, 2020

houqp commented Feb 21, 2020 •

edited

Loading

codecov-io commented Feb 22, 2020 •

edited

Loading