[AIRFLOW-5210] Make finding template files more efficient #5815
Conversation
For large DAGs, iterating over template fields to find template files can be time intensive. Save this time for tasks that do not specify a template file extension.
| content[i] = env.loader.get_source(env, content[i])[0] | ||
| except Exception as e: | ||
| self.log.exception(e) | ||
| if self.template_ext: |
There was a problem hiding this comment.
Wouldn't this stop rendering of template fields ?
There was a problem hiding this comment.
The way I read it, this method iterates over all template_fields and if it finds a field value with an extension listed in template_exts, runs env.loader.get_source(...) and sets the result on the specific field.
It indeed scans over all template_fields even if template_exts is empty, which doesn't make sense so I think this is a valid change.
|
@danfrankj LGTM, the k8s CI step failed. I believe it's a bit buggy so restarted it. |
|
Thanks @danfrankj |
|
You're welcome! Thank you guys. Happy DAG'n |
|
@BasPH was something wrong with this PR? - I'm seeing a message above about a revert |
|
No, my bad, I accidentally clicked the revert button. I deleted the "revert" branch. |
For large DAGs, iterating over template fields to find template files can be time intensive. Save this time for tasks that do not specify a template file extension. (cherry picked from commit eeac823)
For large DAGs, iterating over template fields to find template files can be time intensive. Save this time for tasks that do not specify a template file extension. (cherry picked from commit eeac823)
Jira
https://issues.apache.org/jira/browse/AIRFLOW-5210
Description
For large DAGs (1000s of tasks) iterating over template_fields adds significant time to task execution and is not necessary for tasks that do not specify template_ext
Tests
I can add tests if you think necessary, but this is a very simple change
Code Quality
flake8