Create DAG-level cluster policy #12184
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
turbaszek
force-pushed
the
dag-cluster-policy
branch
from
c28fc42
to
ac07d43
Compare
potiuk
requested changes
Nov 8, 2020
| for task in dag.tasks: | ||
| settings.policy(task) | ||
| settings.task_policy(task) |
There was a problem hiding this comment.
Here we should check if 'policy' is defined and run it as well but with deprecation warning.
There was a problem hiding this comment.
Done, this is what users will see:
root@817b1471dabe:/opt/airflow# airflow scheduler
/opt/airflow/airflow/models/dag.py:61: DeprecationWarning: Using `policy` in airflow_local_settings.py is deprecated. Please rename your `policy` to `task_policy`.
from airflow.models.dagbag import DagBag
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-11-08 19:09:09,005] {scheduler_job.py:1248} INFO - Starting the scheduler
[2020-11-08 19:09:09,005] {scheduler_job.py:1253} INFO - Processing each file at most -1 times
[2020-11-08 19:09:09,114] {scheduler_job.py:1275} INFO - Resetting orphaned tasks for active dag runs
kaxil
reviewed
Nov 9, 2020
kaxil
reviewed
Nov 9, 2020
turbaszek
force-pushed
the
dag-cluster-policy
branch
from
8eccdf9
to
0814119
Compare
kaxil
approved these changes
Nov 9, 2020
mik-laj
approved these changes
Nov 11, 2020
potiuk
approved these changes
Nov 12, 2020
|
The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it! |
github-actions
bot
added
the
full tests needed
turbaszek
force-pushed
the
dag-cluster-policy
branch
from
0814119
to
0c3ef5e
Compare
ashb
requested changes
Nov 12, 2020
airflow/models/dagbag.py
Outdated
Comment on lines
50
to
58
| # TODO: Remove once deprecated | ||
| if hasattr(settings, "policy"): | ||
| warnings.warn( | ||
| "Using `policy` in airflow_local_settings.py is deprecated. " | ||
| "Please rename your `policy` to `task_policy`.", | ||
| DeprecationWarning, | ||
| stacklevel=2, | ||
| ) | ||
| setattr(settings, "task_policy", settings.policy) # pylint: disable=no-member |
There was a problem hiding this comment.
This should be done in airflow.settings.import_local_settings function please.
There was a problem hiding this comment.
That was my intial idea but somehow I decided to do this in dagbag 🤷♂️ Done in a250f41
There was a problem hiding this comment.
I'm just conscious that DagBag may go away at some point soon -- it does a lot less than it used to.
Also top-level code is a bit of a smell to me.
github-actions
bot
removed
the
full tests needed
ashb
reviewed
Nov 12, 2020
ashb
approved these changes
Nov 12, 2020
github-actions
bot
added
the
full tests needed
|
The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it! |
turbaszek
force-pushed
the
dag-cluster-policy
branch
from
890b0dc
to
aac1262
Compare
This commit adds new concept of dag_policy which is checked once for every DAG when creating DagBag. It also improves documentation around cluster policies. closes: apache#12179
Co-authored-by: Kaxil Naik <[email protected]>
Co-authored-by: Ash Berlin-Taylor <[email protected]>
turbaszek
force-pushed
the
dag-cluster-policy
branch
from
aac1262
to
340c130
Compare
kaxil
approved these changes
Nov 13, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds new concept of dag_policy which is checked
once for every DAG when creating DagBag. It also improves
documentation around cluster policies.
closes: #12179
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.