-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a task instance dependency for mapped dependencies #37498
Add a task instance dependency for mapped dependencies #37498
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
4bd1303
to
c2e94c2
Compare
@stevenschaerer can you address the comments and rebase ? |
adbc13a
to
325eace
Compare
@uranusjr Could you please review my latest changes and responses to your comments? Thanks! |
a6b93e7
to
dc90eb3
Compare
dc90eb3
to
ab8da53
Compare
42a694d
to
2f69980
Compare
Co-authored-by: Tzu-ping Chung <[email protected]>
…om the base operator deps
Co-authored-by: Tzu-ping Chung <[email protected]>
86dbbe7
to
e4031b3
Compare
* Add a task instance dependency for mapped dependencies (#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit e2df442)
* Add a task instance dependency for mapped dependencies (#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit e2df442)
* Add a task instance dependency for mapped dependencies (apache#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]>
* Add a task instance dependency for mapped dependencies (apache#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]>
* Add a task instance dependency for mapped dependencies (apache#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]>
* Add a task instance dependency for mapped dependencies (apache#37091) --------- Co-authored-by: Tzu-ping Chung <[email protected]>
This closes #37091.
In particular, dynamically mapped tasks currently show as
failed_upstream
instead ofskipped
if some of the tasks that generate the mapping are skipped and the trigger rule isnone_failed
,all_done
or similar.This is because when trying to expand the mapped task, a failed and skipped mapped dependency behave identically and both result in a NotFullyPopulated being raised.
This PR introduces a new task instance dependency that sets a dynamically mapped task to
skipped
if any of the mapped dependencies are skipped (but not failed) and toupstream_failed
if any of the mapped dependencies are failed. As such, an expansion of the mapped task is never attempted.Before (from #37091):
After: