Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove deprecated SubDags #41390

Merged
merged 4 commits into from
Aug 13, 2024
Merged

Remove deprecated SubDags #41390

merged 4 commits into from
Aug 13, 2024

Conversation

kaxil
Copy link
Member

@kaxil kaxil commented Aug 11, 2024

This PR removes SubDags in favor of TaskGroups fro Airflow 3.0

Subdags have been removed from the following locations:

  • CLI
  • API
  • SubDagOperator

This removal marks the end of Subdag support across all interfaces. Users should transition to using TaskGroups as a more efficient and maintainable alternative.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@kaxil kaxil added the airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes label Aug 11, 2024
@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:core-operators Operators, Sensors and hooks within Core Airflow area:db-migrations PRs with DB migration area:providers area:Scheduler including HA (high availability) scheduler area:serialization provider:celery provider:cncf-kubernetes Kubernetes provider related issues provider:fab provider:google Google (including GCP) related issues labels Aug 11, 2024
@kaxil kaxil added this to the Airflow 3.0.0 milestone Aug 11, 2024
@kaxil kaxil force-pushed the remove-subdags branch 2 times, most recently from 64046c6 to 13118d0 Compare August 11, 2024 23:48
@kaxil kaxil force-pushed the remove-subdags branch 2 times, most recently from e535d08 to 903b20c Compare August 12, 2024 00:03
@kaxil kaxil requested review from eladkal and uranusjr August 13, 2024 07:08
kaxil and others added 3 commits August 13, 2024 11:18
This PR removes SubDags in favor of TaskGroups fro Airflow 3.0

Subdags have been removed from the following locations:

- CLI
- API
- ``SubDagOperator``

This removal marks the end of Subdag support across all interfaces. Users
should transition to using TaskGroups as a more efficient and maintainable
alternative.
Copy link
Contributor

@ephraimbuddy ephraimbuddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concerning the root_dag_id, I guess security/permissions.py should also be updated but then, I think Airflow 3 might no longer be compatible with current providers.

@kaxil
Copy link
Member Author

kaxil commented Aug 13, 2024

Concerning the root_dag_id, I guess security/permissions.py should also be updated but then, I think Airflow 3 might no longer be compatible with current providers.

Yeah, that is mainly to be handled on the Auth provider implementation rather than core Airflow so it works for Airflow 2 & Airflow 3

Copy link
Contributor

@phanikumv phanikumv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll also need to remove docs/apache-airflow/img/subdag_after.png , docs/apache-airflow/img/subdag_before.png and docs/apache-airflow/img/subdag_zoom.png

@kaxil
Copy link
Member Author

kaxil commented Aug 13, 2024

We'll also need to remove docs/apache-airflow/img/subdag_after.png , docs/apache-airflow/img/subdag_before.png and docs/apache-airflow/img/subdag_zoom.png

Done, removed

@jscheffl
Copy link
Contributor

Oh, in numer of LoC removed as PR this is a high chance to get PR of the month :-D

@kaxil kaxil merged commit 6570c6d into apache:main Aug 13, 2024
78 of 79 checks passed
@kaxil kaxil deleted the remove-subdags branch August 13, 2024 20:34
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So much complexity removed ... Good job.

@potiuk
Copy link
Member

potiuk commented Aug 13, 2024

BTW. One more comment #21867 -> I believe retry for failed task group is the only serious missing feature for Task Groups comparing to SubDags.

Retry for Task Groups: #21867

We should likely consider implementing it.

@kaxil
Copy link
Member Author

kaxil commented Aug 13, 2024

BTW. One more comment #21867 -> I believe retry for failed task group is the only serious missing feature for Task Groups comparing to SubDags.

Retry for Task Groups: #21867

We should likely consider implementing it.

Yup, added a label to that issue to track it

Artuz37 pushed a commit to Artuz37/airflow that referenced this pull request Aug 19, 2024
This PR removes SubDags in favor of TaskGroups fro Airflow 3.0

Subdags have been removed from the following locations:

- CLI
- API
- ``SubDagOperator``

This removal marks the end of Subdag support across all interfaces. Users
should transition to using TaskGroups as a more efficient and maintainable
alternative.

---------
Co-authored-by: Brent Bovenzi <[email protected]>
molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024
This PR removes SubDags in favor of TaskGroups fro Airflow 3.0

Subdags have been removed from the following locations:

- CLI
- API
- ``SubDagOperator``

This removal marks the end of Subdag support across all interfaces. Users
should transition to using TaskGroups as a more efficient and maintainable
alternative.

---------
Co-authored-by: Brent Bovenzi <[email protected]>
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Aug 20, 2024
This PR removes SubDags in favor of TaskGroups fro Airflow 3.0

Subdags have been removed from the following locations:

- CLI
- API
- ``SubDagOperator``

This removal marks the end of Subdag support across all interfaces. Users
should transition to using TaskGroups as a more efficient and maintainable
alternative.

---------
Co-authored-by: Brent Bovenzi <[email protected]>
kaxil added a commit to astronomer/airflow that referenced this pull request Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:CLI area:core-operators Operators, Sensors and hooks within Core Airflow area:db-migrations PRs with DB migration area:providers area:Scheduler including HA (high availability) scheduler area:serialization provider:celery provider:cncf-kubernetes Kubernetes provider related issues provider:fab provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants