Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: DAGs are not marked as stale if the dags folder change #41433

Merged
merged 23 commits into from
Aug 28, 2024
Merged

Fix: DAGs are not marked as stale if the dags folder change #41433

merged 23 commits into from
Aug 28, 2024

Conversation

utkarsharma2
Copy link
Contributor

@utkarsharma2 utkarsharma2 commented Aug 13, 2024 •
edited
Loading

Closes: #41432

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Aug 13, 2024
@uranusjr
Copy link
Member

Is a test possible?

@utkarsharma2
Copy link
Contributor Author

Is a test possible?

Yes, I'll add them.

@utkarsharma2
Copy link
Contributor Author

Is a test possible?

@uranusjr, @Lee-W I have added a test, PTAL.

ephraimbuddy
ephraimbuddy reviewed Aug 26, 2024
View reviewed changes
tests/dag_processing/test_job_runner.py Outdated Show resolved Hide resolved
tests/dag_processing/test_job_runner.py Outdated Show resolved Hide resolved
tests/dag_processing/test_job_runner.py Outdated Show resolved Hide resolved
jedcunningham
jedcunningham reviewed Aug 28, 2024
View reviewed changes
airflow/dag_processing/manager.py Outdated Show resolved Hide resolved
airflow/dag_processing/manager.py Outdated Show resolved Hide resolved
@utkarsharma2 utkarsharma2 merged commit 9f30a41 into apache:main Aug 28, 2024
50 checks passed
@utkarsharma2 utkarsharma2 deleted the FixDagParsing branch August 28, 2024 09:07
@ephraimbuddy
Copy link
Contributor

@utkarsharma2 , can you backport this to v2-10-test

@utkarsharma2
Copy link
Contributor Author

Thanks @ephraimbuddy I'll do that.

utkarsharma2 added a commit to astronomer/airflow that referenced this pull request Aug 28, 2024
@utkarsharma2
Fix: DAGs are not marked as stale if the dags folder change (apache#4…
888aff4 
…1433)

* Fix: DAGs are not marked as stale if the AIRFLOW__CORE__DAGS_FOLDER changes

* Update airflow/dag_processing/manager.py

* Add testcase

* Add code comment

* Update code comment

* Update the logic for checking the current dag_directory

* Update testcases

* Remove unwanted code

* Uncomment code

* Add processor_subdir when creating processor_subdir

* Fix test_retry_still_in_executor test

* Remove config from test

* Update airflow/dag_processing/manager.py

Co-authored-by: Jed Cunningham <[email protected]>

* Update if condition for readability

---------

Co-authored-by: Jed Cunningham <[email protected]>
(cherry picked from commit 9f30a41)
@utkarsharma2
Copy link
Contributor Author

Cherry picked - #41829

utkarsharma2 added a commit that referenced this pull request Aug 28, 2024
@utkarsharma2
Fix: DAGs are not marked as stale if the dags folder change (#41433) (#…
996af78 
…41829)

* Fix: DAGs are not marked as stale if the AIRFLOW__CORE__DAGS_FOLDER changes

* Update airflow/dag_processing/manager.py

* Add testcase

* Add code comment

* Update code comment

* Update the logic for checking the current dag_directory

* Update testcases

* Remove unwanted code

* Uncomment code

* Add processor_subdir when creating processor_subdir

* Fix test_retry_still_in_executor test

* Remove config from test

* Update airflow/dag_processing/manager.py

Co-authored-by: Jed Cunningham <[email protected]>

* Update if condition for readability

---------

Co-authored-by: Jed Cunningham <[email protected]>
(cherry picked from commit 9f30a41)
ephraimbuddy added a commit to astronomer/airflow that referenced this pull request Aug 29, 2024
@ephraimbuddy
Revert "Fix: DAGs are not marked as stale if the dags folder change (a…
8953f8b 
…pache#41433)"

This reverts commit 9f30a41.
ephraimbuddy added a commit to astronomer/airflow that referenced this pull request Aug 29, 2024
@ephraimbuddy
Revert "Fix: DAGs are not marked as stale if the dags folder change (a…
60c2fa2 
…pache#41433)"

This reverts commit 9f30a41.
@utkarsharma2 utkarsharma2 added the type:bug-fix Changelog: Bug Fixes label Aug 30, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.10.1 milestone Aug 30, 2024
utkarsharma2 added a commit that referenced this pull request Sep 2, 2024
@utkarsharma2
Fix: DAGs are not marked as stale if the dags folder change (#41433) (#…
1bcf94b 
…41829)

* Fix: DAGs are not marked as stale if the AIRFLOW__CORE__DAGS_FOLDER changes

* Update airflow/dag_processing/manager.py

* Add testcase

* Add code comment

* Update code comment

* Update the logic for checking the current dag_directory

* Update testcases

* Remove unwanted code

* Uncomment code

* Add processor_subdir when creating processor_subdir

* Fix test_retry_still_in_executor test

* Remove config from test

* Update airflow/dag_processing/manager.py

Co-authored-by: Jed Cunningham <[email protected]>

* Update if condition for readability

---------

Co-authored-by: Jed Cunningham <[email protected]>
(cherry picked from commit 9f30a41)
@jcrobak jcrobak mentioned this pull request Sep 9, 2024
Closed
2 tasks
utkarsharma2 added a commit to astronomer/airflow that referenced this pull request Sep 10, 2024
@utkarsharma2
Revert "Fix: DAGs are not marked as stale if the dags folder change (a…
7d905c3 
…pache#41433)"

This reverts commit 9f30a41.
utkarsharma2 added a commit to astronomer/airflow that referenced this pull request Sep 12, 2024
@utkarsharma2
Revert "Fix: DAGs are not marked as stale if the dags folder change (a…
3d432dd 
…pache#41433)"

This reverts commit 9f30a41.
@utkarsharma2 utkarsharma2 mentioned this pull request Sep 12, 2024
Merged
ephraimbuddy pushed a commit that referenced this pull request Sep 12, 2024
@utkarsharma2
Revert "Fix: DAGs are not marked as stale if the dags folder change (#…
3bc2d3a 
…41433)" (#42197)

This reverts commit 9f30a41.
ephraimbuddy added a commit to astronomer/airflow that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Fix: DAGs are not marked as stale if the dags folder change (a…
2bda0a2 
…pache#41433) (apache#41829)"

This reverts commit 996af78.
jedcunningham pushed a commit that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Fix: DAGs are not marked as stale if the dags folder change" …
85c4ef9 
…(41433) (#41829) (#41893) 41829 (#42220)

* Revert "Skip test_scan_stale_dags_when_dag_folder_change in DB isolation mode (#41893)"

This reverts commit 07af14a.

* Revert "Fix: DAGs are not marked as stale if the dags folder change (#41433) (#41829)"

This reverts commit 996af78.
ephraimbuddy added a commit that referenced this pull request Sep 13, 2024
@ephraimbuddy
Revert "Fix: DAGs are not marked as stale if the dags folder change" …
c403e75 
…(41433) (#41829) (#41893) 41829 (#42220)

* Revert "Skip test_scan_stale_dags_when_dag_folder_change in DB isolation mode (#41893)"

This reverts commit 07af14a.

* Revert "Fix: DAGs are not marked as stale if the dags folder change (#41433) (#41829)"

This reverts commit 996af78.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ephraimbuddy ephraimbuddy ephraimbuddy approved these changes

@Lee-W Lee-W Awaiting requested review from Lee-W

@uranusjr uranusjr Awaiting requested review from uranusjr

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

Assignees
No one assigned
Labels
area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes
Projects
None yet
Milestone
Airflow 2.10.1
Development

Successfully merging this pull request may close these issues.

DAGs are not marked as stale if the AIRFLOW__CORE__DAGS_FOLDER changes
5 participants
@utkarsharma2 @uranusjr @ephraimbuddy @Lee-W @jedcunningham