-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-5870] Allow -1 for infinite pool size #6520
[AIRFLOW-5870] Allow -1 for infinite pool size #6520
Conversation
147abe7
to
da70f58
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🙏
0e330a9
to
2b774c7
Compare
Codecov Report
@@ Coverage Diff @@
## master #6520 +/- ##
==========================================
- Coverage 84.08% 83.76% -0.32%
==========================================
Files 635 635
Lines 36849 36851 +2
==========================================
- Hits 30983 30867 -116
- Misses 5866 5984 +118
Continue to review full report at Codecov.
|
2b774c7
to
5ceb0ff
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice optimisation. I will also cherry-pick it to 1.10.*. However I will update the description so that it's clear it also optimises DB access.
(cherry-picked from commit 4a12041)
(cherry-picked from commit 4a12041)
(cherry-picked from commit 4a12041)
Make sure you have checked all steps below.
Jira
Description
Adds the ability to create pools with size=-1, allowing infinite task usage, and allowing the used_slots to return without a db query.
To contextualize this change, we saw spiky DB queries since this has to query all RUNNING task instances, and each task instance that starts running needs to run this query, leading to an n^2 problem.
Tests
Commits
Documentation