-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make the default TI pool slots '1' #8153
Conversation
This is caused by the new `add pool_slots field to task_instance` database migration, which adds a new `pool_slots` database column, which is nullable (hence it is filled with NULLs after the migration).
I have tested this manually on my machine. I installed 1.10.9 ran a few examples DAGs, upgraded to 1.10.10rc2, Visited old Task Instance details of old tasks and was able to see the error, also could see the error in Scheduler if the TIs were still queued before I upgraded to 1.10.10rc2 and applied migrations. I applied the fix in the PR and ran the Webserver and scheduler again and both ran fine and I was able to see Task Instance details. Note: I am not updating all the Null values in TI table for old records as this table can be pretty huge which was also the intention in the original PR: #7160 to keep old behaviour and let the column be Nullable. |
Codecov Report
@@ Coverage Diff @@
## master #8153 +/- ##
==========================================
- Coverage 66.39% 60.55% -5.84%
==========================================
Files 935 935
Lines 45170 45170
==========================================
- Hits 29990 27353 -2637
- Misses 15180 17817 +2637
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @kaxil , quick response, code looking good. One more thing I am sure. Is it the new TI object column pool_slots
with default value 1
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the right fix, or is changing the migration to have a default of 1 better?
Yes |
@ashb I didn't update the migration code as the TI table can be large for many users (which the original PR intended #7160 ). I was thinking we could do that as part of Airflow 2.0 to update NULLs to 1 and make the column not nullable. WDYT? |
Oh yeah, adding a new column with non null default would be a table-locking migration, good point. |
(cherry picked from commit 323c1d1)
This is caused by the new
add pool_slots field to task_instance
database migration, which adds a new
pool_slots
database column,which is nullable (hence it is filled with NULLs after the migration).
This Bug was introduced in #7160 and is a blocker for 1.10.10
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.