-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-5889] Fix polling for AWS Batch job status #6765
Conversation
7c29b4d
to
7f65c55
Compare
Codecov Report
@@ Coverage Diff @@
## master #6765 +/- ##
=========================================
- Coverage 84.32% 84.3% -0.02%
=========================================
Files 672 672
Lines 38179 38210 +31
=========================================
+ Hits 32195 32214 +19
- Misses 5984 5996 +12
Continue to review full report at Codecov.
|
The bug fix in this PR should get into a 1.x release, so it might be better if this PR sneaks in before the AIP-21 stuff in #6764 is resolved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - one small change requested, then we can merge this.
- errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files}
7f65c55
to
c52463e
Compare
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
…6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types (NOT INCLUDED IN THE CHERRY PICK) - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files} (cherry picked from commit 479ee63)
…pache#6765) - errors in polling for job status should not fail the airflow task when the polling hits an API throttle limit; polling should detect those cases and retry a few times to get the job status, only failing the task when the job description cannot be retrieved - added typing for the BatchProtocol method return types, based on the botocore.client.Batch types - applied trivial format consistency using black, i.e. $ black -t py36 -l 96 {files}
PR target
This PR targets a 1.10.x release for this bug fix
1.10.test
branch instead ofmaster
?Jira
Possible conflicts
Description
the airflow task when the polling hits an API throttle
limit; polling should detect those cases and retry a
few times to get the job status
types, based on the botocore.client.Batch types
$ black -t py36 -l 96 {files}
Tests
Commits
Documentation