-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-6730] Use total_seconds instead of seconds #7363
Conversation
4913502
to
b13af3b
Compare
🤦♂️ Guess we've just been lucky and the default timeouts don't fire. |
@@ -94,7 +94,7 @@ def convert_type(self, value, schema_type): | |||
return datetime.timedelta( | |||
hours=formated_time.tm_hour, | |||
minutes=formated_time.tm_min, | |||
seconds=formated_time.tm_sec).seconds | |||
seconds=formated_time.tm_sec).total_seconds() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is I think wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In [7]: timedelta(hours=12, minutes=30, seconds=30).seconds
Out[7]: 45030
In [8]: timedelta(hours=12, minutes=30, seconds=30).total_seconds()
Out[8]: 45030.0
(total_seconds returns a float, previously this was an int. I don't think that is intended.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One change that is wrong
Ah - the "native" representation of a timedelta is (days, seconds) so it's only giving the wrong value when the delta is >= 1 day. So nothing is likely to break as a result of this, but using |
@@ -104,7 +104,7 @@ | |||
"autoscaling_config": {"policy_uri": "autoscaling_policy"}, | |||
"config_bucket": "storage_bucket", | |||
"initialization_actions": [ | |||
{"executable_file": "init_actions_uris", "execution_timeout": "600s"} | |||
{"executable_file": "init_actions_uris", "execution_timeout": "600.0s"} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is my only question -- does DataProc accept this, or does it only accept whole-integer seconds here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the python typing indicates float, but I think that is just within the airflow project.
I will just cast it to int since that is guaranteed to not break anything
37845fa
to
68d9316
Compare
68d9316
to
122d74b
Compare
This passed on my travis-ci so i will merge |
* [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test
* [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4ba)
* [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4ba)
* [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4ba)
* [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4ba)
Issue link: AIRFLOW-6730
Make sure to mark the boxes below before creating PR: [x]
[AIRFLOW-NNNN]
. AIRFLOW-NNNN = JIRA ID** For document-only changes commit message can start with
[AIRFLOW-XXXX]
.In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.