Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: extend CLI NTSC timeout [MLG-870] #8632

Merged
merged 2 commits into from
Jan 3, 2024

Conversation

azhou-determined
Copy link
Contributor

Description

extend timeout of NTSC CLI commands. 5 min is too short since tasks can be in a queued/starting state before they are considered ready.

Test Plan

Commentary (optional)

Checklist

  • Changes have been manually QA'd
  • User-facing API changes need the "User-facing API Change" label.
  • Release notes should be added as a separate file under docs/release-notes/.
    See Release Note for details.
  • Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

@@ -20,6 +20,9 @@ class PageOpts(str, enum.Enum):
# Default max number of times to retry a request.
MAX_RETRIES = 5

# Seconds to wait for an NTSC task to become ready before timeout.
NTSC_TIMEOUT = 60 * 30
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bumping to 30 min, though 15 might be enough? not sure if there's any harm though, this is only being used by the CLI and i assume users would kill the process after some time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my only worry is that we have a few tests which use api.task_is_ready for test purposes. these may start taking too much time to fail, and for them we'll probably want to have task_is_ready accept the timeout as a parameter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, updated

@@ -20,6 +20,9 @@ class PageOpts(str, enum.Enum):
# Default max number of times to retry a request.
MAX_RETRIES = 5

# Seconds to wait for an NTSC task to become ready before timeout.
NTSC_TIMEOUT = 60 * 30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my only worry is that we have a few tests which use api.task_is_ready for test purposes. these may start taking too much time to fail, and for them we'll probably want to have task_is_ready accept the timeout as a parameter.

@azhou-determined azhou-determined merged commit 001d827 into determined-ai:main Jan 3, 2024
55 of 61 checks passed
@dannysauer dannysauer modified the milestone: 0.27.1 Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants