-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add healthcheck support for JetStream #90
Conversation
@vivianrwu could you add a description about why we are adding this PR? thanks! (as well as the test steps |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! After fixing the CI checks, we are good to go!
"LLM orchestrator is being used in offline test mode, and will not" | ||
" respond to gRPC queries - only direct function calls." | ||
) | ||
is_live = self._driver.live |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you share where driver set live status?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for sharing it!
This PR adds the healthcheck support endpoint for JetStream.
This covers checking the model server is alive based on the
self.live
field here - https://github.com/google/JetStream/blob/e19a7906d8cdf1cae658a4c7c4f6f516aade49f9/jetstream/core/orchestrator.py#L377The healthcheck endpoint can be invoked via the following:
Test coverage can be found under
test_server.py
once the server is started.This functionality will be useful for model server liveness and readiness checks.