Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[serve] Enable streaming over ServeHandles #37114

Merged
merged 16 commits into from
Jul 7, 2023

Conversation

edoakes
Copy link
Contributor

@edoakes edoakes commented Jul 5, 2023

Why are these changes needed?

Adds support for handle.options(stream=True) to make streaming calls using the RayServeHandle.

This required two changes:

  • Adding stream=True to the .options codepaths.
  • Adding a branch in handle_request_streaming to handle calls from the HTTP proxy as well as other handle calls.

I also refactored some of the code around exception handling/metrics into a common context manager that's used by both codepaths.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
@edoakes
Copy link
Contributor Author

edoakes commented Jul 6, 2023

#37147 causes the aiowrlock reader lock to raise an exception. It's currently commented out; once the fix for that issue is in, I'll uncomment and this can be merged.

@edoakes edoakes requested a review from a team July 6, 2023 18:56
@edoakes edoakes changed the title [serve][WIP] Enable streaming over ServeHandles [serve] Enable streaming over ServeHandles Jul 6, 2023
Copy link
Contributor

@GeneDer GeneDer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

python/ray/serve/_private/replica.py Outdated Show resolved Hide resolved
python/ray/serve/_private/replica.py Outdated Show resolved Hide resolved
python/ray/serve/_private/replica.py Show resolved Hide resolved
python/ray/serve/tests/test_metrics.py Show resolved Hide resolved
python/ray/serve/tests/test_util.py Show resolved Hide resolved
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
Signed-off-by: Edward Oakes <[email protected]>
@edoakes edoakes mentioned this pull request Jul 7, 2023
8 tasks
@edoakes edoakes merged commit 95a51ce into ray-project:master Jul 7, 2023
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
Adds support for `handle.options(stream=True)` to make streaming calls using the `RayServeHandle`.

This required two changes:

- Adding `stream=True` to the `.options` codepaths.
- Adding a branch in `handle_request_streaming` to handle calls from the HTTP proxy as well as other handle calls.

I also refactored some of the code around exception handling/metrics into a common context manager that's used by both codepaths.

Signed-off-by: e428265 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants