[serve] Add experimental support for StreamingResponse using RayObjectRefGenerator
#35720
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
This reverts commit 122b705. Signed-off-by: SangBin Cho <[email protected]>
This reverts commit 05f468a. Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
Signed-off-by: SangBin Cho <[email protected]>
angelinalg
reviewed
May 25, 2023
angelinalg
reviewed
May 25, 2023
python/ray/serve/_private/replica.py
Outdated
| never_set_event = asyncio.Event() | ||
| await never_set_event.wait() | ||
| If the result is already a Response type, it will be sent directly. Else it | ||
| will be converted to our custom Response type that handles serialization for |
There was a problem hiding this comment.
Suggested change
| will be converted to our custom Response type that handles serialization for | |
| is converted to a custom Response type that handles serialization for |
angelinalg
reviewed
May 25, 2023
angelinalg
reviewed
May 25, 2023
angelinalg
reviewed
May 25, 2023
angelinalg
reviewed
May 25, 2023
Signed-off-by: Edward Oakes <[email protected]>
angelinalg
reviewed
May 25, 2023
|
Tip: Use the |
angelinalg
approved these changes
May 25, 2023
Signed-off-by: Edward Oakes <[email protected]>
Oh wow that is a revelation 😂 |
9 tasks
8 tasks
edoakes
added a commit
to edoakes/ray
that referenced
this pull request
May 26, 2023
…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779 Signed-off-by: Edward Oakes <[email protected]>
ArturNiederfahrenhorst
pushed a commit
that referenced
this pull request
May 26, 2023
…jectRefGenerator` (#35720) (#35811) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on #35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: #35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: #35778 - The timeout set in the HTTP proxy does not apply to streaming responses: #35779 Signed-off-by: Edward Oakes <[email protected]>
8 tasks
edoakes
added a commit
that referenced
this pull request
Jun 12, 2023
Extends #35720 to also add support for websockets (as well as detecting HTTP disconnects).
scv119
pushed a commit
to scv119/ray
that referenced
this pull request
Jun 16, 2023
…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…jectRefGenerator` (ray-project#35720) Adds experimental support for using `StreamingResponse`s to stream intermediate results back to the client. This is currently gated behind a feature flag (must set `RAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1`. This is implemented by using the Ray `ObjectRefStreamingGenerator` interface. When the feature flag is on, the HTTP proxy will use `num_returns="streaming"` for _all_ calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy. Known limitations & follow-ups (to be addressed before a non-experimental release): - Minor performance regression due to an extra RPC from streaming protocol (see the microbenchmark results posted on ray-project#35468). Most of this should be able to be optimized away before turning this on by default. - Streaming is not yet possible using the `ServeHandle` interface: ray-project#35777 - `max_concurrent_queries` is not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: ray-project#35778 - The timeout set in the HTTP proxy does not apply to streaming responses: ray-project#35779 Signed-off-by: e428265 <[email protected]>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
Extends ray-project#35720 to also add support for websockets (as well as detecting HTTP disconnects). Signed-off-by: e428265 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Adds experimental support for using
StreamingResponses to stream intermediate results back to the client. This is currently gated behind a feature flag (must setRAY_SERVE_ENABLE_EXPERIMENTAL_STREAMING=1.This is implemented by using the Ray
ObjectRefStreamingGeneratorinterface. When the feature flag is on, the HTTP proxy will usenum_returns="streaming"for all calls to downstream replicas. The replica code has been modified to incrementally yield raw ASGI messages back to the HTTP proxy.Known limitations & follow-ups (to be addressed before a non-experimental release):
ServeHandleinterface: [serve][streaming] Support streaming w/ model composition (usingServeHandle) #35777max_concurrent_queriesis not respected by the HTTP proxy when streaming is enabled; we do simple round-robin instead: [serve][streaming] Respectmax_concurrent_querieswhen using streaming calls #35778Related issue number
#34266
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.