Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core][Streaming generator] Support num_returns. #46934

Open
rkooo567 opened this issue Aug 2, 2024 · 2 comments
Open

[Core][Streaming generator] Support num_returns. #46934

rkooo567 opened this issue Aug 2, 2024 · 2 comments
Assignees
Labels
core Issues that should be addressed in Ray Core enhancement Request for new feature and/or capability P0 Issues that should be fixed in short order

Comments

@rkooo567
Copy link
Contributor

rkooo567 commented Aug 2, 2024

Description

Sometimes, users want to provide all the object references ahead of time if they know num_returns.

ref1, ref2 = task.options(num_returns=2).remote()

Use case

- even though it’s for unary requests, we actually currently use a streaming call to yield exactly two messages, one ray serve system message and then the actual result of the method call (this is in order for replicas to reject requests at max capacity)
- we want to allow the user to fetch the object ref for the result, but with a streaming call the object ref for the second message won’t be ready until a result is actually yielded. This is the issue, we want to give user an object ref pointing to the second message when the task is scheduled, not when the call returns (which is not possible with streaming right now)
@rkooo567 rkooo567 added enhancement Request for new feature and/or capability triage Needs triage (eg: priority, bug/not-bug, and owning component) core Issues that should be addressed in Ray Core labels Aug 2, 2024
@jjyao jjyao added P0.5 P0 Issues that should be fixed in short order and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) P0.5 labels Aug 2, 2024
@rkooo567
Copy link
Contributor Author

rkooo567 commented Aug 2, 2024

related: #46893

@kyle-v6x
Copy link

kyle-v6x commented Sep 3, 2024

Any updates on this? It's tagged as enhancement but the streaming change made in 2.10.0 results in breaking the _to_object_ref() calls as per #46893. All our Ray deployments are stuck on pre-2.10 for this reason.

@jjyao jjyao assigned jjyao and unassigned rkooo567 Sep 16, 2024
@jjyao jjyao assigned MengjinYan and jjyao and unassigned jjyao and MengjinYan Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Issues that should be addressed in Ray Core enhancement Request for new feature and/or capability P0 Issues that should be fixed in short order
Projects
None yet
Development

No branches or pull requests

4 participants