Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Serve][Cherry-Pick] Fix the max_concurrent_queries issue #32974

Merged
merged 2 commits into from
Mar 7, 2023
Merged

[Serve][Cherry-Pick] Fix the max_concurrent_queries issue #32974

merged 2 commits into from
Mar 7, 2023

Commits on Mar 7, 2023

  1. [Serve] Add hash function of RunningReplicaInfo (ray-project#32772) 

    When the longpoll client timeout happens, the all internal objects will be cleaned up. The longpoll client will try to poll again. When this happens, longpoll client will receive another object which having same information but different object id (ActorHandle) from the controller.
Then after compute_iterable_delta is called, it will replace the same replica in in_flight_queries, and cause router clean up ongoing request_ref, and keep assigning new request to the replica, which will break the max_concurrent_queries parameter.

Related issue number
Closes ray-project#32652

Signed-off-by: Sihan Wang <[email protected]>
    @sihanwang41
    sihanwang41 committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    b16b4e1 View commit details
    Browse the repository at this point in the history

  2. [Serve] Fix the max_concurrent_queries issue (ray-project#33022) 

    For the `hashable` object, __eq__ and __hash__ both need to be provided for correctness.  https://docs.python.org/3.9/glossary.html#term-hashable

And add tests to make sure the long poll timeout issue won't happen.
    @sihanwang41
    sihanwang41 committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    d8efee2 View commit details
    Browse the repository at this point in the history