[serve] change default for max and target ongoing requests #45943
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zcin
force-pushed
the
pr45943
branch
3 times, most recently
from
7b5e387
to
471c780
Compare
zcin
requested review from
edoakes,
shrekris-anyscale,
GeneDer,
akshay-anyscale and
a team
as code owners
zcin
force-pushed
the
pr45943
branch
2 times, most recently
from
5ad7f14
to
2111dab
Compare
GeneDer
reviewed
Jun 18, 2024
shrekris-anyscale
approved these changes
Jun 18, 2024
|
@edoakes could you help take a look when you get the chance? |
edoakes
approved these changes
Jun 24, 2024
can-anyscale
approved these changes
Jun 25, 2024
Change default for `max_ongoing_requests` from 100 to 5 Change default for `target_ongoing_requests` from 1.0 to 2.0. Add warnings in rest api and @serve.deployment about these changes. Signed-off-by: Cindy Zhang <[email protected]>
GeneDer
added a commit
to GeneDer/ray
that referenced
this pull request
Jul 2, 2024
…ay-project#45943)" This reverts commit c40878e.
GeneDer
added a commit
to GeneDer/ray
that referenced
this pull request
Jul 3, 2024
…uests (ray-project#45943)"" This reverts commit b779d4b.
8 tasks
edoakes
pushed a commit
that referenced
this pull request
Jul 5, 2024
…0` and doc change (#46399) <!-- Thank you for your contribution! Please review https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before opening a pull request. --> <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> ## Why are these changes needed? Re: #45943 changed the default `max_ongoing_requests` down to 5 which caused the overall rps lower as will. This PR did two things: 1. Added a set of performance tests for `max_ongoing_requests =100` 2. Added related doc change to explicitly calling out setting up a higher value for `max_ongoing_requests` for lightweight requests ## Related issue number Closes metrics dropping at https://b534fd88.us1a.app.preset.io/explore/?form_data_key=YWenQf8xEN9DHx7wmFVo3cRXYD-xoqObnK7HEC8J2Ho8FMnPCYM-d_as4EO7CG4b&dashboard_page_id=DCipe40Dxo&slice_id=1380 ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Gene Su <[email protected]>
can-anyscale
added a commit
that referenced
this pull request
Jul 10, 2024
linux://python/ray/dashboard:test_serve_dashboard has become timedout and flaky recently (#46459); not sure if it has anything to do with #45943. I just increase its timed out in this PR Test: - CI - https://buildkite.com/ray-project/postmerge/builds/5358 Signed-off-by: can <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[serve] change default for max and target ongoing requests
Change default for
max_ongoing_requestsfrom 100 to 5Change default for
target_ongoing_requestsfrom 1.0 to 2.0.Add warnings in rest api and @serve.deployment about these changes.
Signed-off-by: Cindy Zhang [email protected]