Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set target_throughput to none for NYC taxis nightly benchmarks #4725

Merged
merged 1 commit into from
May 23, 2024

Conversation

jed326
Copy link
Contributor

@jed326 jed326 commented May 23, 2024

Description

Set target_throughput to none for nyc_taxis nightly runs. Ref:
https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/nyc_taxis/test_procedures/default.json#L56-L69

Illustrating the issue with 2 screenshots below:

First, this is p90 CPU usage of the entire test procedure. The section in the middle where the graph is not at 0 is for the distance_amount_agg operation while the p90 CPU is at 0% for all other operations of the procedure.
Screenshot 2024-05-23 at 2 19 38 PM

Similar story is painted by the search threadpool active thread count metric below. These node stats are sampled at 1 second intervals, so we can see from this that after the distance_amount_agg we are not really filling up the search threadpool and thus not really generating real load on the cluster.
Screenshot 2024-05-23 at 2 20 07 PM

This problem is also evident when we look at the performance numbers themselves as we increase the client count for operations other than the distance_amount_agg as we do not see performance decreasing with increased client count for other operations, even in the case of concurrent search enabled which should be consuming all CPU resources at the higher client counts.

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@rishabh6788
Copy link
Collaborator

Please run the below commands to update the test files and add to the PR.

./gradlew clean
./gradlew test -info -Ppipeline.stack.write=true --tests=TestRunNonSecBenchmarkTestScript
./gradlew test -info -Ppipeline.stack.write=true --tests=TestRunSecureBenchmarkTestScript
 ./gradlew test -info -Ppipeline.stack.write=true --tests=TestRunBenchmarkTestScriptDistributionUrl

@rishabh6788 rishabh6788 merged commit cdb8e47 into opensearch-project:main May 23, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants