Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup a nightly runs for semantic search at noaa workload. #4733

Merged
merged 2 commits into from
May 29, 2024
Merged

Setup a nightly runs for semantic search at noaa workload. #4733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4c30dcc
Setup a nightly runs for semantic search at noaa workload
rishabh6788 May 29, 2024
416ec95
Merge branch 'opensearch-project:main' into main
rishabh6788 May 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions jenkins/opensearch/benchmark-test.jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,16 @@ pipeline {
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_3_shards"};TEST_PROCEDURE=restore-from-snapshot
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.xlarge,major-version:2x,cluster-config:x64-r5.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_3_shards"};TEST_PROCEDURE=restore-from-snapshot

H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full

H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full

H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_default"};TEST_PROCEDURE=restore-from-snapshot
H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_default"};TEST_PROCEDURE=restore-from-snapshot
H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.xlarge,major-version:2x,cluster-config:x64-r5.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_default"};TEST_PROCEDURE=restore-from-snapshot
Expand Down
10 changes: 10 additions & 0 deletions ...ns/jenkinsjob-regression-files/opensearch/benchmark-test-distribution-url.jenkinsfile.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,16 @@
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_3_shards"};TEST_PROCEDURE=restore-from-snapshot
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=big5;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.xlarge,major-version:2x,cluster-config:x64-r5.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"big5_3_shards"};TEST_PROCEDURE=restore-from-snapshot

H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 6 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9820/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full

H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r5.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.4xlarge,major-version:2x,cluster-config:x64-r5.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"false"};TEST_PROCEDURE=hybrid-query-aggs-full
H 7 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=noaa_semantic_search;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;DATA_INSTANCE_TYPE=r6g.4xlarge;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.4xlarge,major-version:2x,cluster-config:arm64-r6g.4xlarge-3-data-6-shards-0-replica-snapshot;WORKLOAD_PARAMS={"number_of_shards":"6","max_num_segments":"8","concurrent_segment_search_enabled":"true"};TEST_PROCEDURE=hybrid-query-aggs-full

H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_default"};TEST_PROCEDURE=restore-from-snapshot
H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=nyc_taxis;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"nyc_taxis_default"};TEST_PROCEDURE=restore-from-snapshot
H 8 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;USER_TAGS=run-type:nightly,segrep:disabled,arch:x64,instance-type:r5.xlarge,major-version:2x,cluster-config:x64-r5.xlarge-3-data-3-shards-0-replica-snapshot;WORKLOAD_PARAMS={"snapshot_repo_name":"benchmark-workloads-repo","snapshot_bucket_name":"benchmark-workload-snapshots","snapshot_region":"us-east-1","snapshot_base_path":"workload-snapshots","snapshot_name":"http_logs_default"};TEST_PROCEDURE=restore-from-snapshot
Expand Down
Loading
Loading