Add max_shard_size parameter for Shrink API #5229

gaobinlong · 2022-11-12T09:55:24Z

Signed-off-by: Gao Binlong [email protected]

Description

As the document mentions that we have a parameter max_primary_shard_size, but actually it doesn't exist, and the parameter is used to generate an optimum number_of_shards of the new shrunken index, so it's useful for users. I've implemented the function and changed the parameter's name to max_shard_size so that it can be consistent with the similar function in Shrink Action in Index Management Plugin.

The main changes of this PR are:

Add max_shard_size parameter for Shrink API.
Add max_shard_size parameter for Shrink API in high level rest client.
Add unit test, integration test and yaml rest test.

Issues Resolved

#5170

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Gao Binlong <[email protected]>

github-actions · 2022-11-12T10:25:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6832/
CommitID: 5ef329a
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-12T10:30:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6833/
CommitID: 88a5528
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gao Binlong <[email protected]>

github-actions · 2022-11-13T04:06:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6842/
CommitID: 120e8fc

codecov-commenter · 2022-11-13T04:07:29Z

Codecov Report

Merging #5229 (a1214a6) into main (110e409) will decrease coverage by 0.04%.
The diff coverage is 64.10%.

@@             Coverage Diff              @@
##               main    #5229      +/-   ##
============================================
- Coverage     71.02%   70.98%   -0.05%     
- Complexity    58119    58158      +39     
============================================
  Files          4711     4711              
  Lines        277533   277571      +38     
  Branches      40169    40180      +11     
============================================
- Hits         197125   197030      -95     
- Misses        64256    64402     +146     
+ Partials      16152    16139      -13

Impacted Files	Coverage Δ
...ion/admin/indices/shrink/ResizeRequestBuilder.java	`0.00% <0.00%> (ø)`
...rch/action/admin/indices/shrink/ResizeRequest.java	`56.62% <35.71%> (-1.35%)`	⬇️
...a/org/opensearch/client/indices/ResizeRequest.java	`83.78% <66.66%> (-1.52%)`	⬇️
...on/admin/indices/shrink/TransportResizeAction.java	`65.00% <90.00%> (+5.74%)`	⬆️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	`9.75% <0.00%> (-73.18%)`	⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java	`17.85% <0.00%> (-53.58%)`	⬇️
.../java/org/opensearch/node/NodeClosedException.java	`50.00% <0.00%> (-50.00%)`	⬇️
.../index/shard/IndexShardNotRecoveringException.java	`0.00% <0.00%> (-50.00%)`	⬇️
...regations/metrics/AbstractHyperLogLogPlusPlus.java	`51.72% <0.00%> (-44.83%)`	⬇️
... and 453 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-11-13T04:13:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/6843/
CommitID: 2bad9ff

server/src/main/java/org/opensearch/action/admin/indices/shrink/ResizeRequest.java

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java

kolchfa-aws · 2022-11-16T18:12:01Z

When this is done, please open a documentation issue to update the docs. Thanks!

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong · 2022-11-17T09:43:33Z

@kolchfa-aws, sure, I will do that.

Signed-off-by: Gao Binlong <[email protected]>

github-actions · 2022-11-23T06:42:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7262/
CommitID: 8b7a9d6
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-23T06:54:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7263/
CommitID: 31f728a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gao Binlong <[email protected]>

github-actions · 2022-11-24T08:24:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

URL: https://build.ci.opensearch.org/job/gradle-check/7315/
CommitID: a1214a6
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

gaobinlong · 2022-11-24T10:08:30Z

@reta hi, I've made some change after your last review, can you help to take a look? Thanks.

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java

reta · 2022-11-25T13:14:35Z

@gaobinlong could you please create an issue at https://github.com/opensearch-project/documentation-website to document this new parameter for Shrink API, thank you

dblock

Tiny nit in CHANGELOG, let's keep that clean pls.

Also what's the behavior without max shard size? No limit?

dblock · 2022-11-28T17:56:58Z

CHANGELOG.md

@@ -84,6 +84,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Added
 - Prevent deletion of snapshots that are backing searchable snapshot indexes ([#5069](https://github.com/opensearch-project/OpenSearch/pull/5069))



Delete the new blank line above please.

dblock · 2022-11-28T17:57:40Z

client/rest-high-level/src/test/java/org/opensearch/client/IndicesRequestConvertersTests.java

@@ -702,6 +703,9 @@ private void resizeTest(ResizeType resizeType, CheckedFunction<ResizeRequest, Re
        if (resizeType == ResizeType.SPLIT) {
            resizeRequest.setSettings(Settings.builder().put("index.number_of_shards", 2).build());
        }
+        if (resizeType == ResizeType.SHRINK) {


Nit: add else

dblock · 2022-11-28T17:59:44Z

CHANGELOG.md

@@ -84,6 +84,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Added
 - Prevent deletion of snapshots that are backing searchable snapshot indexes ([#5069](https://github.com/opensearch-project/OpenSearch/pull/5069))

+- Add max_shard_size parameter for Shrink API ([#5229](https://github.com/opensearch-project/OpenSearch/pull/5229))


Lowercase shrink

gaobinlong · 2022-11-29T09:27:25Z

@reta I've opened a document issue yet, thanks again for your review.

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong · 2022-11-29T09:51:21Z

@dblock thanks for your review, I have made some changes yet. Without max_shard_size, there is no limit on shard's storage size, but there is a document count limit for every shard and the upper limit is 2^31-1-128:
https://github.com/apache/lucene/blob/0cc6f695363419ab0f89e2bef5e7595ace077345/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L200).
And we will throw illegalArgumentException if the new shard's document count will exceed the upper limit.

github-actions · 2022-11-29T10:04:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7405/
CommitID: c1240cb

opensearch-trigger-bot · 2022-11-30T15:31:51Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5229-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 953a3d6851632c9b09c8dc15dbff02c8aafac6c1
# Push it to GitHub
git push --set-upstream origin backport/backport-5229-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5229-to-2.x.

dblock · 2022-11-30T15:59:39Z

@gaobinlong Looks like it will need a manual backport to 2.x, thx.

gaobinlong · 2022-12-01T05:58:32Z

@dblock ok, I will do it.

* Add max_shard_size parameter for Shrink API Signed-off-by: Gao Binlong <[email protected]> * add change log Signed-off-by: Gao Binlong <[email protected]> * fix yaml test failed Signed-off-by: Gao Binlong <[email protected]> * optimize the code Signed-off-by: Gao Binlong <[email protected]> * fix test failed Signed-off-by: Gao Binlong <[email protected]> * optimize changelog & code Signed-off-by: Gao Binlong <[email protected]> Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit 953a3d6)

Add max_shard_size parameter for Shrink API

5ef329a

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong requested review from a team and reta as code owners November 12, 2022 09:55

add change log

88a5528

Signed-off-by: Gao Binlong <[email protected]>

fix yaml test failed

2bad9ff

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong force-pushed the add_shrink_parameter branch from 120e8fc to 2bad9ff Compare November 13, 2022 03:39

reta reviewed Nov 14, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/indices/shrink/ResizeRequest.java Outdated Show resolved Hide resolved

reta reviewed Nov 14, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java Show resolved Hide resolved

reta reviewed Nov 14, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java Show resolved Hide resolved

reta reviewed Nov 14, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java Show resolved Hide resolved

kolchfa-aws mentioned this pull request Nov 16, 2022

Removes max_primary_shard_size parameter opensearch-project/documentation-website#1914

Merged

merge main

81bca05

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong added 2 commits November 23, 2022 14:21

optimize the code

7fbca43

Signed-off-by: Gao Binlong <[email protected]>

Merge remote-tracking branch 'upstream/main' into add_shrink_parameter

31f728a

gaobinlong force-pushed the add_shrink_parameter branch from 8b7a9d6 to 31f728a Compare November 23, 2022 06:25

gaobinlong added 2 commits November 24, 2022 15:48

fix test failed

98fe8c1

Signed-off-by: Gao Binlong <[email protected]>

merge main

a1214a6

Signed-off-by: Gao Binlong <[email protected]>

gaobinlong commented Nov 25, 2022

View reviewed changes

server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java Show resolved Hide resolved

reta approved these changes Nov 25, 2022

View reviewed changes

gaobinlong mentioned this pull request Nov 28, 2022

[DOC] Add max_shard_size parameter for Shrink API opensearch-project/documentation-website#2044

Closed

1 task

dblock reviewed Nov 28, 2022

View reviewed changes

gaobinlong added 2 commits November 29, 2022 17:31

optimize changelog & code

2c0ad77

Signed-off-by: Gao Binlong <[email protected]>

Merge remote-tracking branch 'upstream/main' into add_shrink_parameter

c1240cb

dblock approved these changes Nov 30, 2022

View reviewed changes

dblock merged commit 953a3d6 into opensearch-project:main Nov 30, 2022

dblock added the backport 2.x Backport to 2.x branch label Nov 30, 2022

gaobinlong mentioned this pull request Dec 9, 2022

[Backport 2.x] Add max_shard_size parameter for Shrink API #5498

Merged

6 tasks

gaobinlong mentioned this pull request Jan 9, 2023

[DOC] Add max_shard_size parameter to shrink API page opensearch-project/documentation-website#2352

Closed

1 task

gaobinlong mentioned this pull request Dec 4, 2023

Update supported version for max_shard_size parameter in Shrink API #11439

Merged

8 tasks

reta mentioned this pull request Jul 17, 2024

Add Gao Binlong as maintainer #14796

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add max_shard_size parameter for Shrink API #5229

Add max_shard_size parameter for Shrink API #5229

gaobinlong commented Nov 12, 2022 •

edited

Loading

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 13, 2022

codecov-commenter commented Nov 13, 2022 •

edited

Loading

github-actions bot commented Nov 13, 2022

kolchfa-aws commented Nov 16, 2022

gaobinlong commented Nov 17, 2022

github-actions bot commented Nov 23, 2022

github-actions bot commented Nov 23, 2022

github-actions bot commented Nov 24, 2022

gaobinlong commented Nov 24, 2022

reta commented Nov 25, 2022

dblock left a comment •

edited

Loading

dblock Nov 28, 2022 •

edited

Loading

dblock Nov 28, 2022

dblock Nov 28, 2022

gaobinlong commented Nov 29, 2022

gaobinlong commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

opensearch-trigger-bot bot commented Nov 30, 2022

dblock commented Nov 30, 2022

gaobinlong commented Dec 1, 2022

		@@ -84,6 +84,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
		### Added
		- Prevent deletion of snapshots that are backing searchable snapshot indexes ([#5069](https://github.com/opensearch-project/OpenSearch/pull/5069))

Add max_shard_size parameter for Shrink API #5229

Add max_shard_size parameter for Shrink API #5229

Conversation

gaobinlong commented Nov 12, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Nov 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 13, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Nov 13, 2022 • edited Loading

Codecov Report

github-actions bot commented Nov 13, 2022

Gradle Check (Jenkins) Run Completed with:

kolchfa-aws commented Nov 16, 2022

gaobinlong commented Nov 17, 2022

github-actions bot commented Nov 23, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 23, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 24, 2022

Gradle Check (Jenkins) Run Completed with:

gaobinlong commented Nov 24, 2022

reta commented Nov 25, 2022

dblock left a comment • edited Loading

Choose a reason for hiding this comment

dblock Nov 28, 2022 • edited Loading

Choose a reason for hiding this comment

dblock Nov 28, 2022

Choose a reason for hiding this comment

dblock Nov 28, 2022

Choose a reason for hiding this comment

gaobinlong commented Nov 29, 2022

gaobinlong commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Nov 30, 2022

dblock commented Nov 30, 2022

gaobinlong commented Dec 1, 2022

gaobinlong commented Nov 12, 2022 •

edited

Loading

codecov-commenter commented Nov 13, 2022 •

edited

Loading

dblock left a comment •

edited

Loading

dblock Nov 28, 2022 •

edited

Loading