Bugfix: add replica information in remote store restore flow #8951

sachinpkale · 2023-07-28T05:37:08Z

Description

Currently, in remote store restore flow, we only restore primary and ignore replicas. But this creates issue with shards having replicas as mentioned in [BUG] [Remote Store] Shard routing table has wrong number of replicas while restoring an index with >= 1 replicas from remote store #8479
In this change, we add replica information as well in the new routing table that is getting created as part of restore.
We apply following logic:
- If primary of the shard is unassigned, mark it for restore and mark all the replicas for peer recovery
- If primary of the shard is active or initializing, do not mark it for restore. Check all replicas and mark replica for peer recovery only if replica is unassigned.

Related Issues

Resolves #8479

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-28T05:47:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21228/
CommitID: 2b443e7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-28T06:23:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21231/
CommitID: 44c9460
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sachinpkale · 2023-07-28T07:33:01Z

While locally running the tests, following tests are failing intermittently. Looking into it:

org.opensearch.remotestore.RemoteStoreIT.testRTSRestoreWithCommittedDataPrimaryReplicaDown
org.opensearch.remotestore.RemoteStoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown

server/src/main/java/org/opensearch/cluster/routing/IndexRoutingTable.java

server/src/test/java/org/opensearch/cluster/routing/RoutingTableTests.java

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

server/src/main/java/org/opensearch/cluster/routing/IndexRoutingTable.java

github-actions · 2023-07-31T10:46:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21385/
CommitID: c466ba8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-31T14:15:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21410/
CommitID: f8b2e86
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-31T15:41:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21421/
CommitID: f8b2e86
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-01T04:21:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21532/
CommitID: b619807
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-04T05:02:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21859/
CommitID: e752a8b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-04T16:44:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21894/
CommitID: f9b541b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-07T04:42:09Z

Compatibility status:


> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/reporting.git with ref: main
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git]
Compatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 32m 26s

github-actions · 2023-08-07T04:42:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21981/
CommitID: 2474ad2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-07T11:05:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21995/
CommitID: 935ca9c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-07T11:25:08Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 27m 47s

opensearch-trigger-bot · 2023-08-07T11:46:22Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git]
Compatible components: [https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 21m 22s

github-actions · 2023-08-07T12:09:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/21996/
CommitID: 4e1c86e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-08-07T12:11:09Z

Codecov Report

Merging #8951 (d1cac6a) into main (8cfde6c) will increase coverage by 0.01%.
The diff coverage is 57.57%.

@@             Coverage Diff              @@
##               main    #8951      +/-   ##
============================================
+ Coverage     71.08%   71.10%   +0.01%     
- Complexity    57428    57513      +85     
============================================
  Files          4781     4781              
  Lines        271184   271197      +13     
  Branches      39591    39594       +3     
============================================
+ Hits         192780   192843      +63     
- Misses        62122    62138      +16     
+ Partials      16282    16216      -66

Files Changed	Coverage Δ
...arch/index/recovery/RemoteStoreRestoreService.java	`8.47% <0.00%> (+0.27%)`	⬆️
...ch/indices/recovery/PeerRecoveryTargetService.java	`53.94% <0.00%> (+0.65%)`	⬆️
...java/org/opensearch/index/shard/StoreRecovery.java	`56.54% <50.00%> (+0.26%)`	⬆️
.../opensearch/cluster/routing/IndexRoutingTable.java	`75.87% <73.33%> (-0.36%)`	⬇️
...a/org/opensearch/cluster/routing/RoutingTable.java	`94.82% <100.00%> (ø)`
...in/java/org/opensearch/index/shard/IndexShard.java	`69.47% <100.00%> (+0.16%)`	⬆️
...search/index/shard/RemoteStoreRefreshListener.java	`86.18% <100.00%> (+0.71%)`	⬆️

... and 453 files with indirect coverage changes

github-actions · 2023-08-22T05:54:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23156/
CommitID: a8c4eff
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-22T06:05:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23158/
CommitID: a8c4eff

github-actions · 2023-08-23T16:54:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23312/
CommitID: f0c45f1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-24T04:33:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/23371/
CommitID: 2533d9e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-08-24T09:46:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals}

URL: https://build.ci.opensearch.org/job/gradle-check/23389/
CommitID: 185dff0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Sachin Kale <[email protected]>

BhumikaSaini-Amazon

Thanks for this bugfix 🙌

github-actions · 2023-08-28T07:57:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23583/
CommitID: d1cac6a

Signed-off-by: Sachin Kale <[email protected]> (cherry picked from commit 569d5c2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]>

…9576) (cherry picked from commit 569d5c2) Signed-off-by: Sachin Kale <[email protected]>

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]>

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

sachinpkale added the skip-changelog label Jul 28, 2023

psychbot reviewed Jul 28, 2023

View reviewed changes

BhumikaSaini-Amazon reviewed Jul 31, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/routing/IndexRoutingTable.java Show resolved Hide resolved

sachinpkale force-pushed the bugfix-8479 branch from 44c9460 to c466ba8 Compare July 31, 2023 10:37

sachinpkale force-pushed the bugfix-8479 branch from f8b2e86 to b619807 Compare August 1, 2023 03:38

sachinpkale force-pushed the bugfix-8479 branch from b619807 to e752a8b Compare August 4, 2023 04:19

sachinpkale force-pushed the bugfix-8479 branch from f9b541b to 2474ad2 Compare August 7, 2023 04:09

sachinpkale marked this pull request as ready for review August 7, 2023 13:17

sachinpkale requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE and dblock as code owners August 7, 2023 13:17

sachinpkale force-pushed the bugfix-8479 branch from b824549 to a8c4eff Compare August 22, 2023 05:16

BhumikaSaini-Amazon mentioned this pull request Aug 23, 2023

[Remote Store] Fix tests when we restore index without any refresh #9480

Merged

6 tasks

sachinpkale force-pushed the bugfix-8479 branch from a8c4eff to f0c45f1 Compare August 23, 2023 16:10

sachinpkale force-pushed the bugfix-8479 branch from f0c45f1 to 2533d9e Compare August 24, 2023 03:52

sachinpkale requested a review from msfroh as a code owner August 24, 2023 03:52

Sachin Kale added 5 commits August 28, 2023 09:03

Bugfix: add replica information in remote store restore flow

218cb21

Signed-off-by: Sachin Kale <[email protected]>

Fix flaky tests

9c732c9

Signed-off-by: Sachin Kale <[email protected]>

Address PR comments

ef30613

Signed-off-by: Sachin Kale <[email protected]>

Fix to ignore only segment_N file

8f01905

Signed-off-by: Sachin Kale <[email protected]>

Address PR comments

d1cac6a

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale force-pushed the bugfix-8479 branch from 185dff0 to d1cac6a Compare August 28, 2023 07:18

BhumikaSaini-Amazon approved these changes Aug 28, 2023

View reviewed changes

gbbafna approved these changes Aug 28, 2023

View reviewed changes

gbbafna merged commit 569d5c2 into opensearch-project:main Aug 28, 2023
10 of 12 checks passed

gbbafna added the backport 2.x Backport to 2.x branch label Aug 28, 2023

opensearch-trigger-bot bot mentioned this pull request Aug 28, 2023

[Backport 2.x] Bugfix: add replica information in remote store restore flow #9576

Merged

mch2 pushed a commit to mch2/OpenSearch that referenced this pull request Aug 28, 2023

Bugfix: add replica information in remote store restore flow (opensea…

f792a36

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]>

gbbafna pushed a commit that referenced this pull request Aug 29, 2023

Bugfix: add replica information in remote store restore flow (#8951) (#…

de0de7a

…9576) (cherry picked from commit 569d5c2) Signed-off-by: Sachin Kale <[email protected]>

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Aug 29, 2023

Bugfix: add replica information in remote store restore flow (opensea…

c17961b

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]>

sachinpkale mentioned this pull request Aug 30, 2023

[BUG] org.opensearch.remotestore.RemoteStoreRestoreIT.testRTSRestoreWithRefreshedDataPrimaryReplicaDown is flaky #9408

Closed

brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023

Bugfix: add replica information in remote store restore flow (opensea…

5a27f09

…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix: add replica information in remote store restore flow #8951

Bugfix: add replica information in remote store restore flow #8951

sachinpkale commented Jul 28, 2023 •

edited

Loading

github-actions bot commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

sachinpkale commented Jul 28, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023

github-actions bot commented Jul 31, 2023

github-actions bot commented Jul 31, 2023

github-actions bot commented Aug 1, 2023

github-actions bot commented Aug 4, 2023

github-actions bot commented Aug 4, 2023

opensearch-trigger-bot bot commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

opensearch-trigger-bot bot commented Aug 7, 2023

opensearch-trigger-bot bot commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

codecov bot commented Aug 7, 2023 •

edited

Loading

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 23, 2023

github-actions bot commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

BhumikaSaini-Amazon left a comment

github-actions bot commented Aug 28, 2023

Bugfix: add replica information in remote store restore flow #8951

Bugfix: add replica information in remote store restore flow #8951

Conversation

sachinpkale commented Jul 28, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 28, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Jul 28, 2023 • edited Loading

github-actions bot commented Jul 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 4, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 7, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 7, 2023

opensearch-trigger-bot bot commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 7, 2023 • edited Loading

Codecov Report

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 24, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 24, 2023

Gradle Check (Jenkins) Run Completed with:

BhumikaSaini-Amazon left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 28, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Jul 28, 2023 •

edited

Loading

sachinpkale commented Jul 28, 2023 •

edited

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading