[Backport 2.x] Fix flaky test SegmentReplicationWithNodeToNodeIndexShardTests#testReplicaClosesWhileReplicating_AfterGetCheckpoint

[opensearch-trigger-bot]

Backport 5e2034c from #12695.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 9078210

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❌ Gradle check result for 215be52: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

@mch2 This test failed in https://build.ci.opensearch.org/job/gradle-check/35236/consoleText with the following error:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.shard.SegmentReplicationWithNodeToNodeIndexShardTests.testReplicaClosesWhileReplicating_AfterGetCheckpoint" -Dtests.seed=4060F9D0B1F3D9D8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-US -Dtests.timezone=America/Argentina/Salta -Druntime.java=21

org.opensearch.index.shard.SegmentReplicationWithNodeToNodeIndexShardTests > testReplicaClosesWhileReplicating_AfterGetCheckpoint FAILED
    java.lang.AssertionError: Should have resolved listener with failure expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([4060F9D0B1F3D9D8:DD0E4A2AD34C80FC]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.opensearch.index.shard.SegmentReplicationIndexShardTests.startReplicationAndAssertCancellation(SegmentReplicationIndexShardTests.java:1067)
        at org.opensearch.index.shard.SegmentReplicationWithNodeToNodeIndexShardTests.testReplicaClosesWhileReplicating_AfterGetCheckpoint(SegmentReplicationWithNodeToNodeIndexShardTests.java:141)

Does that mean this fix doesn't actually fix the flaky test?

[mch2]

this fails because #12043 was not backported to 2.x that includes changes to make cancel synchronous on replicas. Triggering backport on that first then will rebase this.

[github-actions]

❌ Gradle check result for 9078210: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

FAILURE

#12836

[github-actions]

❕ Gradle check result for 9078210: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.