[BUG] Multiple tests failing with shard is not locked issue

[BhumikaSaini-Amazon]

Describe the bug

1. [BUG] SimpleClusterStateIT.testMetadataVersion is flaky #13207
2. [BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788
3. [BUG] The org.opensearch.remotestore.CreateRemoteIndexIT.testDefaultRemoteStoreNoUserOverride is flaky #12229
4. [BUG] org.opensearch.remotestore.RemoteIndexRecoveryIT.testCancelRecoveryWithAutoExpandReplicas is flaky #11952

The above tests seem to be failing due to this issue. Either we are missing some configuration required for the tests or we need some updates in the testing code. We need to debug and resolve this to avoid additional tests being impacted by the same and getting marked as flaky.

Related component

Other

To Reproduce

Run any of the above tests multiple times (2k+). It would fail at least once due to this issue.

Expected behavior

Tests should not fail due to this issue

Additional Details

No response

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@BhumikaSaini-Amazon Thanks for filing this use, looking forward to the pull request that will address this.

[peternied]

Per @mch2 this could be issue with doc -> seg replication, if that is the case or even if it isn't - should this be fixed before the 2.14 release? Seems like the answer is YES.