…ature flag (opensearch-project#4163)

* Add IndexShard#getLatestReplicationCheckpoint behind segrep enable feature flag

Signed-off-by: Suraj Singh <[email protected]>

* Address review comment. Move tests to SegmentReplicationIndexShardTests

Signed-off-by: Suraj Singh <[email protected]>

* Add segrep enbaled index settings in TargetServiceTests, SourceHandlerTests

Signed-off-by: Suraj Singh <[email protected]>

…rver folder to geo module.

Refactors the GeoBoundsAggregation for geo_point types from the core server 
to the geo module.

Signed-off-by: Navneet Verma <[email protected]>

…ore enabling remote store (opensearch-project#4175)

* Add validator that forces segment replication type before enabling remote store

Signed-off-by: Sachin Kale <[email protected]>

…tion ID instead of DiscoveryNode. (opensearch-project#4182)

* Fix OngoingSegmentReplications to key by allocation ID instead of DiscoveryNode.

This change fixes segrep to work with multiple shards per node by keying ongoing replications on
allocation ID.  This also updates cancel methods to ensure state is properly cleared on shard cancel.

Signed-off-by: Marc Handalian <[email protected]>

* Clean up cancel methods.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

… it more clear (opensearch-project#4199)

* Change remote_store setting validation message to make it more clear

Signed-off-by: Sachin Kale <[email protected]>

… to ignore missing files causing exception (opensearch-project#4185)

* Update Store for segment replication dif

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

* Update recoveryDiff logic to ingore missing files causing exception on replica during copy

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
Co-authored-by: Poojita Raj <[email protected]>

…e segment store (opensearch-project#4020)

* Add RemoteSegmentStoreDirectory to interact with remote segment store

Signed-off-by: Sachin Kale <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

Bumps com.diffplug.spotless from 6.9.0 to 6.9.1.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…kpoint and processing a received checkpoint. (opensearch-project#4157)

* Adding PrimaryMode check before publishing checkpoint.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying spotless check

Signed-off-by: Rishikesh1159 <[email protected]>

* Moving segrep specific tests to SegmentReplicationIndexShardTests.

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding logic and tests for rejecting checkpoints if shard is in PrimaryMode.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying ./gradlew :server:spotlessApply.

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying ./gradlew :server:spotlessApply

Signed-off-by: Rishikesh1159 <[email protected]>

* Changing log level to warn in shouldProcessCheckpoint() of IndexShard.java class.

Signed-off-by: Rishikesh1159 <[email protected]>

* Removing unnecessary lazy logging in shouldProcessCheckpoint().

Signed-off-by: Rishikesh1159 <[email protected]>

Signed-off-by: Rishikesh1159 <[email protected]>

Sets analyzer to regex query string search

Signed-off-by: yyyogev <[email protected]>

Upgrades to lucene-9.4.0-snapshot-ddf0d0a by refactoring to TermOrdValComparator 
as a top level class.

Signed-off-by: Nicholas Walter Knize <[email protected]>

…3785)

* Adding documentation about creation of uber-JARs

Signed-off-by: Pranav Garg <[email protected]>

* Fixing linelint error

Signed-off-by: Pranav Garg <[email protected]>

* Comprehensive changes

Signed-off-by: Pranav Garg <[email protected]>

* Adding PR changes

Signed-off-by: Pranav Garg <[email protected]>

* PR changes

Signed-off-by: Pranav Garg <[email protected]>

Signed-off-by: Pranav Garg <[email protected]>
Co-authored-by: Pranav Garg <[email protected]>

…Sync. (opensearch-project#4224)

This PR Removes an unnecessary call to markAllocationIdAsInSync on the primary shard when replication events complete.
Recovery will manage this initial call.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

…#4226)

Prevents "search with uppercase regex" in 190_index_prefix_search.yml from
running in bwc testing until feature is backported.

Signed-off-by: Nicholas Walter Knize <[email protected]>

…ch-project#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <[email protected]>

Co-authored-by: opensearch-ci-bot <[email protected]>

…ds (opensearch-project#4236)

* [Segment Replication] Add thread sleep to account for replica lag in delete operations test

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments, assertBusy on doc count rather than sleep

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>

…ps. (opensearch-project#4237)

* Segment Replication - Add additional unit tests for update & delete operations.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

… into createpitrest

…#4135)

* Support shard promotion with Segment Replication.

This change adds basic failover support with segment replication.  Once selected, a replica will commit and reopen a writeable engine.

Signed-off-by: Marc Handalian <[email protected]>

* Add check to ensure a closed shard does not publish checkpoints.

Signed-off-by: Marc Handalian <[email protected]>

* Clean up in SegmentReplicationIT.

Signed-off-by: Marc Handalian <[email protected]>

* PR feedback.

Signed-off-by: Marc Handalian <[email protected]>

* Fix merge conflict.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

…opensearch-project#4222)

* Added timing data and more granular stages to SegmentReplicationState

This change introduces instrumentation logging that measures the latency of the various stages of segment replication as seen by each replica. Logs have also been added to the source node for checkpoint publishing and checkpoint metadata responses. All logging is currently at the TRACE level.

Signed-off-by: Kartik Ganesh <[email protected]>

* Fixing SegmentReplicationTarget tests

Signed-off-by: Kartik Ganesh <[email protected]>

* Incorporated PR feedback

Signed-off-by: Kartik Ganesh <[email protected]>

* Fixing SegmentReplicationTargetService tests

Signed-off-by: Kartik Ganesh <[email protected]>

Signed-off-by: Kartik Ganesh <[email protected]>

* Return 400 status code for array out of bound

Signed-off-by: Owais Kazi <[email protected]>

* Spotless apply

Signed-off-by: Owais Kazi <[email protected]>

* PR comments

Signed-off-by: Owais Kazi <[email protected]>

Signed-off-by: Owais Kazi <[email protected]>

… with higher replication checkpoint (opensearch-project#4041)

* [Segment Replication] Update PrimaryShardAllocator to prefer replicas having higher replication checkpoint

Signed-off-by: Suraj Singh <[email protected]>

* Use empty replication checkpoint to avoid NPE

Signed-off-by: Suraj Singh <[email protected]>

* Update NodeGatewayStartedShards to optionally wire in/out ReplicationCheckpoint field

Signed-off-by: Suraj Singh <[email protected]>

* Use default replication checkpoint causing EOF errors on empty checkpoint

* Add indexSettings to GatewayAllocator to allow ReplicationCheckpoint comparator only for segrep enabled indices

* Add unit tests for primary term first replica promotion & comparator fix

* Fix NPE on empty IndexMetadata

* Remove settings from AllocationService and directly inject in GatewayAllocator

* Add more unit tests and minor code clean up

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments & integration test

Signed-off-by: Suraj Singh <[email protected]>

* Fix comparator on null ReplicationCheckpoint

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Bharathwaj G <[email protected]>

…rch-project#4258)

Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

…wer bound (opensearch-project#4270)

Fixes tests to account for TotalHits uncertainty as of Lucene 9.

Signed-off-by: Daniel Widdis <[email protected]>

…4273)

Bumps com.gradle.enterprise from 3.10.3 to 3.11.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072) (opensearch-project#4180)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <[email protected]>

… into pitsegmentsrest1

Signed-off-by: Bharathwaj G <[email protected]>