[Backport] [2.x] Add search backpressure cancellation at the coordinator level (#5605)

[PritLadani]

Description

Cancellation of in-flight SearchTasks based on resource consumption.

This feature aims to identify and cancel resource intensive SearchTasks if they have breached certain
thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the
cluster performance.

1. For resource tracking of SearchTasks, we will use the existing thresholds used for SearchShardTask but with different thresholds.
2. Introduced new dynamic settings for defining thresholds for SearchTask cancellation.
3. Added search_task stats in the response body of _nodes/stats/search_backpressure.

Issues Resolved

#5173

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: PritLadani [email protected]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10678/
• CommitID: 98de3c0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10680/
• CommitID: 5e262a6

[codecov-commenter]

Codecov Report

Merging #6194 (a73f3a3) into 2.x (7df021b) will increase coverage by 0.09%.
The diff coverage is 69.29%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #6194      +/-   ##
============================================
+ Coverage     70.32%   70.42%   +0.09%     
- Complexity    58931    58992      +61     
============================================
  Files          4774     4776       +2     
  Lines        282869   283165     +296     
  Branches      41213    41229      +16     
============================================
+ Hits         198935   199422     +487     
+ Misses        67261    67026     -235     
- Partials      16673    16717      +44     

Impacted Files	Coverage Δ
.../org/opensearch/action/search/SearchShardTask.java	87.50% <ø> (ø)
.../java/org/opensearch/action/search/SearchTask.java	75.00% <ø> (ø)
...uster/routing/allocation/IndexMetadataUpdater.java	86.39% <0.00%> (ø)
...rg/opensearch/common/settings/ClusterSettings.java	91.89% <ø> (ø)
...earch/backpressure/stats/SearchShardTaskStats.java	64.86% <ø> (ø)
...ackpressure/trackers/TaskResourceUsageTracker.java	100.00% <ø> (ø)
...a/org/opensearch/test/OpenSearchIntegTestCase.java	55.02% <0.00%> (-0.82%)	⬇️
...ch/backpressure/stats/SearchBackpressureStats.java	60.00% <36.84%> (-12.23%)	⬇️
...h/search/backpressure/SearchBackpressureState.java	62.96% <44.44%> (-37.04%)	⬇️
...ex/translog/transfer/BlobStoreTransferService.java	70.21% <52.94%> (-16.46%)	⬇️
... and 535 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10679/
• CommitID: f2567b9

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10681/
• CommitID: 89b8471
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[nssuresh2007]

LGTM

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10682/
• CommitID: a77f009
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10684/
• CommitID: a77f009
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[kotwanikunal]

[91mE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/haveged/libhavege1_1.7c-1_amd64.deb  Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out [IP: 185.125.190.36 80]
  
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/haveged/haveged_1.7c-1_amd64.deb  Unable to connect to archive.ubuntu.com:http: [IP: 185.125.190.36 80]
  
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  �[0m

Looks like an unrelated failure. There is also a merge conflict with the changelog.

@PritLadani Can you please rebase off 2.x?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10756/
• CommitID: a73f3a3

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

• URL: https://build.ci.opensearch.org/job/gradle-check/10762/
• CommitID: d262f4e
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[PritLadani]

@Bukhtawar can you please help review and merge the change?