Health api memory optimisation

[SwethaGuptha]

Description

#11684 - on a large cluster the memory allocation of TransportClusterHealthAction is around 33%. Memory/compute requirement of TransportClusterHealthAction is proportional to the number of indices and shards on the cluster. Transport ClusterHealthResponse maintains health objects for individual indices and shards that can increase the memory allocation of this with the growth of indices and shards count. At rest layer, there is a room for optimization for _cluster/health API when request is at cluster/indices level by performing inline aggregations and not creating and maintaing shard health objects.

Changes:

• Introducing a flag honourClusterHealthLevel in ClusterHealthRequest to identify if the request param level should be honored or not when computing health response in TransportClusterHealthAction - Currently level param is only used at REST layer for generating response from the transport response based on the level requested (usage) and has no meaning at transport layer. This flag will help TransportClusterHealthAction in differentiating if indices/shards level health information is required in the transport response or not. By default this flag is set to false to keep the change backward compatible.
• Initializing ClusterIndexHealth without creating the ClusterShardHealth objects per shard when requested level for cluster health is not at shards level, health aggregation are made inline.
• ClusterHealthResponse computes the delayed unassigned shards information by iterating over all shards of the cluster(reference).This is avoided by computing the delayed unassigned shards along with computation of other shard states in ClusterStateHealth.
• Using index based iteration for iterating over ShardRoutings of a shard. For each loop iteration uses an iterator under the hood and since the ShardRouting list maintained in the IndexShardRoutingTable is un-modifiable, a new iterator object is created on the heap.

Memory allocation 5 min profile of an idle cluster before and after optimisation
Before Optimisation

After Optimisation

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for ec0ffc3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4ed3b18: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ebadf42: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[gbbafna]

Introducing a flag honourClusterHealthLevel in ClusterHealthRequest to identify if the request param level should be honored or not when computing health response in TransportClusterHealthAction - Currently level param is only used at REST layer for generating response from the transport response based on the level requested (usage) and has no meaning at transport layer. This flag will help TransportClusterHealthAction in differentiating if indices/shards level health information is required in the transport response or not. By default this flag is set to false to keep the change backward compatible.

How are breaking backward compatibility when the flag is true ? Isn't this construct only used in Transport and not in Rest ?

[sumitasr]

Unit tests are missing for all the changes in source code. Can we please cover those for files where we have added custom logic to iterate and compute health?

UTs covering corner cases will help i guess if you can check on coverage once.

[github-actions]

❌ Gradle check result for 103f468: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 19e5625: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 19e5625: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3817fcd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[SwethaGuptha]

Unit tests are missing for all the changes in source code. Can we please cover those for files where we have added custom logic to iterate and compute health?

UTs covering corner cases will help i guess if you can check on coverage once.

Added UTs.

[github-actions]

❌ Gradle check result for d5c5869: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 77338be: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 75caa4a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[SwethaGuptha]

❌ Gradle check result for 75caa4a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Flaky test #15795

[github-actions]

✅ Gradle check result for 4c9c753: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15492-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 89915b65dc51126189dd42d74c82d2cbac0cb154
# Push it to GitHub
git push --set-upstream origin backport/backport-15492-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15492-to-2.x.

[opensearch-trigger-bot]

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.17
# Create a new branch
git switch --create backport/backport-15492-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 89915b65dc51126189dd42d74c82d2cbac0cb154
# Push it to GitHub
git push --set-upstream origin backport/backport-15492-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-15492-to-2.17.