[Backport 2.16] Make reroute iteration time-bound for large shard allocations

[imRishN]

Backport - #14848

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❕ Gradle check result for 5d69f81: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 54.68750% with 29 lines in your changes missing coverage. Please review.

Project coverage is 71.54%. Comparing base (1c28e4e) to head (5d69f81).
Report is 9 commits behind head on 2.16.

Files	Patch %	Lines
...pensearch/gateway/ShardsBatchGatewayAllocator.java	28.57%	20 Missing ⚠️
...er/routing/allocation/ExistingShardsAllocator.java	0.00%	6 Missing ⚠️
.../cluster/routing/allocation/AllocationService.java	0.00%	2 Missing ⚠️
.../opensearch/common/util/BatchRunnableExecutor.java	94.73%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               2.16   #14930      +/-   ##
============================================
- Coverage     71.55%   71.54%   -0.02%     
- Complexity    62652    62661       +9     
============================================
  Files          5117     5118       +1     
  Lines        293928   294105     +177     
  Branches      42833    42853      +20     
============================================
+ Hits         210319   210408      +89     
- Misses        65975    66126     +151     
+ Partials      17634    17571      -63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andrross]

The original PR #14848 is still open and has unresolved conversations. I think we should get that code finalized before backporting.

[dblock]

Please use labels and automation to do backports unless there's a conflict so we can track backports carefully and make sure everything is backported that should be. Closing.