[ACTION NEEDED] Fix flaky integration tests at distribution level

[gaiksaya]

What is the bug?
It was observed in 2.13.0 and previous other releases that this component manually signed off on the release for failing integration tests. See opensearch-project/opensearch-build#4433 (comment)
The flakiness of the test runs take a lot of time from the release team to collect go/no-go decision and significantly lower the confidence in the release bundles.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Run integration testing for altering and see the failures.
2. Issues can be reproduced using the steps declared in AUTOCUT issues for failed integration testing

What is the expected behavior?
Tests should be consistently passing.

Do you have any additional context?
Please note that this is a hard blocker for 2.14.0 release as per the discussion here

[bbarani]

@goyamegh @praveensameneni Can you please provide your inputs?

[gaiksaya]

Adding 2.14.0 release manager @rishabh6788

[praveensameneni]

closing the issue as there are no test failures after merging the above PR #1005

[praveensameneni]

Re-opening the issue to track the following:
Number of flaky tests that have been ignored with @ignore flag: 9

1. testAlertHistoryRollover_maxAge() throws IOException, InterruptedException {
   updateClusterSetting(ALERT_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
   updateClusterSetting(ALERT_HISTORY_MAX_DOCS.getKey(), "1000");
2. testAlertHistoryRollover_maxAge_low_retention() throws IOException, InterruptedException {
   updateClusterSetting(ALERT_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
   updateClusterSetting(ALERT_HISTORY_MAX_DOCS.getKey(), "1000");
3. public void testAlertHistoryRollover_maxDocs() throws IOException, InterruptedException {
   updateClusterSetting(ALERT_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
   updateClusterSetting(ALERT_HISTORY_MAX_DOCS.getKey(), "1");
4. public void testGetFindings_bySearchString_success() throws IOException {
   String index1 = createTestIndex(randomIndex(), windowsIndexMapping());
5. public void testGetFindings_byStartTimeAndEndTime_success() throws IOException {
   String index1 = createTestIndex(randomIndex(), windowsIndexMapping());
6. public void testGetFindings_rolloverByMaxAge_success() throws IOException, InterruptedException {
   updateClusterSetting(FINDING_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
7. public void testGetFindings_rolloverByMaxDoc_success() throws IOException, InterruptedException {
   updateClusterSetting(FINDING_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
8. testCreateDetectorWithNotCondition_verifyFindings_success() throws IOException {
   String index = createTestIndex(randomIndex(), windowsIndexMapping());
9. testCreateDetectorWithNotCondition_verifyFindings_success_boolAndNum() throws IOException {
   String index = createTestIndex(randomIndex(), windowsIndexMapping());

[rishabh6788]

@praveensameneni Given OS-2.15.0 release will start in a couple of weeks, request you to prioritize the test fixes.
If you need any support from the infrastructure team, please feel free to create an issue or reach out to us.
@getsaurabh02