[ACTION NEEDED] Fix flaky integration tests at distribution level

[gaiksaya]

What is the bug?
It was observed in 2.13.0 and previous other releases that this component manually signed off on the release for failing integration tests. See opensearch-project/opensearch-build#4433 (comment)
The flakiness of the test runs take a lot of time from the release team to collect go/no-go decision and significantly lower the confidence in the release bundles.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Run integration testing for altering and see the failures.
2. Issues can be reproduced using the steps declared in AUTOCUT issues for failed integration testing

What is the expected behavior?
Tests should be consistently passing.

Do you have any additional context?
Please note that this is a hard blocker for 2.14.0 release as per the discussion here

[bbarani]

@martin-gaievski @vibrantvarun Can you please provide your inputs?

[martin-gaievski]

As per our deep dive inconsistency in integ tests is caused by the implementation of memory circuit breaker on ml-commons side. In scope of our tests we're deploying/undeploying local models using ml-commons API and after multiple such calls memory CB became open. I've opened an issue in ml-commons for this matter opensearch-project/ml-commons#2308.
As a short-term mitigation for 2.14 we have optimized our tests (issues #683 and #689) to minimize number of times local model got redeployed. As of now integ tests are constantly pass in our local copy of https://github.com/opensearch-project/opensearch-build/

[gaiksaya]

Adding 2.14.0 release manager @rishabh6788

[martin-gaievski]

Seems that implemented approach is stable, there are no failing tests in 2.14 release pipeline. Resolving this issue