Fix/Isolate flaky e2e tests

[kostis-codefresh]

Several times the e2e tests for a Pull Request fail even when the change is completely unrelated (e.g. docs)

Here are some examples:

• docs: mention docker image for CLI #1120
• docs: minor formatting corrections. Fixes #1113 #1179

We should fix those tests or isolate them or find the underlying issue. Otherwise github contributors will not trust e2e tests or even worse lose time debugging them when in reality they did not break anything with their contribution.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

Observed flaky tests:

TestRolloutAbortRetryPromote: 1
TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds: 2
TestIstioSuite/TestIstioHostSplitExperimentStep: 1
TestAnalysisSuite/TestMultipleAnalysis: 1
TestAnalysisSuite/TestCanaryInlineMultipleAnalysis: 1
TestAWSSuite/TestALBExperimentStepNoSetWeight : 1

[jessesuen]

Agree. The challenge has been the failure mode of these tests have not been reproducible outside of github actions, and they also seem to be not isolated to a single or few tests. I plan on exposing controller logging in the e2e tests to help chase this down.

[huikang]

I have seen the TestCanarySuite/TestCanaryScaleDownDelay failed from time to time and probably worth looking into it.

https://github.com/argoproj/argo-rollouts/pull/1224/checks?check_run_id=2681943983#step:8:649

[todaywasawesome]

May attempt to implement in Codefresh to see if it works better.

[noam-codefresh]

the latest e2e run on #1625 has two failing tests:
TestAnalysisSuite/TestMultipleAnalysis
TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds

[zachaller]

Not fixed 100% but dose improve the situation by allowing reries fixed in #2123