A way to schedule retries at the end of the test run

[sergeshustoff]

Is your feature request related to a problem? Please describe.
Let's say we have a flacky environment that might cause a few tests to fail in a row. If we use retry strategy in attempt to avoid failing test run on such flickers and no batching, it's likely that all the retries would hit the same environment fluke.

Describe the solution you'd like
Maybe it would be beneficial to have an alternative strategy for retry scheduling - scheduling all failed tests at the end of test run and then repeating X times according to the quota?

Describe alternatives you've considered
Tried using random order execution strategy - doesn't help.

Additional context
Add any other context or code about the feature request here.

[Malinskiy]

Hey @sergeshustoff
Assuming such option would exist, for a test execution of the following structure
❌ - ❌ - ❌ - ❌ - 🟢

where ❌ represents test failure and 🟢 represents test passing

the option you're describing would just shift a serial failure to
❌ - 🟢 - ❌ - ❌ - ❌

In short, all your retries would still fail serially, just at the end and not at the beginning

[sergeshustoff]

@Malinskiy in some cases yes. But if there is a significant number of failures, a limited number of devices and failures sometimes caused by test BE short shutdowns or something like that it would reduce the likelihood of failed run with the same number of retries. I'm talking about a cases when it's clearly visible on timeline that all devices fail tasts simultaneously for a few minutes.

It has limited usefulness for most, but sometimes might be useful for end-to-end tests with not quite stable environment (though I'm not entirely sure what exactly went wrong in this specific case)