Flakey tests

[alexec]

This issue is for tracking flakey tests, so we can see how flakey they are.

Flakey tests are often not actually flakey, they instead revealing and edge-case bug that is caused by rare events, such as API disruption.

If a test appears to be flakey, then we simply disable it and wait for the author of the test (and therefore the feature) to investigate it. Either way, it is ideally their responsibility.

Why do we need to aggressively skip flakey tests?

Consider a test suite with 5 tests that each fail on 5% of the time. How often will the whole suite fail? Hint: it is much greater that 5x5%!

https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

• Is TestDoNotCreatePodsUnderStopBehavior flakey? #7155
• TestMonitorProgress takes 20s #7148
• Is TestSuccessfulJobHistoryLimit flakey? #7132
• Is TestCounterMetric flakey? #7120
• Is TestWorkflowTemplateRefWithShutdownAndSuspend flakey? #7119
• Scheduled time when submitting a cron workflow using --from is sometimes wrong #7111
• TestNewOperation might be flakey #7063
• Fix flakey TestDoNotCreatePodsUnderStopBehavior on docker #7040
• TestGetPodByNode is flakey #6458
• TestSignalsSuite/TestInjectedSidecar fails too often to be flakey IMHO #5627
• TestRetriesVariableInPodSpecPatch broken on master? #5526
• TestResubmitPendingPods is flakey #5365
• TestDeletingRunningPodWithOrErrorRetryPolicy is flakey, or is it a bug in retry? #5299
• Flakey tests in TestWorkflowTemplateSuite #4833
• TestDeletingRunningPod is flakey #4778
• "TestSynchronizationWfLevelMutex" is flakey #4629
• Flakey test TestResourceTemplateStopAndTerminate/ResourceTemplateStop #4559