Improving Functional Test Reliability for Beta and Prod Environments in OSSD

[VonnyJap]

What happened:

We run functional tests against the beta and prod environments for Open Source SD using the screwdriver-cd/screwdriver pipeline. Unfortunately, this pipeline has become very unstable, causing significant delays and blocking further releases.

Notable causes of flaky tests:

• As we continue running more tests, we've encountered GitHub rate limiting
• We have also faced issues with longer-running tests that eventually time out.

What you expected to happen:

A reliable set of functional tests that can be used to certify the version release.

How to reproduce it:

Go to the pipeline and select a random event, such as this one. Notice how many restarts are required to get the entire test suite to pass.

[VonnyJap]

@kumada626 @sagar1312 - The plan is right now to tackle the problem with inconsistent and unreliable functional tests by splitting the tasks into two parallel jobs running with two different Github users. This hopefully will help to avoid the rate limiting.

[yakanechi]

@VonnyJap

We have also faced issues with longer-running tests that eventually time out.

To solve the above problem, we proposed the following PR to speed up testing by parallelization.
#3194

However, as mentioned in the PR, the test will run faster than the Github rate limit reset, so the rate limit problem may occur again.

As a workaround, it may be possible to achieve both higher speed and avoidance of ghe rate limit by further dividing the test account as in beta-x1 into accounts like beta, beta-x1, prod, and prod-x1.

[VonnyJap]

@yakanechi - I learned that personal access token has the lowest rate limit. But if we can make a switch to use either oauth or github apps model of auth, it will help to avoid the rate limit. Please checkout the docs here. But we have not had time to incorporate the change.