[datalabeling] testing: wrap rpcs with backoff

[tmatsuo]

fixes #3438
fixes #3439

We have been seeing flaky test results with the datalabeling test backends. This PR is wrapping all the rpc calls with backoff with max_time=60.

I believe this will significantly reduce the test flakiness. Here is an explanation.

If a test consists of multiple consecutive flaky rpc calls, the success rate of the test will become exponentially small as number of rpc grows.

Here is an example:

Let's say the success rate of a single rpc is always 0.9 (for simplicity) and
there is a test which has 8 consecutive rpc calls.

The success rate of the test as a whole:
0.9^8=~0.43
If we wrap this test with pytest.mark.flaky(max_runs=3), the success rate of it:
1 - 0.57^3 = 0.81

If we wrap each rpc calls with retries, the success rate will be significantly higher. Let's say we retry 3 times, then the success rate of a single rpc call (with 3 times retries):
1 - 0.1^3 = 0.999
The success rate of the test as a whole:
0.999^8 = ~0.992

The tests are still flaky, but significantly more robust without retrying the test itself. If we still see test failures, we may need to bump the max_time parameter on backoff.on_exception.

[tmatsuo]

The tests are still failing. According to the message, it seems like the test API backend is flaky.

The options came to mind are:

1. Use backoff to retry the API call for a certain duration.
   Currently flaky doesn't use any delay, so using backoff is probably better. Also, some tests are errored during the setup phase and it doesn't seem flaky retried that case. We can use backoff in the setup code.
   This might mitigate the test flakiness, but no guarantee.

2. Use the production endpoint, and somehow exclude our projects from manual review process (refer the comment).

3. Use mock client for the tests.

@kurtisvg @sgreenberg
WDYT?

[tmatsuo]

Ok, I'll go with the option 1 above first. A new commit is on the way.

[kurtisvg]

Ok, I'll go with the option 1 above first. A new commit is on the way.

Option 1 seems good to me - maybe we should reach out to the eng team and verify that the backend is flakey?

[tmatsuo]

Since it's not possible to run the tests locally, I'd let Kokoro run the tests with the said change only in one file.

[tmatsuo]

The change seems to work. I'll make similar changes to other files too.

[tmatsuo]

I think the code still has a problem around test_delete_dataset... I can ignore error in fixture teardown.

[sgreenberg]

@nnegrey , @dizcology - @tmatsuo asked me who knows about Data Labeling. I looked through the test history and it seems like you two might?

[nnegrey]

1. Seems fine, potentially the test service is flaky.
2. This is an option the product team would explicitly like us to avoid, because the Datalabeling is done by humans so the cost is higher and if our cancellation calls don't happen properly we can get a high bill with how many times all these tests run. The Test Service avoids this by mocking the completion, but I don't believe it runs fast enough for our tests which is why we cancel the jobs.

@tmatsuo, you can run the tests locally, you just have to set: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.kokoro/tests/run_tests.sh#L51

[tmatsuo]

@nnegrey Yeah I know that envvar. When I set this, it errors something like "API not enabled", and the link provided for enabling the API doesn't work. If there's a way to enable the test API other than that link, it should be documented.

Also, I'll appreciate it if you can review, thanks!

@kurtisvg The PR is ready for review, take a look if you have time.

[nnegrey]

Are you using the python project? I believe it may need special white listing to hit the test service.

[tmatsuo]

@nnegrey I'm using my own project. I understand I can use the shared project, but normally I'm using my own project for local testing to avoid contaminating the shared project.

Thanks for the review.

I'm going to merge this with my admin power.