Retry if github returns 403 rate limit warning

[jaqx0r]

Please implement a backoff-and-retry when GitHub returns a 403 error.

As seen in https://github.com/google/mtail/runs/2969598960?check_suite_focus=true

Traceback (most recent call last):
  File "/action/publish_unit_test_results.py", line 91, in <module>
    main(settings)
  File "/action/publish_unit_test_results.py", line 32, in main
    Publisher(settings, gh).publish(stats, results.case_results)
  File "/action/publish/publisher.py", line 64, in publish
    pull = self.get_pull(self._settings.commit)
  File "/action/publish/publisher.py", line 80, in get_pull
    self._logger.debug('found {} pull requests for commit {}'.format(issues.totalCount, commit))
  File "/usr/local/lib/python3.6/site-packages/github/PaginatedList.py", line 154, in totalCount
    "GET", self.__firstUrl, parameters=params, headers=self.__headers
  File "/usr/local/lib/python3.6/site-packages/github/Requester.py", line 319, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/usr/local/lib/python3.6/site-packages/github/Requester.py", line 342, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.RateLimitExceededException: 403 {"documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#abuse-rate-limits", "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."}

[EnricoMi]

This is annoying, thanks for raising this. Problematic is that 403 means "Forbidden", which is not retryable. But the type of exception seems to give a good indication that this is something to retry later. Let me try to sketch out a solution.

[EnricoMi]

@jaqx0r it would we interesting to understand what caused the rate limit exceeded in the first place? Do you see this issue regularily? Is this caused by the rate limit of your token or by the abuse rate limit. The exception message implies the latter.

Maybe in addition retrying those exceptions with some backoff, the action should also throttle down its requests in general to avoid the abuse rate limit. Maybe this is caused by other actions using the same toking as they run in the same workflow.

[EnricoMi]

The action sends requests to GitHub API as quickly as possible. This seems to trigger the abuse rate limit which caused the exception in the first place. I have added a throttling so that GET requests are at least 0.25 seconds apart and write requests 2.0 seconds apart (both used to be less than 0.01 seconds). This should avoid the rate abuse error. I will try to get this logic moved upstream into the PyGitHub client.

Please test this fix by using:

uses: EnricoMi/publish-unit-test-result-action/composite@master

Please let me know if the default throttling settings are still insufficient to avoid abuse rate limits.

I agree that the above exception should be retried, but that should also be done by the PyGitHub client, as there is already code that identifies that situation, as well as urllib already considering retry-after header values.

[EnricoMi]

Released with latest v1 release v1.19:

uses: EnricoMi/publish-unit-test-result-action@v1

[jaqx0r]

Thanks!

…

On Thu, 8 Jul 2021, 01:18 Enrico Minack, ***@***.***> wrote: Released with latest v1 release v1.19: uses: ***@***.*** — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXFX65MNOBP5BL2LGU5GMLTWRV45ANCNFSM47WE52ZA> .

[dan-ih]

This seems to be happening still in 2.4.2, but probably because we are running many of these publish-unit-test-result-actions in parallel. Backoff/retry logic for 403s would be really helpful. @EnricoMi any chance that was considered, or any chance its easy to implement?

[EnricoMi]

Can you share some logs please?

[jaqx0r]

This comment retracted; I realised I was running an older version.