API requests to /backups occasionally times out?

[Sumukh]

API requests to /backups occasionally (9 times in the past 6 days) sometimes result in a gunicorn worker timing out.

getsentry/raven-python#675

This timeout is specifically when gunicorn kills the worker because it was taking too long to respond. Maybe a busy loop or something like that. So this goes unnoticed.

It seems to only be in the backups API endpoint and one other response - so it's possible that these requests are just taking longer to fulfill.

A temporary fix is to increase the worker timeout to say 60 seconds instead of 30. Gunicorn seems to discourage this though.

Generally set to thirty seconds. Only set this noticeably higher if you’re sure of the repercussions for sync workers. For the non sync workers it just means that the worker process is still communicating and is not tied to the length of time required to handle a single request.

[knrafto]

So does that mean some requests to /backups take longer than 30 seconds? Why is that?

[Sumukh]

It's because the request to google to get auth information is taking too long.

from the logs/traceback:

flask_oauthlib Request 'https://www.googleapis.com/oauth2/v1/userinfo' with 'GET' method
10:40:14 requests GET https://www.googleapis.com/oauth2/v1/userinfo
10:40:45 exception SystemExit: 1

I think a solution would be to get userinfo from cache instead of hitting google everytime (with an expiry of an hour).