Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

Closed
bencaine1 opened this issue Sep 11, 2018 · 12 comments
Closed

BigQuery: Add BadGateway to DEFAULT_RETRY predicate #5918

bencaine1 opened this issue Sep 11, 2018 · 12 comments
Assignees
@tseaver
Labels
api: bigquery Issues related to the BigQuery API. backend priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@bencaine1
Copy link

OS: Linux dc32b7e8763a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.5.0

I'm getting BadGateway exceptions periodically when running queries with the default retry object. Here's an example (relevant part of the stack trace):

  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 2605, in done
    location=self.location)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 579, in _get_query_results
    retry, method='GET', path=path, query_params=extra_params)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 331, in _call_api
    return call()
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
BadGateway: 502 GET https://www.googleapis.com/bigquery/v2/projects/packard-campbell-synth/queries/c4af3c68-d204-455e-877f-9f81e1c77d68?maxResults=0: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

It would be useful to add this to the DEFAULT_RETRY object in https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/retry.py.
@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Sep 11, 2018
@tseaver tseaver mentioned this issue Sep 11, 2018
Merged
tseaver added a commit that referenced this issue Sep 11, 2018
@tseaver
Retry '502 Bad Gateway' errors by default. (#5930)
f3e00ad 
Closes #5918.
@bencaine1
Copy link
Author

I've patched the library at commit f3e00ad, and we're still getting BadGateways.

I think it's because the BadGateway is raised before the job is done, and therefore looking at the job's errors isn't going to show BadGateway. So I believe you'll need to catch BadGateway using something like the if_exception_type method (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L78).

@tseaver
Copy link
Contributor

tseaver commented Sep 13, 2018

@bencaine1 Thanks for the followup. The issue is that the BadGateway you report is being raised from Client._get_query_results, which doesn't take a retry argument.

@tseaver tseaver reopened this Sep 13, 2018
@tseaver
Copy link
Contributor

tseaver commented Sep 13, 2018

Oops, Client._get_query_results does take a retry argument, and QueryJob.result passes it.

@tseaver
Copy link
Contributor

tseaver commented Sep 13, 2018

@bencaine1 Can you manage to trap the code when the BadGateway is raised, and print out the full response body?

@tswast Any clues from your side how we should be handling this?

@tswast
Copy link
Contributor

tswast commented Sep 13, 2018

It could be that errors or reason isn't getting populated? I think 502 comes from the layer before it actually hit's BigQuery, so maybe the BigQuery team's advice to look at the body rather than error codes shouldn't apply for 502?

@tswast
Copy link
Contributor

tswast commented Sep 13, 2018

CC @shollyman

@tswast
Copy link
Contributor

tswast commented Sep 13, 2018

From the error message, it looks like the body is HTML, not JSON, so probably the reason field isn't populated.

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/api_core/google/api_core/exceptions.py#L362-L388

@shollyman
Copy link
Contributor

Yes, this is likely a GFE response, which means we're not going to get a structured error back.

@tseaver tseaver self-assigned this Sep 18, 2018
@tseaver tseaver mentioned this issue Sep 18, 2018
Merged
tseaver added a commit that referenced this issue Sep 18, 2018
@tseaver
Add support for retrying unstructured 429 / 500 / 502 responses. (#6011)
103ced7 
Closes #5918.
@bencaine1
Copy link
Author

Thanks! We've patched this version and we haven't seen any BadGateways yet. Will keep you posted :)

@bencaine1
Copy link
Author

bencaine1 commented Oct 4, 2018 •
edited
Loading

Sorry to reopen a closed issue, but we just got another BadGateway.

Stack trace looks similar:

  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1232, in query
    query_job._begin(retry=retry)
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 543, in _begin
    method='POST', path=path, data=self._build_resource())
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 331, in _call_api
    return call()
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/opt/conda/lib/python2.7/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadGateway: 502 POST https://www.googleapis.com/bigquery/v2/projects/packard-campbell-synth/jobs: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

Something I just noticed about this stack trace: it's failing on this line in api_core/retry.py:

  File "/opt/conda/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()

But that line is inside a try/except, where the except clause catches all Exceptions. So maybe this isn't even raising an Exception and going into the retry logic?

@bencaine1
Copy link
Author

@tseaver, PTAL when you get a chance. Thanks!

@tseaver
Copy link
Contributor

tseaver commented Oct 22, 2018

@bencaine1 is your application passing a non-default retry argument to client.query?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tseaver tseaver

Labels
api: bigquery Issues related to the BigQuery API. backend priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tseaver @tswast @shollyman @bencaine1