Storage: 416 exception in download with chunked size for 0 byte files

[inferno-chromium]

Environment details

1. Specify the API at the beginning of the title : Storage
2. OS type and version: Linux, Ubuntu 16
3. Python version and virtual environment information: 2.7.15
4. google-cloud-storage-1.12.0

Steps to reproduce

  bucket = client.bucket('clusterfuzz-data')
  blob = bucket.blob('/zero.file', chunk_size=1024 * 1024 * 100)
  blob.download_to_filename('any_local_path')

google.api_core.exceptions.RequestRangeNotSatisfiable: 416 GET https://www.googleapis.com/download/storage/v1/b/clusterfuzz-data/o/zero.file?alt=media: (u'Request failed with status code', 416, u'Expected one of', 200, 206)

Need to fix this similar to googleapis/google-cloud-go#242

Can you suggest a workaround or do a release for this. this is blocking our switch to chunked size.

Orthogonal question: I also wonder why there is no default chunked size, we were hitting silent ooms with python process failures since default chunk size was not set. We were trying this on like 20 gb file.

[inferno-chromium]

@javisantana who seems to have implemented range downloads in 44cebf8#diff-0e25e032e1cb97e8a46cc72571c09e2f

[tseaver]

I can reproduce this exception. The issue is in google.resumable_media, where the ChunkedResponse is sending the following header: Range: bytes=0-104857599. The server is interpreting that as an "absolute" demand, rather than an "at-most this many bytes" constraint.

The API docs for the Range header point to RFC 7233 for the download semantics. Per that RFC, the server is sending back the header, Content-Range: bytes */0, which specifies a range which would be satisfied.

ISTM that google.resumable_media could retry the request after the 416 response, parsing that Content-Range header to get the "correct" maximum size. @frankyn can you confirm? Or is the back-end being too strict in how it responds to the chunk request?

[frankyn]

Hi @coryan, can you help with this?

[coryan]

Retrying with a different range sounds good, with the caveat that somehow the application should be able to say "all or nothing", I anticipate applications that need that many bytes or cannot do any work, and retrying is wasteful in that case.

Also (as you probably know already) there is nothing to retry if the response includes Content-Range: bytes */0.

[tseaver]

@inferno-chromium The fix will be available in an upcoming release of google-resumable-media.

[inferno-chromium]

Thanks!