Problems uploading files to Cloud Storage: com.google.cloud.storage.StorageException: Error writing request body to server

[prismec]

We randomly get the exception stated below when uploading files to Cloud Storage. The file size does not seem to be the relevant factor - the files where we see the problems are in the range between 25MB and 22GB.

com.google.cloud.storage.StorageException: Error writing request body to server
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:704)
	at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
	at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:122)
	at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:149)
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
	at java.nio.channels.Channels.writeFully(Channels.java:101)
	at java.nio.channels.Channels.access$000(Channels.java:61)
	at java.nio.channels.Channels$1.write(Channels.java:174)
	at smarter.ecommerce.commons.io.DelegatingOutputStream.write(DelegatingOutputStream.java:27)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
        .......
        Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3518)
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3501)
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
	at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:685)
	... 85 more

We already had a look at the integrated retry handling and it looks like that exceptions of this kind are not retried when sending a chunk to Cloud Storage. Unfortunately it's very hard to reproduce with a test. But the problem occurs randomly, but frequently (multiple times a day)!

This might be related: https://stackoverflow.com/questions/50019241/google-cloud-storage-throwing-error-for-multiple-large-file-uploads-concurrently

The code that we are using is pretty much the same as stated in the linked stackoverflow article.

[prismec]

Access to Cloud Storage is done from outside of Google Cloud. Maybe this information helps in analysis.

[nitishkumar71]

I don't think this is an issue of access within google cloud or outside of it. because in case of stackoverflow question, access was being done from Google Cloud itself.

[Rajind]

We are experiencing the same issue randomly and our file sizes are less than 1MB. Further, uploading happens from a GCloud VM instance (i.e upload happens within the GCloud network itself). @prismec were you able to find a solution to this?

[prismec]

No update on our side but it happens so frequently that we had to rollback to another storage engine until a solution for this problem ist available.

[yihanzhen]

This sounds bad. I'll see if can dig up something wrong.

[prismec]

Any updates available on this issue ?

[yihanzhen]

I suspect this is a network issue and retrying the request would fix the issue. However I need to double check with the server side and waiting for their confirmation. Once they are on the same page the above PR can be merged.

[yihanzhen]

The change was included in the latest release 0.54.beta. Please update to the latest version of google-cloud-storage and let us know if the problem was still there.

[zguth]

I have a similar issue with a different exception:
com.google.cloud.storage.StorageException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed) at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220) at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:292) at com.google.cloud.storage.StorageImpl.create(StorageImpl.java:148) at com.google.cloud.storage.Bucket.create(Bucket.java:937) at org.sonatype.nexus.blobstore.gcloud.internal.GoogleCloudBlobStore.lambda$0(GoogleCloudBlobStore.java:158) at org.sonatype.nexus.blobstore.gcloud.internal.GoogleCloudBlobStore.createInternal(GoogleCloudBlobStore.java:402) at org.sonatype.nexus.blobstore.gcloud.internal.GoogleCloudBlobStore.create(GoogleCloudBlobStore.java:155)

What do you think, is it the same issue?

[yihanzhen]

@zguth Thanks for letting us know! Can you file a separate issue and provide a full stack trace?

[zguth]

Yes, it's here: #3678