Incorrect input stream reading when retrying request

[zoltan-mihalyi]

When okhttp retries a failed request, minio keeps reading from the half-consumed InputStream, which causes the second request to be different (or EOFException is thrown).

Expected Behavior

The second request should be the same as the first

Current Behavior

The second request body contains incorrect data/fails with EOFException

Possible Solution

• HttpRequestBody should reset the stream to the initial position on the second writeTo call.

Steps to Reproduce (for bugs)

1. call minioClient.putObject with an InputStream
2. destroy the first PUT request socket (i used a proxy server to achieve this)

My test code:

    public static void main(String[] args) throws Exception {
        MinioClient minioClient = new MinioClient("http://localhost:8000", "minioadmin", "minioadmin");
        minioClient.putObject("test", "obj", new FileInputStream("test.txt"), new PutObjectOptions(-1, 64 * 1024 * 1024));
    }

if the InputStream was smaller than the part size:

Exception in thread "main" java.io.EOFException
	at okio.RealBufferedSink.write(RealBufferedSink.java:115)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:78)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at io.minio.MinioClient.execute(MinioClient.java:1097)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePut(MinioClient.java:1410)
	at io.minio.MinioClient.executePut(MinioClient.java:1424)
	at io.minio.MinioClient.uploadPart(MinioClient.java:5492)
	at io.minio.MinioClient.putObject(MinioClient.java:3740)
	at io.minio.MinioClient.putObject(MinioClient.java:3870)

if the InputStream was biggerr than the part size:

Exception in thread "main" error occurred
ErrorResponse(code = XAmzContentSHA256Mismatch, message = The provided 'x-amz-content-sha256' header does not match what was computed., bucketName = test, objectName = obj, resource = /test/obj, requestId = 160CBA3FB738FCAB, hostId = 661bb666-f640-472d-97cd-5d9e9e830fe9)
request={method=PUT, url=http://localhost:8000/test/obj?uploadId=916089ad-5aa6-462b-9623-6f349c21aa5e&partNumber=2, headers=Host: localhost:8000
Accept-Encoding: identity
User-Agent: MinIO (amd64; amd64) minio-java/dev
Content-MD5: +VzuiidtaOn3F5hn/eQZ4w==
x-amz-content-sha256: 72673a4765445f2c0341de4b217bbb7619ef6593f891c277f6dadb743ecfcab0
x-amz-date: 20200507T110550Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200507/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*
}
response={code=400, headers=accept-ranges: bytes
content-length: 360
content-security-policy: block-all-mixed-content
content-type: application/xml
server: MinIO/RELEASE.2020-05-01T22-19-14Z
vary: Origin
x-amz-request-id: 160CBA3FB738FCAB
x-xss-protection: 1; mode=block
date: Thu, 07 May 2020 11:05:52 GMT
Connection: keep-alive
}

	at io.minio.MinioClient.execute(MinioClient.java:1211)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePut(MinioClient.java:1410)
	at io.minio.MinioClient.executePut(MinioClient.java:1424)
	at io.minio.MinioClient.uploadPart(MinioClient.java:5492)
	at io.minio.MinioClient.putObject(MinioClient.java:3740)
	at io.minio.MinioClient.putObject(MinioClient.java:3870)

Your Environment

• Version used: 7.0.2
• Environment name and version (e.g. nginx 1.9.1):
• Server type and version: docker image: minio/minio:RELEASE.2020-05-06T23-23-25Z
• Operating System and version: Windows 10

[balamurugana]

@zoltan-mihalyi Could you paste your client code causing the failure? If you use customOkHttpClient you can set false to retryOnConnectionFailure()

[zoltan-mihalyi]

@balamurugana I updated the issue description to include my code.

I tried with a custom OkHttpClient instance, using retryOnConnectionFailure(false) which works as expected:
Connection reset by peer: socket write error

However, there are other cases when RequestBody#writeTo is called multiple times, for example following redirects (this can be turned off aswell)

This is why okhttp does not have a default InputStream-based RequestBody. square/okhttp#2424

[balamurugana]

@zoltan-mihalyi I don't see your code in the description. This is the reason MinioClient can be created with customOkHttpClient, so that user has full control on what to do on various cases.

[zoltan-mihalyi]

@balamurugana sorry for that, forget to save.
I see your point. But minio should work correctly in this case, especially this is the default config.

• The error messages are misleading.
• The exception is thrown too late. Possibly after an 500MiB chunk is uploaded (which we shouldnt even try, because the stream is half-consumed).
• There is no reason to configure customOkHttpClient to use retry, if minio fails to handle this situation