Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect input stream reading when retrying request #924

Closed
zoltan-mihalyi opened this issue May 7, 2020 · 4 comments · Fixed by #928
Closed

Incorrect input stream reading when retrying request #924

zoltan-mihalyi opened this issue May 7, 2020 · 4 comments · Fixed by #928

Comments

@zoltan-mihalyi
Copy link

zoltan-mihalyi commented May 7, 2020

When okhttp retries a failed request, minio keeps reading from the half-consumed InputStream, which causes the second request to be different (or EOFException is thrown).

Expected Behavior

The second request should be the same as the first

Current Behavior

The second request body contains incorrect data/fails with EOFException

Possible Solution

  • HttpRequestBody should reset the stream to the initial position on the second writeTo call.

Steps to Reproduce (for bugs)

  1. call minioClient.putObject with an InputStream
  2. destroy the first PUT request socket (i used a proxy server to achieve this)

My test code:

    public static void main(String[] args) throws Exception {
        MinioClient minioClient = new MinioClient("http://localhost:8000", "minioadmin", "minioadmin");
        minioClient.putObject("test", "obj", new FileInputStream("test.txt"), new PutObjectOptions(-1, 64 * 1024 * 1024));
    }

if the InputStream was smaller than the part size:

Exception in thread "main" java.io.EOFException
	at okio.RealBufferedSink.write(RealBufferedSink.java:115)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:78)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at io.minio.MinioClient.execute(MinioClient.java:1097)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePut(MinioClient.java:1410)
	at io.minio.MinioClient.executePut(MinioClient.java:1424)
	at io.minio.MinioClient.uploadPart(MinioClient.java:5492)
	at io.minio.MinioClient.putObject(MinioClient.java:3740)
	at io.minio.MinioClient.putObject(MinioClient.java:3870)

if the InputStream was biggerr than the part size:

Exception in thread "main" error occurred
ErrorResponse(code = XAmzContentSHA256Mismatch, message = The provided 'x-amz-content-sha256' header does not match what was computed., bucketName = test, objectName = obj, resource = /test/obj, requestId = 160CBA3FB738FCAB, hostId = 661bb666-f640-472d-97cd-5d9e9e830fe9)
request={method=PUT, url=http://localhost:8000/test/obj?uploadId=916089ad-5aa6-462b-9623-6f349c21aa5e&partNumber=2, headers=Host: localhost:8000
Accept-Encoding: identity
User-Agent: MinIO (amd64; amd64) minio-java/dev
Content-MD5: +VzuiidtaOn3F5hn/eQZ4w==
x-amz-content-sha256: 72673a4765445f2c0341de4b217bbb7619ef6593f891c277f6dadb743ecfcab0
x-amz-date: 20200507T110550Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200507/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*
}
response={code=400, headers=accept-ranges: bytes
content-length: 360
content-security-policy: block-all-mixed-content
content-type: application/xml
server: MinIO/RELEASE.2020-05-01T22-19-14Z
vary: Origin
x-amz-request-id: 160CBA3FB738FCAB
x-xss-protection: 1; mode=block
date: Thu, 07 May 2020 11:05:52 GMT
Connection: keep-alive
}

	at io.minio.MinioClient.execute(MinioClient.java:1211)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePut(MinioClient.java:1410)
	at io.minio.MinioClient.executePut(MinioClient.java:1424)
	at io.minio.MinioClient.uploadPart(MinioClient.java:5492)
	at io.minio.MinioClient.putObject(MinioClient.java:3740)
	at io.minio.MinioClient.putObject(MinioClient.java:3870)

Your Environment

  • Version used: 7.0.2
  • Environment name and version (e.g. nginx 1.9.1):
  • Server type and version: docker image: minio/minio:RELEASE.2020-05-06T23-23-25Z
  • Operating System and version: Windows 10
@klauspost klauspost transferred this issue from minio/minio May 7, 2020
@balamurugana
Copy link
Member

@zoltan-mihalyi Could you paste your client code causing the failure? If you use customOkHttpClient you can set false to retryOnConnectionFailure()

@zoltan-mihalyi
Copy link
Author

@balamurugana I updated the issue description to include my code.

I tried with a custom OkHttpClient instance, using retryOnConnectionFailure(false) which works as expected:
Connection reset by peer: socket write error

However, there are other cases when RequestBody#writeTo is called multiple times, for example following redirects (this can be turned off aswell)

This is why okhttp does not have a default InputStream-based RequestBody. square/okhttp#2424

@balamurugana
Copy link
Member

balamurugana commented May 7, 2020

@zoltan-mihalyi I don't see your code in the description. This is the reason MinioClient can be created with customOkHttpClient, so that user has full control on what to do on various cases.

@zoltan-mihalyi
Copy link
Author

@balamurugana sorry for that, forget to save.
I see your point. But minio should work correctly in this case, especially this is the default config.

  • The error messages are misleading.
  • The exception is thrown too late. Possibly after an 500MiB chunk is uploaded (which we shouldnt even try, because the stream is half-consumed).
  • There is no reason to configure customOkHttpClient to use retry, if minio fails to handle this situation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants