-
Notifications
You must be signed in to change notification settings - Fork 626
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3 client falling over after 4-5 hours of processing 200 files per minute #2657
Comments
Hi @JamieClayton7 , From reading about it in similar issues on github aws/aws-sdk-go#411 It seems like this error is being thrown when the I'm not sure what You can enable the SDK logger like so: cfg, err := config.LoadDefaultConfig(
context.TODO(),
config.WithRegion("us-east-1"), // or any other region
config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody),
)
if err != nil {
log.Fatal(err)
} Thanks, |
Thanks @RanVaknin As If the first X files we upload work correctly, and the remaining uploads work correctly after a brute force of retries, it suggests there is an issue server-side. The func ConvertNdJson[T interface{}](val *[]T) (io.Reader, error) {
b := make([]byte, 0)
w := bytes.NewBuffer(b)
enc := json.NewEncoder(w)
for _, item := range *val {
if err := enc.Encode(item); err != nil {
return nil, err
}
}
return w, nil
} |
Hi @JamieClayton7 , Thanks for the additional info. Based on what you mentioned wrt retries, it does sound like an intermittent service related issue. In order for this to be investigated by the S3 service team, I'll need a request ID of one of the failing requests. You can see the request ID if you enable the response logging. Thanks, |
Here are various failing response bodies with codes:
|
@RanVaknin is there any update to this? I've had to push a monkey patch to production to restart the container upon RequestTimeout to get around this problem. |
Hi @JamieClayton7 , Thanks for your patience. I had to synthesize all of the similar issues from the other SDKs, to make this ticket actionable by the S3 team. We have an internal ticket up with S3 (internal ticket I will do my best to keep you updated on the matter. Thanks, |
i also face this issue, the request id is SD8632ECNY3JT5GP if it's of any help. |
Acknowledgements
go get -u github.com/aws/aws-sdk-go-v2/...
)Describe the bug
My application polls SQS for messages generated by S3 event notification on ObjectCreated. The object is pulled using GetObject, we do some processing of the ndjson file and re-upload (PutObject) the file to the bucket with updated contents. This consistently works fine for 4-5 hours on average until the client seems to fall over, with every request failing with the following error message:
If I restart the service, we are back to working correctly. The same client is being used across a maximum of 50 goroutines. The service is running in AWS ECS Fargate with Go version 1.22.3.
Edit: For context - each file is ~2MB.
Expected Behavior
The S3 client to consistently PutObject without error.
Current Behavior
After a period of time (4-5 hours on average), the client begins to throw the same error for every PutObject operation:
Reproduction Steps
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2/service/s3 v1.54.2
github.com/aws/aws-sdk-go-v2 v1.27.0
Compiler and Version used
1.22.3
Operating System and version
Alpine 3.20 https://hub.docker.com/layers/library/alpine/3.20/images/sha256-216266c86fc4dcef5619930bd394245824c2af52fd21ba7c6fa0e618657d4c3b
The text was updated successfully, but these errors were encountered: