S3 export failure "RequestError: send request failed"

[mightyguava]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

I'm running a dumpling export against S3 on a very large database (5TB).

2. What did you expect to see?

Successful export

3. What did you see instead?

Received a

4. Versions of the cluster

   • Dumpling version (run dumpling -V):

     v4.0.7-9-gb84f64f
     

   • Source database version (execute SELECT version(); in a MySQL client):

     Aurora 5.7.12
     

5. Operation logs

Git commit hash: b84f64ff362cedcb795aa23fa1188ba7b7c9a7d7
Git branch:      master
Build timestamp: 2020-10-27 04:21:05Z
Go version:      go version go1.15.3 linux/amd64

[2020/10/30 16:16:55.136 +00:00] [INFO] [config.go:180] ["detect server type"] [type=MySQL]
[2020/10/30 16:16:55.136 +00:00] [INFO] [config.go:198] ["detect server version"] [version=5.7.12-log]
[2020/10/30 16:16:59.156 +00:00] [INFO] [ir_impl.go:208] ["get estimated rows count"] [estimateCount=2918441068]
[2020/10/30 16:57:13.163 +00:00] [ERROR] [writer_util.go:350] ["writing failed"] [string="<REDACTED>"] [writer="&export.InterceptFileWriter{Writer:(*storage.uploaderWriter)(0xc06691c680), Once:sync.Once{done:0x1, m:sync.Mutex{state:0, sema:0x0}}, initRoutine:(func(context.Context) error)(0xf6d100), err:error(nil), SomethingIsWritten:true}"] [error="RequestError: send request failed\ncaused by: Put \"https://s3.us-west-2.amazonaws.com<PREFIX>/<TABLE>.4818.sql?partNumber=47&uploadId=hGlqlOrmk_CFgZySeien404sQYuxds59PllLdZeGLUxNNDfyAwWi.koWS.jyw_s_XDfDhCxHf_7a7kulfV7ELlpWJCKvXpx6l2aTb4ufGd8BMX1c5ERSUF3s9FW1sLQk9cEcH7hcLKVTAsZcZcdDaw--\": write tcp 10.137.13.153:56172->52.218.221.176:443: use of closed network connection"] [stack="github.com/pingcap/dumpling/v4/export.writeBytes\n\t/go/dumpling/v4/export/writer_util.go:350\ngithub.com/pingcap/dumpling/v4/export.(*writerPipe).Run\n\t/go/dumpling/v4/export/writer_util.go:66\ngithub.com/pingcap/dumpling/v4/export.WriteInsert.func1\n\t/go/dumpling/v4/export/writer_util.go:137"]
[2020/10/30 18:23:08.973 +00:00] [INFO] [ir_impl.go:208] ["get estimated rows count"] [estimateCount=61042457]
[2020/10/30 18:23:10.001 +00:00] [INFO] [ir_impl.go:208] ["get estimated rows count"] [estimateCount=425612999]
[2020/10/30 18:31:43.140 +00:00] [ERROR] [main.go:230] ["dump failed error stack info"] [error="RequestError: send request failed\ncaused by: Put \"https://s3.us-west-2.amazonaws.com/<PREFIX>/<TABLE>.4818.sql?partNumber=47&uploadId=hGlqlOrmk_CFgZySeien404sQYuxds59PllLdZeGLUxNNDfyAwWi.koWS.jyw_s_XDfDhCxHf_7a7kulfV7ELlpWJCKvXpx6l2aTb4ufGd8BMX1c5ERSUF3s9FW1sLQk9cEcH7hcLKVTAsZcZcdDaw--\": write tcp 10.137.13.153:56172->52.218.221.176:443: use of closed network connection"] [stack="main.main\n\t/go/dumpling/cmd/dumpling/main.go:230\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"]

dump failed: RequestError: send request failed
caused by: Put "https://s3.us-west-2.amazonaws.com/<PREFIX>/<TABLE>.4818.sql?partNumber=47&uploadId=hGlqlOrmk_CFgZySeien404sQYuxds59PllLdZeGLUxNNDfyAwWi.koWS.jyw_s_XDfDhCxHf_7a7kulfV7ELlpWJCKvXpx6l2aTb4ufGd8BMX1c5ERSUF3s9FW1sLQk9cEcH7hcLKVTAsZcZcdDaw--": write tcp 10.137.13.153:56172->52.218.221.176:443: use of closed network connection

6. Configuration of the cluster and the task

        --consistency=lock
        -o=s3://<PREFIX>?region=us-west-2
        --rows=200000
        --threads=128

[lichunzhu]

@mightyguava We have added more retry mechanism for this situation, including increasing the aws maxRetries, retrying on failed chunks, and increasing the maxIdlePerConn for HTTP client.
We also tried to kill the socket connection when dumpling is running, but we didn't reproduce this error. However, this dumpling can still dump and automatically retry even if the connection is closed.
Did the situation described in the issue happen frequently? If so, would you like to test the dumpling on this branch?

[mightyguava]

Sure I can test this tomorrow. Thanks for putting in a fix so quickly! Our export takes several hours to run so I’ve only seen this issue once.

[mightyguava]

This appears to be an issue that was fixed in a recent version of aws sdk aws/aws-sdk-go#3406. The fix was in 1.34.3.

Looks like br has the fix pulled in on master so I might just need to run with a new dumpling build.

[lichunzhu]

Yes. Thanks for pointing this out!

[mightyguava]

It completed on the first attempt with no errors! 🎊

[lichunzhu]

Great! Did you use the dumpling built from #182?

[mightyguava]

Yes, I built dumpling from db9e827

[lichunzhu]

Get it.

[lichunzhu]

I will close this issue now. If this happened again you can reopen this issue again.