Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
When uploading large amounts of data to S3, we occasionally see failures where the AWS sdk tries to use a closed network connection. The upstream bug appears to be aws/aws-sdk-go#3406. I'm not sure why the error manifests but it's causing us significant pain. Rather than retry the entire base backup, we'll retry the WAL segment upload. ``` ERROR: 2020/08/06 08:39:52.198782 failed to upload 'basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br' to bucket 'S3_BUCKET': MultipartUpload: upload multipart failed caused by: RequestError: send request failed caused by: Put https://S3_BUCKET/basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br?partNumber=2: write tcp 10.64.18.161:42118->52.216.134.19:443: use of closed network connection ERROR: 2020/08/06 08:39:52.198805 upload: could not upload 'base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br' ERROR: 2020/08/06 08:39:52.198818 failed to upload 'basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br' to bucket 'S3_BUCKET': MultipartUpload: upload multipart failed caused by: RequestError: send request failed caused by: Put https://S3_BUCKET/basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br?partNumber=2 write tcp 10.64.18.161:42118->52.216.134.19:443: use of closed network connection ERROR: 2020/08/06 08:39:52.198833 Unable to complete uploads ```
- Loading branch information