Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: backup command with parameter s3 endpoint has suffix '/' #30138

Closed
wants to merge 39 commits into from
Closed
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
cc5c8d5
fix issue 27015
fengou1 Sep 27, 2021
65ccd96
Merge branch 'master' into issue_27015
fengou1 Sep 27, 2021
bd531c8
fix comments - add general function isRetryableError
fengou1 Sep 28, 2021
a3b0df0
Merge branch 'issue_27015' of https://github.com/fengou1/tidb into is…
fengou1 Sep 28, 2021
c1ddff9
Merge branch 'master' into issue_27015
fengou1 Sep 28, 2021
0fbd620
remote uncessary package errors
fengou1 Sep 28, 2021
0f1976c
Merge branch 'issue_27015' of https://github.com/fengou1/tidb into is…
fengou1 Sep 28, 2021
6dcdbdb
reused the retry code from lightning
fengou1 Sep 29, 2021
fe147b9
Merge branch 'master' into issue_27015
fengou1 Sep 29, 2021
8dcda02
Merge branch 'master' into issue_27015
fengou1 Oct 7, 2021
e97b450
refactoring retryable
fengou1 Oct 7, 2021
a825bea
Merge branch 'master' into issue_27015
fengou1 Oct 9, 2021
440cdf5
Merge branch 'master' into issue_27015
fengou1 Oct 12, 2021
c53ddc1
Merge branch 'master' into issue_27015
fengou1 Oct 12, 2021
d4004f3
backup failed summary should be quieter
fengou1 Oct 12, 2021
5a9ff74
Merge branch 'issue_27015' of https://github.com/fengou1/tidb into is…
fengou1 Oct 12, 2021
38e08e6
Merge branch 'master' into issue_27015
fengou1 Oct 12, 2021
afe3938
Merge branch 'master' into issue_27015
fengou1 Oct 12, 2021
ac36702
Merge branch 'master' into issue_27015
fengou1 Oct 12, 2021
07d1696
Merge branch 'master' into issue_27015
fengou1 Oct 13, 2021
b7ced92
Merge branch 'master' into issue_27015
fengou1 Oct 14, 2021
a1f292b
optimize error log
fengou1 Nov 10, 2021
b3f50cc
Merge branch 'master' into issue_27015
fengou1 Nov 10, 2021
fbdf057
fixed unit test for dumpling
fengou1 Nov 10, 2021
1a0001c
Merge branch 'master' into issue_27015
fengou1 Nov 15, 2021
c90fd96
Merge branch 'master' into issue_27015
fengou1 Nov 16, 2021
71d1d28
add a retry test
fengou1 Nov 22, 2021
001f048
Merge branch 'master' into issue_27015
fengou1 Nov 22, 2021
80cc581
fix endpoint with suffix "/"
fengou1 Nov 24, 2021
3f1e668
Merge branch 'pingcap:master' into issue_30104
fengou1 Nov 25, 2021
610467d
remove code in another pr
fengou1 Nov 25, 2021
dc74eb1
Merge branch 'issue_30104' of https://github.com/fengou1/tidb into is…
fengou1 Nov 25, 2021
728e942
fix build issue
fengou1 Nov 25, 2021
9e1928c
fix issue from comments
fengou1 Nov 25, 2021
31cd1e5
Merge branch 'master' into issue_30104
fengou1 Nov 25, 2021
35e9aeb
Merge branch 'master' into issue_30104
fengou1 Nov 28, 2021
4e1b453
Merge branch 'master' into issue_30104
fengou1 Nov 29, 2021
9b42b4a
remove useless code
fengou1 Nov 29, 2021
e676e69
Merge branch 'master' into issue_30104
fengou1 Nov 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion br/pkg/backup/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,13 @@ func (bc *Client) BackupRanges(
elctx := logutil.ContextWithField(ectx, logutil.RedactAny("range-sn", id))
err := bc.BackupRange(elctx, sk, ek, req, metaWriter, progressCallBack)
if err != nil {
return errors.Trace(err)
// The error due to context cancel, stack trace is meaningless, the stack shall be suspended (also clear)
if errors.Cause(err) == context.Canceled {
return errors.SuspendStack(err)
} else {
return errors.Trace(err)
}
fengou1 marked this conversation as resolved.
Show resolved Hide resolved

}
return nil
})
Expand Down
5 changes: 5 additions & 0 deletions br/pkg/storage/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,11 @@ func newS3Storage(backend *backuppb.S3, opts *ExternalStorageOptions) (*S3Storag
WithS3ForcePathStyle(qs.ForcePathStyle).
WithRegion(qs.Region)
request.WithRetryer(awsConfig, defaultS3Retryer())

if len(qs.Endpoint) > 0) {
strings.TrimSuffix(qs.Endpoint, "/")
fengou1 marked this conversation as resolved.
Show resolved Hide resolved
}

if qs.Endpoint != "" {
awsConfig.WithEndpoint(qs.Endpoint)
}
Expand Down
11 changes: 10 additions & 1 deletion br/pkg/summary/collector.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,14 @@
package summary

import (
"context"
"strings"
"sync"
"time"

"github.com/docker/go-units"
"github.com/pingcap/log"
"github.com/pkg/errors"
"go.uber.org/zap"
)

Expand Down Expand Up @@ -188,9 +190,16 @@ func (tc *logCollector) Summary(name string) {
}

if len(tc.failureReasons) != 0 || !tc.successStatus {
canceledUnits := make([]string, 0)
for unitName, reason := range tc.failureReasons {
logFields = append(logFields, zap.String("unit-name", unitName), zap.Error(reason))
if errors.Cause(reason) != context.Canceled {
logFields = append(logFields, zap.String("unit-name", unitName), zap.Error(reason))
} else {
canceledUnits = append(canceledUnits, unitName)
}
}
// only print total number of cancel unit
log.Info("units canceled", zap.Int("cancel-unit", len(canceledUnits)))
fengou1 marked this conversation as resolved.
Show resolved Hide resolved
tc.log(name+" failed summary", logFields...)
return
}
Expand Down