cmd/sync: add 'max-failure' to exit when the transfer encounters more than a certain number of errors

[zhijian-pro]

close #5140

[davies]

Maybe --max-errors or --max-failure? we have --max-failure for warmup

[zhijian-pro]

Maybe --max-errors or --max-failure? we have --max-failure for warmup

ok

[codecov]

Codecov Report

Attention: Patch coverage is 10.86957% with 41 lines in your changes missing coverage. Please review.

Project coverage is 44.60%. Comparing base (b6582db) to head (7b54930).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/sync/sync.go	0.00%	40 Missing ⚠️
pkg/sync/config.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5142      +/-   ##
==========================================
- Coverage   44.65%   44.60%   -0.05%     
==========================================
  Files         159      159              
  Lines       43347    43471     +124     
==========================================
+ Hits        19355    19389      +34     
- Misses      21566    21656      +90     
  Partials     2426     2426              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jiefenghuang]

for supporting fast failed, add context to worker may be more concise and simple

func worker(ctx context.Context, tasks <-chan object.Object, src, dst object.ObjectStorage, config *Config) {
    for {
        select {
            case <-ctx.Done():
                // cancel
                return
             case obj := <-tasks:
                 // origin logic
        }
        if failed >= xxx {
            ctx.cancel()
            ...
        }
    }
}

[zhijian-pro]

for supporting fast failed, add context to worker may be more concise and simple

func worker(ctx context.Context, tasks <-chan object.Object, src, dst object.ObjectStorage, config *Config) {
    for {
        select {
            case <-ctx.Done():
                // cancel
                return
             case obj := <-tasks:
                 // origin logic
        }
        if failed >= xxx {
            ctx.cancel()
            ...
        }
    }
}

ok

[jiefenghuang]

LGTM

[davies]

this may not work in cluster mode

[zhijian-pro]

for supporting fast failed, add context to worker may be more concise and simple

func worker(ctx context.Context, tasks <-chan object.Object, src, dst object.ObjectStorage, config *Config) {
    for {
        select {
            case <-ctx.Done():
                // cancel
                return
             case obj := <-tasks:
                 // origin logic
        }
        if failed >= xxx {
            ctx.cancel()
            ...
        }
    }
}

Let's do it the easy way, it's a little bit more complicated