lightning: The performance drop 100% from 470s to 970s

[Yui-Song]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. deploy a tidb cluster with 1tidb and 6 tikv
2. prepare data of ycsb with configurations:

-p recordcount=100000000 -p operationcount=1000000 -p workload=core -p fieldcount=10 -p fieldlength=100 -p requestdistribution=uniform1

4. export the data with dumpling
5. import the data into a new deploy cluster with lightning

2. What did you expect to see? (Required)

No performance regression

3. What did you see instead (Required)

1. With the default configuration, the performance drops 100%, the duration of import increases from 470s to 560+s.
   

2. With lightning setting duplicate-resolution = "remove", the performance drops 100%, the duration of import increases from 470s to 970s.
   

4. What is your TiDB version? (Required)

bad commit: 89bf743
good commit: 0c2d07d

[D3Hunter]

the issue might caused by checksum-via-sql, but i tries to run with either checksum-via-sql set to false or true cannot reproduce the perf degradation in the same env.(after testbed workflow finished)

will try to run with checksum-via-sql=false in the same testbed workflow

[D3Hunter]

the slow might caused by unstable test env, there's no much change in lightning in those days, but it's fast in 2023-07-02 22:36:53.000 ~ 2023-07-04 03:03:59.000

[Yui-Song]

/close

[ti-chi-bot]

@Yui-Song: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[D3Hunter]

#44887 hasn't merged into 6.5 branch, so this issue doesn't affect it