Dumpling or TiDB needs to improve table scan capability to avoid TiDB OOM

[cyliu0]

Feature Request

Is your feature request related to a problem? Please describe:

During testing dumpling w/ Dynamic Regions feature, which means the tidb cluster config contains this

PD: 
config: |
          scheduler.max-merge-region-size = 400
          scheduler.max-merge-region-keys = 10240000
TiKV:
config: |
          coprocessor.region-split-size = "1000M"
          coprocessor.region-max-size = "1500M"
          coprocessor.region-split-keys = 96000000
          coprocessor.region-max-keys = 144000000
          coprocessor.enable-region-bucket = true
          coprocessor.region-bucket-size = "128M"
          raftstore.raft-log-gc-size-limit = "7G"
          raftstore.raft-log-gc-count-limit = 7340032
          server.max-snapshot-file-raw-size = "128MB"

The tikv region sizes are > 1200MB.
The tidb-server was OOM when dump with 10 threads /dumpling -h "downstream-tidb.brie-acceptance-long-time-tps-810231-1-41.svc" -P "4000" -u "root" --output "/dump_from_downstream" --filetype "csv" --filesize "256M" --rows "10000" --threads "10" --database "H". The same command will be just fine with region sizes are < 250MB. After I reduce the threads to 4, there is no OOM anymore.

Describe the feature you'd like:

I'd like one of the following features, which might solve the problem:

1. Dumpling adapts the threads parameter automatically according to the TiDB Memory & Region Size to avoid TiDB OOM
2. TiDB improves the table scan capability to avoid TiDB OOM

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

tidb log

2022/05/19 06:54:02.937 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:10.320311672s txnStartTS:433309150543347713 region_id:108941 store_addr:downstream-tikv-1.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:8240 kv_wait_ms:0 kv_read_ms:5225 processed_versions:2240797 total_versions:2240802 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4481592 rocksdb_cache_hit_count:35 rocksdb_read_count:17823 rocksdb_read_byte:714680035"] [conn=6844325742487011761]
[2022/05/19 06:54:02.937 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1315386106] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:05.576 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:12.95942351s txnStartTS:433309150543347713 region_id:108937 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:10228 kv_wait_ms:0 kv_read_ms:6503 processed_versions:2244301 total_versions:2244306 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4488600 rocksdb_cache_hit_count:34 rocksdb_read_count:17852 rocksdb_read_byte:715812248"] [conn=6844325742487011759]
[2022/05/19 06:54:05.576 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1317441626] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:10.523 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:17.905938623s txnStartTS:433309150543347713 region_id:108933 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:12574 kv_wait_ms:0 kv_read_ms:8047 processed_versions:2248515 total_versions:2248520 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4497028 rocksdb_cache_hit_count:35 rocksdb_read_count:17880 rocksdb_read_byte:717054844"] [conn=6844325742487011777]
[2022/05/19 06:54:10.523 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1319909496] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:12.818 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:15.61202841s txnStartTS:433309150543347713 region_id:108949 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:9206 kv_wait_ms:0 kv_read_ms:5770 processed_versions:2237333 total_versions:2237338 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4474664 rocksdb_cache_hit_count:37 rocksdb_read_count:17793 rocksdb_read_byte:713510707"] [conn=6844325742487011765]
[2022/05/19 06:54:12.818 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1313357045] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:12.948 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:20.330461005s txnStartTS:433309150543347713 region_id:108945 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:13309 kv_wait_ms:0 kv_read_ms:8047 processed_versions:2242600 total_versions:2242605 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4485198 rocksdb_cache_hit_count:35 rocksdb_read_count:17838 rocksdb_read_byte:715255815"] [conn=6844325742487011763]
[2022/05/19 06:54:12.949 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1316439747] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:13.240 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:15.033448311s txnStartTS:433309150543347713 region_id:108953 store_addr:downstream-tikv-3.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:7372 kv_wait_ms:0 kv_read_ms:3716 processed_versions:2244316 total_versions:2244321 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4488630 rocksdb_cache_hit_count:17885 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=6844325742487011767]
[2022/05/19 06:54:13.240 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1317457204] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:13.450 +00:00] [INFO] [session.go:3414] ["use snapshot schema"] [conn=6844325742487011761] [schemaVersion=51]
[2022/05/19 06:54:13.450 +00:00] [INFO] [session.go:3414] ["use snapshot schema"] [conn=6844325742487011761] [schemaVersion=51]
[2022/05/19 06:54:13.451 +00:00] [INFO] [session.go:3414] ["use snapshot schema"] [conn=6844325742487011761] [schemaVersion=51]
[2022/05/19 06:54:14.277 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:15.45340872s txnStartTS:433309150543347713 region_id:108965 store_addr:downstream-tikv-1.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:8415 kv_wait_ms:0 kv_read_ms:5285 processed_versions:2236597 total_versions:2236602 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4473192 rocksdb_cache_hit_count:37 rocksdb_read_count:17786 rocksdb_read_byte:713265412"] [conn=6844325742487011773]
[2022/05/19 06:54:14.277 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1312918108] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:14.672 +00:00] [WARN] [memory_usage_alarm.go:140] ["tidb-server has the risk of OOM. Running SQLs and heap profile will be recorded in record path"] ["is server-memory-quota set"=false] ["system memory total"=17179869184] ["system memory usage"=14486913024] ["tidb-server memory usage"=15089110912] [memory-usage-alarm-ratio=0.8] ["record path"="/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record"]
[2022/05/19 06:54:14.798 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:16.205983764s txnStartTS:433309150543347713 region_id:108957 store_addr:downstream-tikv-1.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:8597 kv_wait_ms:1 kv_read_ms:5288 processed_versions:2237876 total_versions:2237881 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4475750 rocksdb_cache_hit_count:37 rocksdb_read_count:17799 rocksdb_read_byte:713678519"] [conn=6844325742487011771]
[2022/05/19 06:54:14.798 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1313670375] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:15.707 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:16.448130941s txnStartTS:433309150543347713 region_id:108969 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:11292 kv_wait_ms:0 kv_read_ms:7440 processed_versions:2227968 total_versions:2227973 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4455934 rocksdb_cache_hit_count:40 rocksdb_read_count:17715 rocksdb_read_byte:710425882"] [conn=6844325742487011775]
[2022/05/19 06:54:15.707 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1307847699] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]
[2022/05/19 06:54:15.902 +00:00] [INFO] [coprocessor.go:852] ["[TIME_COP_PROCESS] resp_time:17.212196365s txnStartTS:433309150543347713 region_id:108961 store_addr:downstream-tikv-2.downstream-tikv-peer.brie-acceptance-long-time-tps-810231-1-41.svc:20160 kv_process_ms:11995 kv_wait_ms:0 kv_read_ms:8016 processed_versions:2234683 total_versions:2234688 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:4469364 rocksdb_cache_hit_count:36 rocksdb_read_count:17774 rocksdb_read_byte:712661853"] [conn=6844325742487011769]
[2022/05/19 06:54:15.902 +00:00] [INFO] [coprocessor.go:1259] ["memory exceeds quota, destroy one token now."] [consumed=1311787278] [quota=1073741824] ["total token count"=2] ["remaining token count"=2]

dumpling log

/ # /dumpling  -h "downstream-tidb.brie-acceptance-long-time-tps-810231-1-41.svc" -P "4000" -u "root" --output "/dump_from_downstream" -
-filetype "csv" --filesize "256M" --rows "10000" --threads "10" --database "H"
Release version: v6.1.0-nightly
Git commit hash: 828a255111cc7aab19754e092062d81e845123ff
Git branch:      heads/refs/tags/v6.1.0-nightly
Build timestamp: 2022-05-17 10:57:01Z
Go version:      go version go1.18.2 linux/amd64

[2022/05/19 06:53:51.725 +00:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=v6.1.0-nightly] ["Git Commit Hash"=828a255111cc7aab19754e092062d81e845123ff] ["Git Branch"=heads/refs/tags/v6.1.0-nightly] ["Build timestamp"="2022-05-17 10:57:01"] ["Go Version"="go version go1.18.2 linux/amd64"]
[2022/05/19 06:53:51.729 +00:00] [INFO] [version.go:362] ["detect server version"] [type=TiDB] [version=6.1.0-nightly]
[2022/05/19 06:53:51.759 +00:00] [INFO] [client.go:392] ["[pd] create pd client with endpoints"] [pd-address="[downstream-pd-0.downstream-pd-peer.brie-acceptance-long-time-tps-810231-1-41.svc:2379]"]
[2022/05/19 06:53:51.762 +00:00] [INFO] [base_client.go:350] ["[pd] switch leader"] [new-leader=http://downstream-pd-0.downstream-pd-peer.brie-acceptance-long-time-tps-810231-1-41.svc:2379] [old-leader=]
[2022/05/19 06:53:51.763 +00:00] [INFO] [base_client.go:105] ["[pd] init cluster id"] [cluster-id=7098902460217101368]
[2022/05/19 06:53:51.763 +00:00] [INFO] [client.go:687] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/05/19 06:53:51.763 +00:00] [INFO] [dump.go:1420] ["generate dumpling gc safePoint id"] [id=dumpling_1652943231763904994]
[2022/05/19 06:53:51.766 +00:00] [INFO] [dump.go:116] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"azblob\":{\"endpoint\":\"\",\"account-name\":\"\",\"account-key\":\"\",\"access-tier\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoSequences\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"downstream-tidb.brie-acceptance-long-time-tps-810231-1-41.svc\",\"Port\":4000,\"Threads\":10,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"/dump_from_downstream\",\"StatusAddr\":\":8281\",\"Snapshot\":\"433309150543347713\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[\"H\"],\"Where\":\"\",\"FileType\":\"csv\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"6.1.0-nightly\",\"HasTiKV\":true},\"Rows\":10000,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":268435456,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"433309150543347713\"},\"Tables\":{},\"CollationCompatible\":\"loose\"}"]
[2022/05/19 06:53:52.612 +00:00] [INFO] [writer.go:254] ["no data written in table chunk"] [database=H] [table=t_o_shard] [chunkIdx=0]
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:37: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[mysql] 2022/05/19 06:54:18 packets.go:73: unexpected EOF
[mysql] 2022/05/19 06:54:18 packets.go:428: busy buffer
[2022/05/19 06:54:18.424 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=217114] ["finished size"=120699984] [error="invalid connection"]
[2022/05/19 06:54:18.425 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=91220] ["finished size"=50803135] [error="invalid connection"]
[2022/05/19 06:54:18.425 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=134727] ["finished size"=75033618] [error="invalid connection"]
[2022/05/19 06:54:18.425 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=113839] ["finished size"=63286449] [error="invalid connection"]
[2022/05/19 06:54:18.425 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=54530] ["finished size"=30370497] [error="invalid connection"]
[2022/05/19 06:54:18.425 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=382017] ["finished size"=212756034] [error="invalid connection"]
[2022/05/19 06:54:18.426 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=163536] ["finished size"=91077340] [error="invalid connection"]
[2022/05/19 06:54:18.426 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=177811] ["finished size"=99028308] [error="invalid connection"]
[2022/05/19 06:54:18.426 +00:00] [WARN] [writer_util.go:319] ["fail to dumping table(chunk), will revert some metrics and start a retry if possible"] [database=H] [table=t_o_shard] ["finished rows"=289924] ["finished size"=161467035] [error="invalid connection"]
[2022/05/19 06:54:18.832 +00:00] [INFO] [collector.go:204] ["units canceled"] [cancel-unit=0]
[2022/05/19 06:54:18.832 +00:00] [INFO] [collector.go:205] ["backup failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [unit-name="dump table data"] [error="sql: SELECT * FROM `H`.`t_o_shard` WHERE (`col1`>2747766125656 and `col1`<2992213486942)or(`col1`=2747766125656 and(`col2`>88780951220974 or(`col2`=88780951220974 and `col3`>=25798133891694)))or(`col1`=2992213486942 and(`col2`<150421408843347 or(`col2`=150421408843347 and `col3`<201725852064984)))  ORDER BY `col1`,`col2`,`col3`: invalid connection; dial tcp 10.104.110.253:4000: connect: connection refused; dial tcp 10.104.110.253:4000: connect: connection refused"] [errorVerbose="the following errors occurred:\n -  invalid connection\n    sql: SELECT * FROM `H`.`t_o_shard` WHERE (`col1`>2747766125656 and `col1`<2992213486942)or(`col1`=2747766125656 and(`col2`>88780951220974 or(`col2`=88780951220974 and `col3`>=25798133891694)))or(`col1`=2992213486942 and(`col2`<150421408843347 or(`col2`=150421408843347 and `col3`<201725852064984)))  ORDER BY `col1`,`col2`,`col3`\n    github.com/pingcap/tidb/dumpling/export.(*tableData).Start\n    \tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:210\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:202\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571\n -  dial tcp 10.104.110.253:4000: connect: connection refused\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.Trace\n    \tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\n    github.com/pingcap/tidb/dumpling/export.createConnWithConsistency\n    \tgithub.com/pingcap/tidb/dumpling/export/sql.go:886\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).Dump.func3\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:198\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:196\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571\n -  dial tcp 10.104.110.253:4000: connect: connection refused\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.Trace\n    \tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\n    github.com/pingcap/tidb/dumpling/export.createConnWithConsistency\n    \tgithub.com/pingcap/tidb/dumpling/export/sql.go:886\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).Dump.func3\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:198\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:196\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571"]
[2022/05/19 06:54:18.832 +00:00] [INFO] [client.go:768] ["[pd] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
[2022/05/19 06:54:18.832 +00:00] [INFO] [client.go:706] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/05/19 06:54:18.832 +00:00] [ERROR] [main.go:76] ["dump failed error stack info"] [error="sql: SELECT * FROM `H`.`t_o_shard` WHERE (`col1`>2747766125656 and `col1`<2992213486942)or(`col1`=2747766125656 and(`col2`>88780951220974 or(`col2`=88780951220974 and `col3`>=25798133891694)))or(`col1`=2992213486942 and(`col2`<150421408843347 or(`col2`=150421408843347 and `col3`<201725852064984)))  ORDER BY `col1`,`col2`,`col3`: invalid connection; dial tcp 10.104.110.253:4000: connect: connection refused; dial tcp 10.104.110.253:4000: connect: connection refused"] [errorVerbose="the following errors occurred:\n -  invalid connection\n    sql: SELECT * FROM `H`.`t_o_shard` WHERE (`col1`>2747766125656 and `col1`<2992213486942)or(`col1`=2747766125656 and(`col2`>88780951220974 or(`col2`=88780951220974 and `col3`>=25798133891694)))or(`col1`=2992213486942 and(`col2`<150421408843347 or(`col2`=150421408843347 and `col3`<201725852064984)))  ORDER BY `col1`,`col2`,`col3`\n    github.com/pingcap/tidb/dumpling/export.(*tableData).Start\n    \tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:210\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:202\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571\n -  dial tcp 10.104.110.253:4000: connect: connection refused\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.Trace\n    \tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\n    github.com/pingcap/tidb/dumpling/export.createConnWithConsistency\n    \tgithub.com/pingcap/tidb/dumpling/export/sql.go:886\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).Dump.func3\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:198\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:196\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571\n -  dial tcp 10.104.110.253:4000: connect: connection refused\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.Trace\n    \tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\n    github.com/pingcap/tidb/dumpling/export.createConnWithConsistency\n    \tgithub.com/pingcap/tidb/dumpling/export/sql.go:886\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).Dump.func3\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:198\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:196\n    github.com/pingcap/tidb/br/pkg/utils.WithRetry\n    \tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:51\n    github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:184\n    github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:107\n    github.com/pingcap/tidb/dumpling/export.(*Writer).run\n    \tgithub.com/pingcap/tidb/dumpling/export/writer.go:85\n    github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4\n    \tgithub.com/pingcap/tidb/dumpling/export/dump.go:316\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \tgolang.org/x/[email protected]/errgroup/errgroup.go:57\n    runtime.goexit\n    \truntime/asm_amd64.s:1571"]

dump failed: sql: SELECT * FROM `H`.`t_o_shard` WHERE (`col1`>2747766125656 and `col1`<2992213486942)or(`col1`=2747766125656 and(`col2`>88780951220974 or(`col2`=88780951220974 and `col3`>=25798133891694)))or(`col1`=2992213486942 and(`col2`<150421408843347 or(`col2`=150421408843347 and `col3`<201725852064984)))  ORDER BY `col1`,`col2`,`col3`: invalid connection; dial tcp 10.104.110.253:4000: connect: connection refused; dial tcp 10.104.110.253:4000: connect: connection refused

[tiancaiamao]

Duplicated by #31981
And TiDB v6.2+ solve this by the coprocessor paging protocol #35068

[tiancaiamao]

Ref #35633