Lightning physical import precheck failed due to only 3 out of 10 TiKVs have regions

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Deploy 6.5 nightly TiDB cluster with 9 TiKVs and 6 TiFlash nodes.
2. Do lightning physical import

2. What did you expect to see? (Required)

Lighting import should succeed.

3. What did you see instead (Required)

Lighting precheck failed. There are 9 TiKVs nodes, but only 3 of them has regions

4. What is your TiDB version? (Required)

lightning-0:/# /tidb-lightning -V
Release Version: v6.5.0-nightly
Git Commit Hash: f096450c339152e583eb578912ddc0ef227125df
Git Branch: heads/refs/tags/v6.5.0-nightly
Go Version: go1.19.12
UTC Build Time: 2023-09-08 10:22:58
Race Enabled: false

/ # /tikv-server -V
TiKV
Release Version:   6.5.4
Edition:           Community
Git Commit Hash:   6087ea61f99aaa9275d1834657d2d883736381cb
Git Commit Branch: heads/refs/tags/v6.5.5
UTC Build Time:    2023-09-09 06:47:09

mysql> SELECT tidb_version();
+------------------------------------------------------------
| tidb_version()                                             
+------------------------------------------------------------
| Release Version: v6.5.5
Edition: Community
Git Commit Hash: 4237dd2511c872692998b89eb07c0f8890de14d8
Git Branch: heads/refs/tags/v6.5.5
UTC Build Time: 2023-09-08 09:50:00
GoVersion: go1.19.12
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv |
+------------------------------------------------------------
1 row in set (0.00 sec)

[fubinzh]

For the other precheck issue with cdc capture, it should be #41040

[lance6716]

Do you mean the problem is TiFlash node should not be considered when checking region balance?

[fubinzh]

No, Only some of TiKVs have regions during precheck.
When the issue happens, the cluster was not in a health state as some tikv stores disconnected, due to below TiKV configurations which resulted in PD headthy check not works as expected.

[raftstore]
pd-heartbeat-tick-interval = "3m"
pd-store-heartbeat-tick-interval = "30s"