Scheduling is blocked for around 5 mins after transferring the PD leader

[rleungx]

Bug Report

What did you do?

Transfer PD leader manually.

What did you expect to see?

The scheduling can run immediately after the leader is transferred.

What did you see instead?

:34.066 +08:00] [INFO] [coordinator.go:302] ["coordinator starts to collect cluster information"]
[2022/03/29 14:29:34.068 +08:00] [INFO] [id.go:122] ["idAllocator allocates a new id"] [alloc-id=2000]
[2022/03/29 14:29:34.068 +08:00] [INFO] [util.go:77] ["load cluster version"] [cluster-version=6.0.0-alpha]
[2022/03/29 14:29:34.068 +08:00] [INFO] [server.go:1381] ["PD cluster leader is ready to serve"] [pd-leader-name=pd-0]
[2022/03/29 14:34:29.850 +08:00] [INFO] [grpc_service.go:1375] ["update service GC safe point"] [service-id=gc_worker] [expire-at=9223372036854775807] [safepoint=432153577346301952]
[2022/03/29 14:34:34.067 +08:00] [INFO] [coordinator.go:305] ["coordinator has finished cluster information preparation"]
[2022/03/29 14:34:34.067 +08:00] [INFO] [coordinator.go:315] ["coordinator starts to run schedulers"]
[2022/03/29 14:34:34.068 +08:00] [INFO] [coordinator.go:363] ["create scheduler with independent configuration"] [scheduler-name=balance-hot-region-scheduler]
[2022/03/29 14:34:34.069 +08:00] [INFO] [coordinator.go:363] ["create scheduler with independent configuration"] [scheduler-name=balance-leader-scheduler]
[2022/03/29 14:34:34.070 +08:00] [INFO] [coordinator.go:363] ["create scheduler with independent configuration"] [scheduler-name=balance-region-scheduler]
[2022/03/29 14:34:34.070 +08:00] [INFO] [coordinator.go:385] ["create scheduler"] [scheduler-name=balance-region-scheduler] [scheduler-args="[]"]
[2022/03/29 14:34:34.071 +08:00] [INFO] [coordinator.go:385] ["create scheduler"] [scheduler-name=balance-leader-scheduler] [scheduler-args="[]"]
[2022/03/29 14:34:34.071 +08:00] [INFO] [coordinator.go:385] ["create scheduler"] [scheduler-name=balance-hot-region-scheduler] [scheduler-args="[]"]
[2022/03/29 14:34:34.072 +08:00] [INFO] [coordinator.go:285] ["coordinator begins to actively drive push operator"]
[2022/03/29 14:34:34.072 +08:00] [INFO] [coordinator.go:220] ["coordinator begins to check suspect key ranges"]

It is blocked for around 5 minutes to prepare.

What version of PD are you using (pd-server -V)?

master

[rleungx]

Caused by #2591. Before we don't sync the leader information, after transferring the PD leader, all regions will be regarded as isNew. After this PR, these regions won't be regarded as isNew, so the prepare checker won't collect these regions.

[rleungx]

Thanks to @HunDunDM, after discussion with him, I think adding a flag in region info could be a good way to go.