Unstable test TestClientErrNoPendingRegion

[sleepymole]

What did you do?

https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/41294/pipeline/

What did you expect to see?

CI passed.

What did you see instead?

[2022-10-31T09:26:07.395Z] === FAIL: cdc/kv TestClientErrNoPendingRegion (2.53s)

[2022-10-31T09:26:07.395Z] [2022/10/31 17:18:30.576 +08:00] [INFO] [client.go:538] ["event feed started"] [namespace=default] [changefeed=changefeed-test] [tableID=0] [tableName=] [startTs=100] [span="[61, 63)"]

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:30.580 +08:00] [INFO] [client.go:722] ["creating new stream to store to send request"] [namespace=default] [changefeed=changefeed-test] [regionID=3] [requestID=69] [storeID=1] [addr=127.0.0.1:38570]

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:31.078 +08:00] [WARN] [client.go:1146] ["failed to receive from stream"] [namespace=default] [changefeed=changefeed-test] [addr=127.0.0.1:38570] [storeID=1] [error="injected error"] [errorVerbose="injected error\ngithub.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/ut-check/tiflow/cdc/kv/client.go:1133\ngithub.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/ut-check/tiflow/cdc/kv/client.go:732\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"]

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:31.101 +08:00] [INFO] [client.go:1092] ["stream to store closed"] [namespace=default] [changefeed=changefeed-test] [addr=127.0.0.1:38570] [storeID=1]

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:31.101 +08:00] [INFO] [region_worker.go:422] ["region worker closed by error"] [namespace=default] [changefeed=changefeed-test]

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:31.580 +08:00] [INFO] [client.go:722] ["creating new stream to store to send request"] [namespace=default] [changefeed=changefeed-test] [regionID=4] [requestID=70] [storeID=1] [addr=127.0.0.1:38570]

[2022-10-31T09:26:07.396Z]     client_test.go:285: 

[2022-10-31T09:26:07.396Z]         	Error Trace:	/home/jenkins/agent/workspace/ut-check/tiflow/cdc/kv/client_test.go:285

[2022-10-31T09:26:07.396Z]         	            				/home/jenkins/agent/workspace/ut-check/tiflow/cdc/kv/client_test.go:2932

[2022-10-31T09:26:07.396Z]         	            				/home/jenkins/agent/workspace/ut-check/tiflow/cdc/kv/client_test.go:2871

[2022-10-31T09:26:07.396Z]         	Error:      	Expected nil, but got: &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %s, error: %s", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc000c8d080), args:[]interface {}{"20", (*errors.fundamental)(0xc000c8d080)}, file:"/home/jenkins/agent/workspace/ut-check/tiflow/pkg/retry/retry_with_opt.go", line:69}

[2022-10-31T09:26:07.396Z]         	Test:       	TestClientErrNoPendingRegion

[2022-10-31T09:26:07.396Z] [2022/10/31 17:18:33.102 +08:00] [INFO] [region_cache.go:1025] ["mark store's regions need be refill"] [store=127.0.0.1:38570]

Versions of the cluster

master

[sleepymole]

Duplicate with #2688.