flaky test(dm): testScheduler.TestScheduler fail

[maxshuang]

Which jobs are flaking?

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/13202/pipeline

Which test(s) are flaking?

scheduler_test.go:82: testScheduler.TestScheduler

Jenkins logs or GitHub Actions link

[2021-11-29T09:19:13.305Z] ----------------------------------------------------------------------

[2021-11-29T09:19:13.305Z] FAIL: scheduler_test.go:82: testScheduler.TestScheduler

[2021-11-29T09:19:13.305Z] 

[2021-11-29T09:19:13.305Z] scheduler_test.go:85:

[2021-11-29T09:19:13.305Z]     t.testSchedulerProgress(c, restartNewInstance)

[2021-11-29T09:19:13.305Z] scheduler_test.go:451:

[2021-11-29T09:19:13.305Z]     // CASE 4.6: try remove source when subtasks exist.

[2021-11-29T09:19:13.305Z]     // wait for source bound recovered

[2021-11-29T09:19:13.305Z]     c.Assert(utils.WaitSomething(30, 100*time.Millisecond, func() bool {

[2021-11-29T09:19:13.305Z]         bounds := s.BoundSources()

[2021-11-29T09:19:13.305Z]         return len(bounds) == 2 && bounds[0] == sourceID1 && bounds[1] == sourceID2

[2021-11-29T09:19:13.305Z]     }), IsTrue)

[2021-11-29T09:19:13.305Z] ... obtained bool = false

[2021-11-29T09:19:13.305Z] 

[2021-11-29T09:19:13.305Z] {"level":"warn","ts":"2021-11-29T17:17:52.256+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-25ccc4ec-2876-4fa1-b4b2-e99050705e14/localhost:41807447936948209690","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: request timed out"}

[2021-11-29T09:19:13.305Z] 

[2021-11-29T09:19:13.305Z] ----------------------------------------------------------------------

[2021-11-29T09:19:13.305Z] FAIL: scheduler_test.go:1473: testScheduler.TestStartSourcesWithoutSourceConfigsInEtcd

[2021-11-29T09:19:13.305Z] 

[2021-11-29T09:19:13.305Z] scheduler_test.go:1499:

[2021-11-29T09:19:13.305Z]     c.Assert(s.AddWorker(workerName1, workerAddr1), IsNil)

[2021-11-29T09:19:13.305Z] ... value *errors.withStack = etcdserver: request timed out ("etcdserver: request timed out")

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?

[lance6716]

generally utils.WaitSomething fails means the CI is too slow. Although we have retried many time but the expected condition is still not met.