Tracking issue for flaky tests

[Rustin170506]

This is a tracking issue for unstable tests, and each unstable test is linked as a subtask.

TiCDC

• Test DoCornerCases fails #2273
• leak test TestStreamSendWithError unstable #2289
• Test TestFlowControlBasic fails #2293
• unit test ownerSuite.TestWatchFeedChange unstable [Test deprecated] #2296 Test deprecated
• unit test kafkaSuite .TestSaramaProducer unstable #2314
• unstable unit test frontier.(*tsHeapSuite).TestUpdateTs timeout #2345
• unstable integration test, unified_sort timeout #2346
• unstable unit test, invalid memory address or nil pointer dereference in kv client test #2347
• integration test unified_sorter_sort_dir_conflict is not stable #2348
• unit test etcdSuite.TestKVClientForceReconnect is not stable #2349
• unit test ownerSuite.TestWatchFeedChange is not stable #2405 Test deprecated
• unit test workerPoolSuite.TestMultiError is not stable #2469
• unit test managerSuite.TestManagerAddRemoveTable is not stable #2500
• test ddlPullerSuite.TestPuller is unstable #2540
• unstable unit test frontier.(*tsHeapSuite).TestRemoveNode timeout #2614
• unit test sink.(*managerSuite).TestManagerAddRemoveTable timeout #2624
• unstable integration test in gc_safepoint #2650
• kafka consumer met wait topic create timeout error #2651
• check_safepoint_forward in integration test gc_safepoint is not correct #2653
• integration test region_merge often meets query execution interruption error #2662
• unstable test etcdSuite.TestClientErrNoPendingRegion #2688
• unstable test TestConcurrentProcessRangeRequest #2694
• unit test etcdWorkerSuite.TestEtcdSum panic #2731
• unstable tests etcdSuite.TestRegionWorkerExitWhenIsIdle #2747
• unstable test clientSuite.TestRegionWorkerExitWhenIsIdle #4567
• unstable test etcdSuite.TestPutAllChangeFeedStatus #2748
• unstable test etcdWorkerSuite.TestEtcdBank #2792
• unit test tsHeapSuite.TestUpdateTs failed with unexpected minTs #2797
• unit test sorterSuite.TestSorterIOError #2813
• http_api flaking test #2958
• CI: kafka integration tests move_table meet panic: the commitTs of the emit row is less than the received row. #2887
• integration test get stuck during tidb cluster startup #2993
• unstable leak test reports unexpected goroutines #3041
• Flaking test: TestDoCancelInfiniteRetry #3115
• Test TestApplyMeetSinkError is unstable #3185
• Flaking integration multi_changefeed test #3207
• Unstable unit test TestServerWrapperDelayed FAIL: OnNewMessage(*p2p.MessagePacket) #3232
• tar: tikv3.log: file changed as we read it #3301
• FAIL github.com/pingcap/ticdc/cdc/kv 600.625s #3302
• CDC Integration Tests Cron keeps failing #3341
• Flaking integration test unified_sorter #3381
• Integration test changefeed_pause_resume failure, kafka consumer met table not exists error. #3399
• Integration test clustered_index_test find data inconsistency on t_set/t_enum #3421
• Flaking test: TestCleanerWriteRateLimited #3443
• Test TestHandleJobsDontBlock fails #3626
• tokenRegionSuite.TestRouter fails in leak tests #3679
• TestEtcdBank is unstable #3750
• Integration test ddl_sequence is unstable #3805
• new_ci_collation_with_old_value_test flakes due to etcd worker fails to commit #3815
• flaky test(ticdc/pkg/p2p): TestMessageClientBasics #3825
• Flaking Test：TestWriterRedoGC #3850
• Flaking Test：TestClientSendAnomalies #3945
• TestBroadcast is unstable #3963
• TestMySQLSinkFlushResovledTs is unstable. #3989
• panic: close of closed channel in cdc/kv #4135
• cdc/kv timed out #4139
• Flaky test：process/agent_test.go. TestAgentBasics #4319
• Flaky test：redo/writer  #4320
• integration test cdc_tiflash_test is unstable #4416
• data race when running leak test #4447
• tiflash integration fails when hibernate-regions-compatible enabled #4461
• data race found in TestRevisionNotFallBack #4473
• Flaky test: data race found in TestGetOwnerRevision #4480
• Flaky test: data race found in TestRevisionNotFallBack #4549
• Flaky test: clientSuite.TestDropStaleRequest #4586
• integration-kafka-tests/default_value fails #4600
• flaky test : integration test kafka_big_messages #4731
• TestMysqlSinkWorker has data race #4765
• Flaking: integration default value test #4785
• Test ddl_attributes is unstable #3180
• Flaky test: panic: Log in goroutine after TestGetOwnerRevision has completed #4856
• Kafka many_pk_or_uk fails #4895
• unstable kafka integration test generated columns #4939
• Flaking availability integration test #4958
• TestClientSendAnomalies is unstable #4975
• data race of github.com/benbjohnson/clock #4961
• unstable ut TestUpdateTs #4941
• TestHandleError is unstable #5064
• flaky test- multi-topic integration test fail #5076
• consistent_replicate_gbk fails #5083
• rename_tables_test fails #5084
• kafka sarama client data race #5090
• TestMessageClientBasicMultiTopics is unstable #5210
• Goroutine leak in mysql sink #5356
• TestExecRenameTablesDDL is unstable #5406
• consistent_replicate_gbk losing data after applying redolog #5107
• TestStreamRecvWithErrorNormal is unstable #5445
• Flaking test TestWriterRedoGC #5736
• Flaking test: etcd test data race #5792
• Flaking test: TestIteratorWithTableFilter #5798
• Flaking test: big_txn #6046
• Flaking Test: consistent_replicate_s3 #6167
• [Flaking test]: ticdc workerpool test case timeout after 10min #6804
• unstable test TestNewMySQLBackendExecDML #6895
• Unstable test pdutil.TestTimeFromPD #7201

---

DM

If you encounter the unstable test repeatedly, you can add +n in the end of following test. For example,

• DM Binlog 999999 keeps failing #3283 +1, +4, ...

We will give priority to frequent unstable tests.

• DM Binlog 999999 keeps failing #3283
• for stop-task, declarative API should wait the task is stoppped #3254
• testRelaySuite.TestHandleEvent #3328
• test "full_mode" failed to auto remove dump files #3335
• test "safe_mode" failed to get "result": true #3336
• Test upstream-database-switch is not stable #3353
• loader panic: send on closed channel #3380
• cannot fetch downstream table schema #3394
• log doesn't contain fail to delete shard DDL infos and lock operations #3398
• pause-relay fail to get expected result #3400
• test dm_worker_task_state metric check failed #3401
• log doesn't contain will try purge whole relay dir for new relay log #3404
• test: openapi operate-schema #3407
• adjust_gtid: OUTPUT DOES NOT CONTAIN 'dm-integration-test-67nc6-rwrg6-bin.000001' #3409
• online_ddl test sync_diff failed #3450
• Data race in unit test dm/dm/master.TestMaster() #3495
• https://github.com/pingcap/tiflow/issues3518
• flaky test(dm): testScheduler.TestScheduler fail #3657
• unstable integration test: data race found in case ha_master #3728
• TestEtcdUtil is unstable #3777
• server_test.TestServer data race #3785
• flaky test(dm): TestServer – github.com/pingcap/ticdc/dm/dm/worker data race #3791
• TestShardDD is unstable #3913
• test_stop_task_before_checkpoint is unstable  #3915
• DM-119 is not stable #4029
• testOptimist.TestOptimistLockConflict is not stabled #4131
• TestMaster timed out #4132
• DM many tables with dump status #4134
• dm-syncer IT is not stable #4158
• DM TestStartTaskWithRemoveMeta #4168
• all_mode is not stable #4171
• gtid test is not stable #4174
• test(dm): new_relay is not stable #4243
• test(dm): drop_column_with_index is unstable  #4246
• DM integration test full_mode is unstable. #4255
• DM: TestMaster is unstable. #4312
• DM TestRedirectRequestToLeader #4554
• DM TestFailToStartLeader #4565
• TestFailToStartLeader #4875
• DM worker can't exit in CI #5089
• syncer reports error reading from server: EOF and paused #5235
• DM optimistic test fail to check conflicting statement #5488
• DM optimistic fail sync data when restart master or worker frequently  #5489
• DM full mode failed to check dump file which should be removed #5490
• log doesn't contain finish to handle ddls in optimistic shard mode.*alter table shardddl1.tb1 change d f int #5729
• validation status test new\/ignored\/resolved: 1\/0\/0 count: 0 != expected: 1 #5730
• [sequence_sharding_optimistic] check diff failed at last #5731
• GENERATED ALWAYS AS(id+2) VIRTUAL count: 0 != expected: 1 #5732
• ERROR 1146 (42S02) at line 1: Table 'sharding_seq_tmp.t1' doesn't exist #5733
• test(dm): TestTableSchemaChecking #7569

---

Engine

• Unstable unit test TestStartEtcdTimeout in dataflow engine #5515
• Unit test TestCleanUpStaleResourcesOnStartUp failed #5699
• Unstable test election.TestElectorBasic #7168

[Rustin170506]

Basic guidelines for fixing these tests.

1. discover unstable tests.
2. Create a corresponding issue to track the unstable tests.
   3. Submit a PR to temporarily skip the unstable test. (The expected time to fix must be set in the PR)
3. start fixing the unstable tests. (assign the issue to someone)
   5. Bring the test back online.

First, our integration tests need to support skipping some unstable tests, and I will adapt the sh script to support this mechanism.

[asddongmen]

Great works! I will close it since it is stale.