Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestMessageClientBasicMultiTopics is unstable #5210

Closed
Tracked by #2246
CharlesCheung96 opened this issue Apr 19, 2022 · 4 comments · Fixed by #5245
Closed
Tracked by #2246

TestMessageClientBasicMultiTopics is unstable #5210

CharlesCheung96 opened this issue Apr 19, 2022 · 4 comments · Fixed by #5245
Assignees
@liuzix
Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. type/enhancement The issue or PR belongs to an enhancement.

Comments

@CharlesCheung96
Copy link
Contributor

CharlesCheung96 commented Apr 19, 2022 •
edited
Loading

Which jobs are flaking?

TestMessageClientBasicMultiTopics

Which test(s) are flaking?

leak_test

Jenkins logs or GitHub Actions link

--- FAIL: TestMessageClientBasicMultiTopics (49.96s)
[2022-04-19T08:32:29.707Z]     server_client_integration_test.go:155: 
[2022-04-19T08:32:29.707Z]         	Error Trace:	server_client_integration_test.go:155
[2022-04-19T08:32:29.707Z]         	            				asm_amd64.s:1571
[2022-04-19T08:32:29.707Z]         	Error:      	Received unexpected error:
[2022-04-19T08:32:29.707Z]         	            	context deadline exceeded
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/errors.AddStack
[2022-04-19T08:32:29.707Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/errors.Trace
[2022-04-19T08:32:29.707Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p/internal.(*SendChan).SendSync
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/internal/send_chan.go:77
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).sendMessage
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/client.go:441
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).SendMessage
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/client.go:385
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p.runP2PIntegrationTest.func6
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/server_client_integration_test.go:154
[2022-04-19T08:32:29.707Z]         	            	runtime.goexit
[2022-04-19T08:32:29.707Z]         	            		/usr/local/go/src/runtime/asm_amd64.s:1571
[2022-04-19T08:32:29.707Z]         	Test:       	TestMessageClientBasicMultiTopics
[2022-04-19T08:32:29.707Z]     server_client_integration_test.go:155: 
[2022-04-19T08:32:29.707Z]         	Error Trace:	server_client_integration_test.go:155
[2022-04-19T08:32:29.707Z]         	            				asm_amd64.s:1571
[2022-04-19T08:32:29.707Z]         	Error:      	Received unexpected error:
[2022-04-19T08:32:29.707Z]         	            	context deadline exceeded
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/errors.AddStack
[2022-04-19T08:32:29.707Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/errors.Trace
[2022-04-19T08:32:29.707Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p/internal.(*SendChan).SendSync
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/internal/send_chan.go:77
[2022-04-19T08:32:29.707Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).sendMessage
[2022-04-19T08:32:29.707Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/client.go:441
[2022-04-19T08:32:29.708Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).SendMessage
[2022-04-19T08:32:29.708Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/client.go:385
[2022-04-19T08:32:29.708Z]         	            	github.com/pingcap/tiflow/pkg/p2p.runP2PIntegrationTest.func6
[2022-04-19T08:32:29.708Z]         	            		/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/p2p/server_client_integration_test.go:154

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/24341/pipeline
https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9241/pipeline/

Anything else we need to know

  • Does this test exist for other branches as well?

  • Has there been a high frequency of failure lately?
@CharlesCheung96 CharlesCheung96 added the component/test Unit tests and integration tests component. label Apr 19, 2022
@CharlesCheung96 CharlesCheung96 mentioned this issue Apr 19, 2022
Closed
@CharlesCheung96 CharlesCheung96 added the area/ticdc Issues or PRs related to TiCDC. label Apr 20, 2022
@Rustin170506
Copy link
Member

Rustin170506 commented Apr 20, 2022 •
edited
Loading

another case: https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline

[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-229)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-230)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-231)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-232)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-233)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-234)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-235)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-236)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-237)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-238)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-239)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-240)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-241)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-242)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-243)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-244)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-245)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-246)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-247)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-248)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-249)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-250)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-251)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-252)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-253)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-254)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-255)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-256)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-257)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-258)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-259)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-260)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-261)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-262)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-263)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-264)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-265)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-266)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-267)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-268)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-269)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-270)[](https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9220/pipeline#step-49-log-271)[2022-04-20T06:57:09.466Z] === Failed

[2022-04-20T06:57:09.466Z] === FAIL: pkg/p2p TestMessageClientBasicMultiTopics (10.10s)

[2022-04-20T06:57:09.467Z] [2022/04/20 14:54:50.699 +08:00] [INFO] [server.go:536] ["peer connection received"] [senderID=test-client-1] [senderAdvertiseAddr=] [addr=127.0.0.1:47994] [epoch=1]

[2022-04-20T06:57:09.467Z] [2022/04/20 14:55:00.628 +08:00] [WARN] [client.go:168] ["peer message client detected error, restarting"] [error="context deadline exceeded"] [errorVerbose="context deadline exceeded\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p/internal.(*SendChan).Receive\n\t/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/internal/send_chan.go:124\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runTx\n\t/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:231\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func1\n\t/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:200\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"]

[2022-04-20T06:57:09.467Z]     server_client_integration_test.go:155: 

[2022-04-20T06:57:09.467Z]         	Error Trace:	server_client_integration_test.go:155

[2022-04-20T06:57:09.467Z]         	            				asm_amd64.s:1571

[2022-04-20T06:57:09.467Z]         	Error:      	Received unexpected error:

[2022-04-20T06:57:09.467Z]         	            	context deadline exceeded

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/errors.AddStack

[2022-04-20T06:57:09.467Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/errors.Trace

[2022-04-20T06:57:09.467Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p/internal.(*SendChan).SendSync

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/internal/send_chan.go:77

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).sendMessage

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:441

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).SendMessage

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:385

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p.runP2PIntegrationTest.func6

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/server_client_integration_test.go:154

[2022-04-20T06:57:09.467Z]         	            	runtime.goexit

[2022-04-20T06:57:09.467Z]         	            		/usr/local/go/src/runtime/asm_amd64.s:1571

[2022-04-20T06:57:09.467Z]         	Test:       	TestMessageClientBasicMultiTopics

[2022-04-20T06:57:09.467Z]     server_client_integration_test.go:155: 

[2022-04-20T06:57:09.467Z]         	Error Trace:	server_client_integration_test.go:155

[2022-04-20T06:57:09.467Z]         	            				asm_amd64.s:1571

[2022-04-20T06:57:09.467Z]         	Error:      	Received unexpected error:

[2022-04-20T06:57:09.467Z]         	            	context deadline exceeded

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/errors.AddStack

[2022-04-20T06:57:09.467Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/errors.Trace

[2022-04-20T06:57:09.467Z]         	            		/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p/internal.(*SendChan).SendSync

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/internal/send_chan.go:77

[2022-04-20T06:57:09.467Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).sendMessage

[2022-04-20T06:57:09.467Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:441

[2022-04-20T06:57:09.468Z]         	            	github.com/pingcap/tiflow/pkg/p2p.(*MessageClient).SendMessage

[2022-04-20T06:57:09.468Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/client.go:385

[2022-04-20T06:57:09.468Z]         	            	github.com/pingcap/tiflow/pkg/p2p.runP2PIntegrationTest.func6

[2022-04-20T06:57:09.468Z]         	            		/home/jenkins/agent/workspace/ut-check/tiflow/pkg/p2p/server_client_integration_test.go:154

[2022-04-20T06:57:09.468Z]         	            	runtime.goexit

[2022-04-20T06:57:09.468Z]         	            		/usr/local/go/src/runtime/asm_amd64.s:1571

[2022-04-20T06:57:09.468Z]         	Test:       	TestMessageClientBasicMultiTopics

@zhaoxinyu
Copy link
Contributor

another case: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/24449/pipeline

@Rustin170506
Copy link
Member

another case: https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9310/pipeline

@buchuitoudegou
Copy link
Contributor

Seems the same case:
https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/9387/pipeline/

@liuzix liuzix mentioned this issue Apr 22, 2022
Merged
@liuzix liuzix added the type/enhancement The issue or PR belongs to an enhancement. label Apr 22, 2022
ti-chi-bot pushed a commit that referenced this issue Apr 24, 2022
@liuzix
pkg/p2p: reduce workload of TestMessageClientBasicMultiTopics (#5245)
a4cfd53 
close #5210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pkg/p2p: reduce workload of TestMessageClientBasicMultiTopics liuzix/ticdc
5 participants
@zhaoxinyu @liuzix @buchuitoudegou @Rustin170506 @CharlesCheung96