Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test：process/agent_test.go. TestAgentBasics #4319

Closed
Tracked by #2246
maxshuang opened this issue Jan 13, 2022 · 0 comments · Fixed by #4827
Closed
Tracked by #2246

Flaky test：process/agent_test.go. TestAgentBasics #4319

maxshuang opened this issue Jan 13, 2022 · 0 comments · Fixed by #4827
Assignees
@liuzix
Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. type/enhancement The issue or PR belongs to an enhancement.

Comments

@maxshuang
Copy link
Contributor

Which jobs are flaking?

https://ci.pingcap.net/blue/organizations/jenkins/atom-ut/detail/atom-ut/9288/tests/

Which test(s) are flaking?

tiflow/cdc/processor/agent_test.go
TestAgentBasics

Jenkins logs or GitHub Actions link

=== RUN   TestAgentBasics
[2022/01/13 16:55:31.187 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=capture-0] [senderAdvertiseAddr=] [addr=127.0.0.1:40048] [epoch=1]
[2022/01/13 16:55:31.274 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=capture-1] [senderAdvertiseAddr=] [addr=127.0.0.1:49532] [epoch=1]
[2022/01/13 16:55:31.389 +08:00] [INFO] [agent.go:417] ["owner updated"] [changefeed=cf-1] [newOwner="{\"OwnerCaptureID\":\"capture-0\",\"OwnerRev\":1}"]
[2022/01/13 16:55:31.485 +08:00] [INFO] [agent_mock.go:97] [AddTable] [tableID=1]
[2022/01/13 16:55:31.576 +08:00] [INFO] [agent_mock.go:97] [AddTable] [tableID=1]
    agent_test.go:248: PASS:	AddTable(string,int64)
    agent_test.go:248: PASS:	AddTable(string,int64)
    agent_test.go:248: PASS:	GetCheckpoint()
[2022/01/13 16:55:31.830 +08:00] [WARN] [client.go:167] ["peer message client detected error, restarting"] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runTx\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:232\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:199\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.830 +08:00] [WARN] [server.go:630] ["peer-to-peer message handler error"] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).receive\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/server.go:722\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).SendMessage.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/server.go:629\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.830 +08:00] [INFO] [client.go:116] ["peer message client exited"] [addr=127.0.0.1:34981] [captureID=capture-1] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).Run\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:133\ngithub.com/pingcap/tiflow/pkg/p2p.(*messageRouterImpl).GetClient.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/message_router.go:144\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.831 +08:00] [WARN] [message_router.go:145] ["p2p client exited with error"] [addr=127.0.0.1:34981] [targetCapture=capture-1] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).Run\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:133\ngithub.com/pingcap/tiflow/pkg/p2p.(*messageRouterImpl).GetClient.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/message_router.go:144\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.832 +08:00] [WARN] [server.go:630] ["peer-to-peer message handler error"] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).receive\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/server.go:722\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).SendMessage.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/server.go:629\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.832 +08:00] [WARN] [server.go:633] ["error receiving from peer"] [error="context canceled"]
[2022/01/13 16:55:31.832 +08:00] [WARN] [client.go:167] ["peer message client detected error, restarting"] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runRx\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:349\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func2\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:204\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.833 +08:00] [INFO] [client.go:116] ["peer message client exited"] [addr=127.0.0.1:37491] [captureID=capture-0] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).Run\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:133\ngithub.com/pingcap/tiflow/pkg/p2p.(*messageRouterImpl).GetClient.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/message_router.go:144\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/01/13 16:55:31.833 +08:00] [WARN] [message_router.go:145] ["p2p client exited with error"] [addr=127.0.0.1:37491] [targetCapture=capture-0] [error="context canceled"] [errorVerbose="context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).Run\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/client.go:133\ngithub.com/pingcap/tiflow/pkg/p2p.(*messageRouterImpl).GetClient.func1\n\t/home/jenkins/agent/workspace/atom-ut/tiflow/pkg/p2p/message_router.go:144\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
--- FAIL: TestAgentBasics (0.66s)
panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("GetCheckpoint").Return(...) first, or remove the GetCheckpoint() call.
	This method was unexpected:
		GetCheckpoint()
		
	at: [agent_mock.go:145 agent.go:328 checkpoint_sender.go:91 agent.go:333 agent.go:201 agent.go:155 agent_test.go:253] [recovered]
	panic: 
assert: mock: I don't know what to return because the method call was unexpected.
	Either do Mock.On("GetCheckpoint").Return(...) first, or remove the GetCheckpoint() call.
	This method was unexpected:
		GetCheckpoint()
		
	at: [agent_mock.go:145 agent.go:328 checkpoint_sender.go:91 agent.go:333 agent.go:201 agent.go:155 agent_test.go:253]
goroutine 344 [running]:
testing.tRunner.func1.2(0x5d8a0c0, 0xc00075d010)
	/usr/local/go/src/testing/testing.go:1143 +0x49f
testing.tRunner.func1(0xc000304d80)
	/usr/local/go/src/testing/testing.go:1146 +0x695
panic(0x5d8a0c0, 0xc00075d010)
	/usr/local/go/src/runtime/panic.go:971 +0x499
github.com/stretchr/testify/mock.(*Mock).fail(0xc000b6ca10, 0x64fdaaf, 0xc1, 0xc00032c5c0, 0x4, 0x4)
	/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:254 +0x21f
github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc000b6ca10, 0x7fa7a15, 0xd, 0x0, 0x0, 0x0, 0xc00032c380, 0x4, 0x4)
	/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:418 +0xea5
github.com/stretchr/testify/mock.(*Mock).Called(0xc000b6ca10, 0x0, 0x0, 0x0, 0xc000b4e508, 0x0, 0x0)
	/go/pkg/mod/github.com/stretchr/[email protected]/mock/mock.go:383 +0x1ee
github.com/pingcap/tiflow/cdc/scheduler.(*MockTableExecutor).GetCheckpoint(0xc000b6ca10, 0xc000e9c350, 0x2)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/scheduler/agent_mock.go:145 +0x69
github.com/pingcap/tiflow/cdc/scheduler.(*BaseAgent).sendCheckpoint.func1(0xc000c92288, 0x78a64b8, 0xc0008a51a0)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/scheduler/agent.go:328 +0x10f
github.com/pingcap/tiflow/cdc/scheduler.(*checkpointTsSenderImpl).SendCheckpoint(0xc000c92240, 0x78a64b8, 0xc0008a51a0, 0xc00075cda0, 0x14fe20c, 0xc000148658)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/scheduler/checkpoint_sender.go:91 +0x85
github.com/pingcap/tiflow/cdc/scheduler.(*BaseAgent).sendCheckpoint(0xc000f7f9e0, 0x78a64b8, 0xc0008a51a0, 0x2, 0x7efc5b280008)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/scheduler/agent.go:333 +0xda
github.com/pingcap/tiflow/cdc/scheduler.(*BaseAgent).Tick(0xc000f7f9e0, 0x78a64b8, 0xc0008a51a0, 0x0, 0x2)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/scheduler/agent.go:201 +0xf6
github.com/pingcap/tiflow/cdc/processor.(*agentImpl).Tick(0xc000849e00, 0x78a64b8, 0xc0008a51a0, 0x12a05f201, 0x5f5e100)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/processor/agent.go:155 +0x633
github.com/pingcap/tiflow/cdc/processor.TestAgentBasics(0xc000304d80)
	/home/jenkins/agent/workspace/atom-ut/tiflow/cdc/processor/agent_test.go:253 +0x1290
testing.tRunner(0xc000304d80, 0x6513488)
	/usr/local/go/src/testing/testing.go:1193 +0x203
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x5d8

Anything else we need to know

  • Does this test exist for other branches as well?

  • Has there been a high frequency of failure lately?
@maxshuang maxshuang added the component/test Unit tests and integration tests component. label Jan 13, 2022
@maxshuang maxshuang changed the title Flaky test： TestAgentBasics Flaky test：process/agent_test.go. TestAgentBasics Jan 13, 2022
@maxshuang maxshuang mentioned this issue Jan 13, 2022
Closed
@maxshuang maxshuang added the area/ticdc Issues or PRs related to TiCDC. label Jan 13, 2022
@liuzix liuzix mentioned this issue Mar 9, 2022
Merged
@liuzix liuzix added the type/enhancement The issue or PR belongs to an enhancement. label Mar 9, 2022
ti-chi-bot pushed a commit that referenced this issue Mar 14, 2022
@liuzix
p2p,processor(cdc): fix two unstable tests (#4827)
0a4fc06 
close #3945, close #4319
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

p2p,processor(cdc): fix two unstable tests liuzix/ticdc
2 participants
@maxshuang @liuzix