Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: client_test.TestHandleError is not stable #177

Open
pingyu opened this issue Jul 22, 2022 · 3 comments · Fixed by #178
Open

cdc: client_test.TestHandleError is not stable #177

pingyu opened this issue Jul 22, 2022 · 3 comments · Fixed by #178
Labels
status/stale type/bug Something isn't working

Comments

@pingyu
Copy link
Collaborator

pingyu commented Jul 22, 2022

Bug Report

1. Describe the bug

client_test.TestHandleError is not stable.

CI link: https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true

From the stack, goroutine raise a exception from here, and stuck at here.

goroutine 2021 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc0008bb358?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc0008bb350)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc00055d4a0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000b114a0?, {0x5b19d2c, 0x25}, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc00055d4a0)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2809
reflect.Value.call({0x5a59ee0?, 0xc000711300?, 0xc000bdccd0?}, {0x5aa2b5e, 0x4}, {0xc000879eb8, 0x1, 0xc000bdccf0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc000711300?, 0xc000481a40?}, {0xc000bdceb8, 0x1, 0x1})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:[339](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:340) +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc00055d4a0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:734 +0x5de

Full stack:

=== FAIL: cdc/kv Test (unknown)
Error: 7/22 08:56:34.747 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/22 08:56:35.295 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/22 08:56:36.557 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:33771, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/22 08:56:44.541 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
coverage: 63.9% of statements
panic: test timed out after 30m0s
goroutine 2208 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:2029 +0xbb
created by time.goFunc
	/opt/hostedtoolcache/go/1.18.3/x64/src/time/sleep.go:176 +0x48
goroutine 1 [chan receive, 30 minutes]:
testing.(*T).Run(0xc000b78b60, {0x5aa2f56, 0x4}, 0x5bd2c78)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1487 +0x750
testing.runTests.func1(0x0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1839 +0x9a
testing.tRunner(0xc000b78b60, 0xc0008dfb68)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
testing.runTests(0xc000a790e0?, {0x8a65680, 0x2, 0x2}, {0x40?, 0x55ed100?, 0x91ae5a0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1837 +0x7e5
testing.(*M).Run(0xc000a790e0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1719 +0xa72
main.main()
	_testmain.go:125 +0x3aa
goroutine 1966 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:174 +0x259
created by go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:169 +0xed
goroutine 2021 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc0008bb358?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc0008bb350)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc00055d4a0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000b114a0?, {0x5b19d2c, 0x25}, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc00055d4a0)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2809
reflect.Value.call({0x5a59ee0?, 0xc000711300?, 0xc000bdccd0?}, {0x5aa2b5e, 0x4}, {0xc000879eb8, 0x1, 0xc000bdccf0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc000711300?, 0xc000481a40?}, {0xc000bdceb8, 0x1, 0x1})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:339 +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc00055d4a0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:734 +0x5de
goroutine 2032 [select]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchersLoop(0xc000202960)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:233 +0x249
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:95 +0xb75
goroutine 26 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x265
goroutine 54 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001c3e80)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x177
created by go.opencensus.io/stats/view.init.0
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf7
goroutine 1965 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000a9a180)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7
goroutine 2042 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).linearizableReadLoop(0xc0002e3c00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/v3_server.go:738 +0x178
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2055 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc00147e4c0?}, 0xc0006b5f80?, 0xc000a20000?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000bb9c00, {0x6f18c88, 0xc000d9e000})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func2()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x66
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x8d8
goroutine 2080 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc000b94d50?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000b94d48)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000b94d40)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed(0xc0001b7a40, {0x6f19750, 0xc000b94540}, 0x64)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:590 +0xaef
github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).EventFeed(0xc000a2b9a0, {0x6f19750, 0xc000b94540}, {{0xc0008bb8ff, 0x1, 0x1}, {0xc0008bb90f, 0x1, 0x1}}, 0x64, ...)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:405 +0x179
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:568 +0x27c
created by github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:566 +0xe3f
goroutine 2098 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc00147e4c0?}, 0xc0006b4d20?, 0xc000c304e0?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc000c0c000, {0x6f18c88, 0xc000d9f280})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:[303](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:304)9 +0x5a7
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0x59
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0xf58
goroutine 33 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e3f0, {0x6f19750, 0xc000c04140})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 32 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e360, {0x6f19750, 0xc000c04140})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 31 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e2d0, {0x6f19750, 0xc000c04140})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 30 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc00077e240, {0x6f19750, 0xc000c04140})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 29 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000c04190?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000c04188)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000c04180)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run(0xc00077c150, {0x6f19750, 0xc000c040c0})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:75 +0x2c8
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:827 +0x72
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 27 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000c04110?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000c04108)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000c04100)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool({0x6f19788, 0xc000054128})
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:829 +0x16f
github.com/tikv/migration/cdc/cdc/kv.Test.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:62 +0x3d
created by github.com/tikv/migration/cdc/cdc/kv.Test
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:61 +0x66
goroutine 153 [select, 29 minutes]:
github.com/pingcap/check.(*suiteRunner).doRun(0xc000b1cbd0, 0x0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:667 +0x2a5
github.com/pingcap/check.(*suiteRunner).run(0xc000b1cbd0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:696 +0xe6
github.com/pingcap/check.Run({0x5a59ee0, 0xc000711300}, 0xc000c00248?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:150 +0x4a
github.com/pingcap/check.RunAll(0xc000c13d02?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:117 +0x259
github.com/pingcap/check.TestingT(0xc000b78d00)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:99 +0x647
github.com/tikv/migration/cdc/cdc/kv.Test(0x0?)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:64 +0x70
testing.tRunner(0xc000b78d00, 0x5bd2c78)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1486 +0x725
goroutine 2087 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d318, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0013de218, 0xc0008c2000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0013de200, {0xc0008c2000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0013de200, {0xc0008c2000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0004d6038, {0xc0008c2000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0007a61e0, {0xc000af6040, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc0007a61e0}, {0xc000af6040, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000af6040?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc0007a61e0?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000af6000)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001660000)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x585
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x2772
goroutine 2040 [select, 11 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.monitorFileDescriptor(0xc0002e3c00?, 0xc000bf7aa0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/metrics.go:237 +0x3f9
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Start.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:802 +0x65
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2024 [select]:
go.etcd.io/etcd/server/v3/mvcc/backend.(*backend).run(0xc000b13b00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:421 +0x235
created by go.etcd.io/etcd/server/v3/mvcc/backend.newBackend
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:229 +0xd36
goroutine 1969 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2
goroutine 1967 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2
goroutine 2085 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*sizedRegionRouter).Run(0xc000dab9b0, {0x6f19750, 0xc000b94d00})
	/home/runner/work/migration/migration/cdc/cdc/kv/token_region.go:151 +0x1fb
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func5()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:584 +0x82
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 1968 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2
goroutine 2081 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).dispatchRequest(0xc0001b7a40, {0x6f19750, 0xc000b94d00})
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:870 +0x4d6
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:511 +0x5e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 226 [select, 29 minutes]:
github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc000b1cb40)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:464 +0x153
created by github.com/pingcap/check.(*resultTracker).start
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:444 +0x8f
goroutine 2048 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d7c8, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000b19918, 0x1?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b19900)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b19900)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc001347770)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc001347770)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*rwTimeoutListener).Accept(0xc000ad4f20)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/timeout_listener.go:36 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000a1c640)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:557 +0x178
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc000ad4f00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:585 +0x176
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:579 +0xe32
goroutine 2041 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorVersions(0xc0002e3c00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2465 +0x149
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2090 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000a2a410, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007a65a0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:332 +0x1cd
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:329 +0x269b
goroutine 2092 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d138, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0013de598, 0xc0008d0000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0013de580, {0xc0008d0000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0013de580, {0xc0008d0000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000415860, {0xc0008d0000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0009f88d0, {0xc0008d0000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/buffer.go:53 +0x2f8
github.com/soheilhy/cmux.(*MuxConn).Read(0xc0009f88c0, {0xc0008d0000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:297 +0x51
bufio.(*Reader).Read(0xc0007a62a0, {0xc000af6120, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc0007a62a0}, {0xc000af6120, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000af6120?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc0007a62a0?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000af60e0)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000cbba0, 0x4ed04d6?, 0xc000b06f80?)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:611 +0x145
google.golang.org/grpc.(*Server).serveStreams(0xc000bb9c00, {0x6f31410?, 0xc0000cbba0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:906 +0x267
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:848 +0x65
created by google.golang.org/grpc.(*Server).handleRawConn
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:847 +0x245
goroutine 2065 [chan receive, 29 minutes]:
go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:252 +0x71
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:251 +0x65f
goroutine 2044 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorDowngrade(0xc0002e3c00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2586 +0x156
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2028 [select]:
go.etcd.io/etcd/server/v3/lease.(*lessor).runLoop(0xc000b13c80)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:619 +0x195
created by go.etcd.io/etcd/server/v3/lease.newLessor
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:237 +0x5f7
goroutine 2047 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ad4f20?}, 0xc0006e5e60?, 0xc000bf7da0?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc000af6c40, {0x6f18c88, 0xc000ad53a0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:551 +0xa4f
goroutine 2046 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ad4f20?}, 0xc0006e5e00?, 0xc000bf7d40?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000bb9880, {0x6f18c88, 0xc000ad5340})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:545 +0x63f
goroutine 2039 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile(0xc0002e3c00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:876 +0x519
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2025 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/wal.(*filePipeline).run(0xc000b3c9c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:97 +0x1e5
created by go.etcd.io/etcd/server/v3/wal.newFilePipeline
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:55 +0x4af
goroutine 2045 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2033 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncVictimsLoop(0xc000202960)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:259 +0x209
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:96 +0xbf2
goroutine 2036 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run(0xc0002e3c00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:1117 +0xb65
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:861 +0x159e
goroutine 2026 [select]:
go.etcd.io/etcd/raft/v3.(*node).run(0xc0006e56e0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:341 +0xc1d
created by go.etcd.io/etcd/raft/v3.StartNode
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:230 +0x31e
goroutine 2054 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2
goroutine 2029 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc0006e5a40)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7
goroutine 2088 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000a2a320, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007a6240)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x106
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x[312](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:313)9
goroutine 2091 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000cbba0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1131 +0x3cf
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:[341](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:342) +0x271a
goroutine 2082 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore(0xc0001b7a40, {0x6f19750?, 0xc000b94d00?}, 0xc000b94d40)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:690 +0x21c
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 2052 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f70b5f2d5e8, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000b19998, 0xc000817501?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b19980)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b19980)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc001[347](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:348)7d0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc0013477d0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*limitListener).Accept(0xc0013477e8)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/limit_listen.go:46 +0x6c
go.etcd.io/etcd/client/pkg/v3/transport.(*keepaliveListener).Accept(0xc00147e4c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/keepalive_listener.go:49 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000b6ba90)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve(0xc0009f0000, 0xc0002e3c00, 0xc0004bb960, {0x6efd080, 0xc000b47980}, 0xc00147f080, {0xc000ad5540, 0x2, 0x2})
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:195 +0x1f65
go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1(0x0?)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:734 +0x145
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:733 +0x126f
goroutine 2056 [select, 29 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000b980c0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x150
created by google.golang.org/grpc.newCCBalancerWrapper
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x4b7
goroutine 2084 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:546 +0x289
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
goroutine 2083 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func3()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:520 +0x169
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected][502](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:503)17-0de[741](https://github.com/tikv/migration/runs/7465132496?check_suite_focus=true#step:4:742)cfad7f/errgroup/errgroup.go:72 +0x12f

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

  • TiKV:
  • TiKV CDC:
  • TiKV BR:
  • TiKV Online Bulk Load:
@pingyu pingyu added the type/bug Something isn't working label Jul 22, 2022
@pingyu pingyu mentioned this issue Jul 22, 2022
Merged
@pingyu
Copy link
Collaborator Author

pingyu commented Jul 28, 2022

Reopen because another fail case is met.
https://github.com/tikv/migration/runs/7559704549?check_suite_focus=true

=== Failed
=== FAIL: cdc/kv Test (unknown)
Error: 7/28 12:22:27.405 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/28 12:22:28.277 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/28 12:22:29.627 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:40175, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/28 12:22:34.333 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
coverage: 66.9% of statements
panic: test timed out after 30m0s

goroutine 2276 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:2029 +0xbb
created by time.goFunc
	/opt/hostedtoolcache/go/1.18.3/x64/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 30 minutes]:
testing.(*T).Run(0xc000775380, {0x5aa2f56, 0x4}, 0x5bd2c78)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1487 +0x750
testing.runTests.func1(0x0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1839 +0x9a
testing.tRunner(0xc000775380, 0xc000c47b68)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
testing.runTests(0xc000b2cf00?, {0x8a65680, 0x2, 0x2}, {0x40?, 0x55ed100?, 0x91ae5a0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1837 +0x7e5
testing.(*M).Run(0xc000b2cf00)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1719 +0xa72
main.main()
	_testmain.go:125 +0x3aa

goroutine 38 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x265

goroutine 2053 [chan receive, 29 minutes]:
go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:252 +0x71
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:251 +0x65f

goroutine 2020 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run(0xc000c54a00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:1117 +0xb65
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:861 +0x159e

goroutine 2032 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f5a1e0cd7c8, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000b59218, 0x1596[301](https://github.com/tikv/migration/runs/7559704549?check_suite_focus=true#step:4:302)?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b59200)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b59200)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc000f20d98)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc000f20d98)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*rwTimeoutListener).Accept(0xc000ebb660)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/timeout_listener.go:36 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc0000ad2c0)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:557 +0x178
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc000ebb640)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:585 +0x176
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:579 +0xe32

goroutine 58 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00061df00)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x177
created by go.opencensus.io/stats/view.init.0
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf7

goroutine 2060 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0008eaea0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1131 +0x3cf
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:341 +0x271a

goroutine 2057 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0009006e0, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001b5f860)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x106
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x3129

goroutine 2075 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*sizedRegionRouter).Run(0xc000cc5e90, {0x6f19750, 0xc000ba8b80})
	/home/runner/work/migration/migration/cdc/cdc/kv/token_region.go:151 +0x1fb
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func5()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:584 +0x82
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2070 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc000ba8bd0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000ba8bc8)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ba8bc0)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed(0xc000a188c0, {0x6f19750, 0xc000ba8580}, 0x64)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:590 +0xaef
github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).EventFeed(0xc000a80be0, {0x6f19750, 0xc000ba8580}, {{0xc0007d329f, 0x1, 0x1}, {0xc0007d32af, 0x1, 0x1}}, 0x64, ...)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:405 +0x179
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:568 +0x27c
created by github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:566 +0xe3f

goroutine 2028 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorDowngrade(0xc000c54a00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2586 +0x156
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2036 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f5a1e0cd408, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:[302](https://github.com/tikv/migration/runs/7559704549?check_suite_focus=true#step:4:303) +0x89
internal/poll.(*pollDesc).wait(0xc000b59298, 0xc000899501?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000b59280)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc000b59280)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc000f20de0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc000f20de0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*limitListener).Accept(0xc000f20df8)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/limit_listen.go:46 +0x6c
go.etcd.io/etcd/client/pkg/v3/transport.(*keepaliveListener).Accept(0xc000d545d0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/keepalive_listener.go:49 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc0008fdc20)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve(0xc000a66000, 0xc000c54a00, 0xc0004e8d60, {0x6efd080, 0xc000afc4c0}, 0xc0002b7380, {0xc000af0920, 0x2, 0x2})
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:195 +0x1f65
go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1(0x0?)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:734 +0x145
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:733 +0x126f

goroutine 2072 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore(0xc000a188c0, {0x6f19750?, 0xc000ba8b80?}, 0xc000ba8bc0)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:690 +0x21c
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2030 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ebb660?}, 0xc000733bc0?, 0xc000bf9b00?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000219180, {0x6f18c88, 0xc000af0760})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:545 +0x63f

goroutine 224 [select, 29 minutes]:
github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc000b30f30)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:464 +0x153
created by github.com/pingcap/check.(*resultTracker).start
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:444 +0x8f

goroutine 2056 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f5a1e0cd048, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000bb4318, 0xc000d10000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000bb4300, {0xc000d10000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc000bb4300, {0xc000d10000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0005dcac0, {0xc000d10000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc001b5f800, {0xc0012d30e0, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc001b5f800}, {0xc0012d30e0, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0012d30e0?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc001b5f800?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012d30a0)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0008d70e0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x585
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x2772

goroutine 43 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc000b30990, {0x6f19750, 0xc00062ef40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 42 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc000049710, {0x6f19750, 0xc00062ef40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 41 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc000049680, {0x6f19750, 0xc00062ef40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 40 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc0000495f0, {0x6f19750, 0xc00062ef40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 39 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc00062f0d0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc00062f0c8)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc00062f0c0)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run(0xc000c5b500, {0x6f19750, 0xc00062eb40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:75 +0x2c8
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:827 +0x72
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 157 [select, 29 minutes]:
github.com/pingcap/check.(*suiteRunner).doRun(0xc000b30fc0, 0x0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:667 +0x2a5
github.com/pingcap/check.(*suiteRunner).run(0xc000b30fc0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:696 +0xe6
github.com/pingcap/check.Run({0x5a59ee0, 0xc0005dc380}, 0xc0005cb268?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:150 +0x4a
github.com/pingcap/check.RunAll(0xc000815d02?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:117 +0x259
github.com/pingcap/check.TestingT(0xc000775520)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:99 +0x647
github.com/tikv/migration/cdc/cdc/kv.Test(0x0?)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:64 +0x70
testing.tRunner(0xc000775520, 0x5bd2c78)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1486 +0x725

goroutine 158 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc00062ef10?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc00062ef08)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc00062ef00)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool({0x6f19788, 0xc000054128})
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:829 +0x16f
github.com/tikv/migration/cdc/cdc/kv.Test.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:62 +0x3d
created by github.com/tikv/migration/cdc/cdc/kv.Test
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:61 +0x66

goroutine 1978 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc0007d2f68?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc0007d2f60)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc0001c2960)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000be94a0?, {0x5b1da9f, 0x26}, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc0001c2960)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2814
reflect.Value.call({0x5a59ee0?, 0xc0005dc380?, 0xc00008ecd0?}, {0x5aa2b5e, 0x4}, {0xc000813eb8, 0x1, 0xc00008ecf0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc0005dc380?, 0xc00045d800?}, {0xc00008eeb8, 0x1, 0x1})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:339 +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc0001c2960)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:734 +0x5de

goroutine 1981 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 2031 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f160a8?, 0xc000ebb660?}, 0xc000733c20?, 0xc000bf9b60?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc0012d2ee0, {0x6f18c88, 0xc000af07c0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:551 +0xa4f

goroutine 2008 [select]:
go.etcd.io/etcd/server/v3/mvcc/backend.(*backend).run(0xc000ada540)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:421 +0x235
created by go.etcd.io/etcd/server/v3/mvcc/backend.newBackend
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:229 +0xd36

goroutine 2054 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc000d545d0?}, 0xc001b5f740?, 0xc000e5da40?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc0012d2fc0, {0x6f18c88, 0xc000739be0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0x59
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0xf58

goroutine 2024 [select, 11 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.monitorFileDescriptor(0xc000c54a00?, 0xc000bf9860)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/metrics.go:237 +0x3f9
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Start.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:802 +0x65
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2026 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).linearizableReadLoop(0xc000c54a00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/v3_server.go:738 +0x178
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2073 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func3()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:520 +0x169
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2071 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).dispatchRequest(0xc000a188c0, {0x6f19750, 0xc000ba8b80})
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:870 +0x4d6
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:511 +0x5e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2029 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2039 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16048?, 0xc000d545d0?}, 0xc000f5f560?, 0xc000bf9ec0?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000219500, {0x6f18c88, 0xc000af0ba0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func2()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x66
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x8d8

goroutine 2009 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/wal.(*filePipeline).run(0xc000f2c500)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:97 +0x1e5
created by go.etcd.io/etcd/server/v3/wal.newFilePipeline
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:55 +0x4af

goroutine 1982 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 2061 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f5a1e0cd138, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000bb4418, 0xc000e5e000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000bb4400, {0xc000e5e000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc000bb4400, {0xc000e5e000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0005dcab0, {0xc000e5e000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
github.com/soheilhy/cmux.(*bufferedReader).Read(0xc000a702b0, {0xc000e5e000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/buffer.go:53 +0x2f8
github.com/soheilhy/cmux.(*MuxConn).Read(0xc000a702a0, {0xc000e5e000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:297 +0x51
bufio.(*Reader).Read(0xc001b5f8c0, {0xc0012d31c0, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6d00, 0xc001b5f8c0}, {0xc0012d31c0, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0012d31c0?, 0x9?, 0x9?}, {0x6ef6d00?, 0xc001b5f8c0?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012d3180)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0008eaea0, 0x5?, 0x100000001?)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:611 +0x145
google.golang.org/grpc.(*Server).serveStreams(0xc000219500, {0x6f31410?, 0xc0008eaea0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:906 +0x267
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:848 +0x65
created by google.golang.org/grpc.(*Server).handleRawConn
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:847 +0x245

goroutine 2038 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2023 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile(0xc000c54a00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:876 +0x519
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2010 [select]:
go.etcd.io/etcd/raft/v3.(*node).run(0xc0007331a0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:341 +0xc1d
created by go.etcd.io/etcd/raft/v3.StartNode
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:230 +0x31e

goroutine 2059 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0009007d0, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001b5fb00)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:332 +0x1cd
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:329 +0x269b

goroutine 2040 [select, 29 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ba84c0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x150
created by google.golang.org/grpc.newCCBalancerWrapper
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x4b7

goroutine 1985 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:174 +0x259
created by go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:169 +0xed

goroutine 1984 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000c6e2a0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7

goroutine 1983 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 2017 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncVictimsLoop(0xc000544c80)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:259 +0x209
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:96 +0xbf2

goroutine 2016 [select]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchersLoop(0xc000544c80)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:233 +0x249
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:95 +0xb75

goroutine 2013 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc0007336e0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7

goroutine 2012 [select]:
go.etcd.io/etcd/server/v3/lease.(*lessor).runLoop(0xc000ada6c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:619 +0x195
created by go.etcd.io/etcd/server/v3/lease.newLessor
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:237 +0x5f7

goroutine 2074 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:546 +0x289
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2025 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorVersions(0xc000c54a00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2465 +0x149
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

DONE 226 tests, 1 failure in 2059.616s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

@pingyu pingyu reopened this Jul 28, 2022
@pingyu
Copy link
Collaborator Author

pingyu commented Aug 2, 2022

Another case
https://github.com/tikv/migration/runs/7628273817?check_suite_focus=true

=== Failed
=== FAIL: cdc/kv Test (unknown)
Error: 8/02 09:18:33.779 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/02 09:18:34.888 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/02 09:18:36.321 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:45633, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/02 09:18:41.395 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
coverage: 67.7% of statements
panic: test timed out after 30m0s

goroutine 2222 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:2029 +0xbb
created by time.goFunc
	/opt/hostedtoolcache/go/1.18.3/x64/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 30 minutes]:
testing.(*T).Run(0xc000b7ad00, {0x5aa2f56, 0x4}, 0x5bd2c58)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1487 +0x750
testing.runTests.func1(0x0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1839 +0x9a
testing.tRunner(0xc000b7ad00, 0xc00095fb68)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
testing.runTests(0xc000ac7040?, {0x8a65680, 0x2, 0x2}, {0x40?, 0x55ed100?, 0x91ae560?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1837 +0x7e5
testing.(*M).Run(0xc000ac7040)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1719 +0xa72
main.main()
	_testmain.go:125 +0x3aa

goroutine 2044 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f89b3fa38b8, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:[302](https://github.com/tikv/migration/runs/7628273817?check_suite_focus=true#step:4:303) +0x89
internal/poll.(*pollDesc).wait(0xc0004f2b98, 0xc000df4000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004f2b80, {0xc000df4000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0004f2b80, {0xc000df4000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000011ef0, {0xc000df4000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
github.com/soheilhy/cmux.(*bufferedReader).Read(0xc000a040f0, {0xc000df4000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/buffer.go:53 +0x2f8
github.com/soheilhy/cmux.(*MuxConn).Read(0xc000a040e0, {0xc000df4000, 0x8000, 0x8000})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:297 +0x51
bufio.(*Reader).Read(0xc000d16240, {0xc000d43460, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6ce0, 0xc000d16240}, {0xc000d43460, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000d43460?, 0x9?, 0x9?}, {0x6ef6ce0?, 0xc000d16240?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000d43420)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000c1ad00, 0x1596325?, 0x4f964b6?)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:611 +0x145
google.golang.org/grpc.(*Server).serveStreams(0xc000bb5500, {0x6f313f0?, 0xc000c1ad00})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:906 +0x267
google.golang.org/grpc.(*Server).handleRawConn.func1()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:848 +0x65
created by google.golang.org/grpc.(*Server).handleRawConn
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:847 +0x245

goroutine 2018 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).linearizableReadLoop(0xc0000dee00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/v3_server.go:738 +0x178
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 39 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	/home/runner/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x265

goroutine 2065 [chan receive, 29 minutes]:
go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:252 +0x71
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).registerGateway
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:251 +0x65f

goroutine 2026 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000aabd40)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7

goroutine 1995 [select]:
go.etcd.io/etcd/server/v3/lease.(*lessor).runLoop(0xc000c263c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:619 +0x195
created by go.etcd.io/etcd/server/v3/lease.newLessor
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/lease/lessor.go:237 +0x5f7

goroutine 60 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000847f00)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0x177
created by go.opencensus.io/stats/view.init.0
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xf7

goroutine 2023 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16088?, 0xc000d133e0?}, 0xc000aab8c0?, 0xc000f9bbc0?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc000d42a80, {0x6f18c68, 0xc000d136a0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:551 +0xa4f

goroutine 2022 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16088?, 0xc000d133e0?}, 0xc000aab860?, 0xc000f9bb60?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000d0e540, {0x6f18c68, 0xc000d13640})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:545 +0x63f

goroutine 2008 [select]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchersLoop(0xc000ac6c80)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:233 +0x249
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:95 +0xb75

goroutine 2070 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:546 +0x289
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2043 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000c1ad00)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1131 +0x3cf
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:341 +0x271a

goroutine 2067 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).dispatchRequest(0xc000841900, {0x6f19730, 0xc000c05000})
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:870 +0x4d6
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:511 +0x5e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2040 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001b6c3c0, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000d16180)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x106
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x3129

goroutine 2000 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f89b3fa39a8, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc001b29298, 0xc000cdd501?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc001b29280)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc001b29280)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc000bcd908)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc000bcd908)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*limitListener).Accept(0xc000bcd920)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/limit_listen.go:46 +0x6c
go.etcd.io/etcd/client/pkg/v3/transport.(*keepaliveListener).Accept(0xc000dc4520)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/keepalive_listener.go:49 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000101450)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve(0xc000700000, 0xc0000dee00, 0xc000943960, {0x6efd060, 0xc000a5eec0}, 0xc000abe450, {0xc001b1ce60, 0x2, 0x2})
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:195 +0x1f65
go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1(0x0?)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:734 +0x145
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:733 +0x126f

goroutine 2015 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile(0xc0000dee00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:876 +0x519
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2017 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorVersions(0xc0000dee00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2465 +0x149
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2063 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc000c05050?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000c05048)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000c05040)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed(0xc000841900, {0x6f19730, 0xc000c04800}, 0x64)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:590 +0xaef
github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).EventFeed(0xc000a99630, {0x6f19730, 0xc000c04800}, {{0xc000d336df, 0x1, 0x1}, {0xc000d336ef, 0x1, 0x1}}, 0x64, ...)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:405 +0x179
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:568 +0x27c
created by github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:566 +0xe3f

goroutine 2066 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16028?, 0xc000dc4520?}, 0xc001bd09c0?, 0xc000cfbe60?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
net/http.(*Server).Serve(0xc0007dd880, {0x6f18c68, 0xc000d1ccc0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x5a7
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0x59
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:137 +0xf58

goroutine 164 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc0005db560, {0x6f19730, 0xc000b2dac0})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 163 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc0005db4d0, {0x6f19730, 0xc000b2dac0})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 162 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc0005db440, {0x6f19730, 0xc000b2dac0})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 161 [select]:
github.com/tikv/migration/cdc/pkg/workerpool.(*worker).run(0xc0005daea0, {0x6f19730, 0xc000b2dac0})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:328 +0x27d
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run.func1()
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:67 +0x54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 160 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000b2db10?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000b2db08)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000b2db00)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/pkg/workerpool.(*defaultPoolImpl).Run(0xc0004e8150, {0x6f19730, 0xc000b2da40})
	/home/runner/work/migration/migration/cdc/pkg/workerpool/pool_impl.go:75 +0x2c8
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:827 +0x72
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 159 [select, 29 minutes]:
github.com/pingcap/check.(*suiteRunner).doRun(0xc000a4f170, 0x0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:667 +0x2a5
github.com/pingcap/check.(*suiteRunner).run(0xc000a4f170)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:696 +0xe6
github.com/pingcap/check.Run({0x5a59ee0, 0xc000715668}, 0xc0004ea248?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:150 +0x4a
github.com/pingcap/check.RunAll(0xc0004fdd02?)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:117 +0x259
github.com/pingcap/check.TestingT(0xc000b7aea0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/run.go:99 +0x647
github.com/tikv/migration/cdc/cdc/kv.Test(0x0?)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:64 +0x70
testing.tRunner(0xc000b7aea0, 0x5bd2c58)
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.18.3/x64/src/testing/testing.go:1486 +0x725

goroutine 21 [semacquire, 30 minutes]:
sync.runtime_Semacquire(0xc000b2da90?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000b2da88)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
golang.org/x/sync/errgroup.(*Group).Wait(0xc000b2da80)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x37
github.com/tikv/migration/cdc/cdc/kv.RunWorkerPool({0x6f19768, 0xc000136008})
	/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:829 +0x16f
github.com/tikv/migration/cdc/cdc/kv.Test.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:62 +0x3d
created by github.com/tikv/migration/cdc/cdc/kv.Test
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:61 +0x66

goroutine 228 [select, 29 minutes]:
github.com/pingcap/check.(*resultTracker)._loopRoutine(0xc000a4f0e0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:464 +0x153
created by github.com/pingcap/check.(*resultTracker).start
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:444 +0x8f

goroutine 2069 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func3()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:520 +0x169
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2071 [select]:
github.com/tikv/migration/cdc/cdc/kv.(*sizedRegionRouter).Run(0xc000d464b0, {0x6f19730, 0xc000c05000})
	/home/runner/work/migration/migration/cdc/cdc/kv/token_region.go:151 +0x1fb
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func5()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:584 +0x82
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2024 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f89b3fa3f48, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc001b29218, 0x62e8eb01?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc001b29200)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x425
net.(*netFD).accept(0xc001b29200)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x4a
net.(*TCPListener).accept(0xc000bcd8c0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x45
net.(*TCPListener).Accept(0xc000bcd8c0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x68
go.etcd.io/etcd/client/pkg/v3/transport.(*rwTimeoutListener).Accept(0xc000d133e0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/transport/timeout_listener.go:36 +0x45
github.com/soheilhy/cmux.(*cMux).Serve(0xc000d5e5a0)
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:170 +0x122
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:557 +0x178
go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc000d133c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:585 +0x176
created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/etcd.go:579 +0xe32

goroutine 1996 [select, 29 minutes]:
go.etcd.io/etcd/pkg/v3/schedule.(*fifo).run(0xc000bbb3e0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:143 +0x409
created by go.etcd.io/etcd/pkg/v3/schedule.NewFIFOScheduler
	/home/runner/go/pkg/mod/go.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:70 +0x2f7

goroutine 2035 [select, 29 minutes]:
github.com/soheilhy/cmux.muxListener.Accept({{0x6f16028?, 0xc000dc4520?}, 0xc001b66000?, 0xc000c06ea0?})
	/home/runner/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 +0xda
google.golang.org/grpc.(*Server).Serve(0xc000bb5500, {0x6f18c68, 0xc000afa640})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:780 +0x5cb
go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func2()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x66
created by go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/embed/serve.go:120 +0x8d8

goroutine 2021 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2005 [semacquire, 29 minutes]:
sync.runtime_Semacquire(0xc000d332f8?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc000d332f0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/sync/waitgroup.go:136 +0x9c
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError.func1()
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:534 +0x85
runtime.Goexit()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:500 +0x1be
github.com/pingcap/check.(*C).stopNow(...)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:108
github.com/pingcap/check.(*C).FailNow(0xc00057cff0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:37 +0x45
github.com/pingcap/check.(*C).Fatalf(0xc000cd94a0?, {0x5b1da89, 0x26}, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/helpers.go:151 +0x157
github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestHandleError(0x3600000000000000?, 0xc00057cff0)
	/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:677 +0x2814
reflect.Value.call({0x5a59ee0?, 0xc000715668?, 0xc000bdecd0?}, {0x5aa2b5e, 0x4}, {0xc0004ffeb8, 0x1, 0xc000bdecf0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:556 +0xd9d
reflect.Value.Call({0x5a59ee0?, 0xc000715668?, 0xc000059840?}, {0xc000bdeeb8, 0x1, 0x1})
	/opt/hostedtoolcache/go/1.18.3/x64/src/reflect/value.go:339 +0xd8
github.com/pingcap/check.(*suiteRunner).forkTest.func1(0xc00057cff0)
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:850 +0xa72
github.com/pingcap/check.(*suiteRunner).forkCall.func1()
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:739 +0x11f
created by github.com/pingcap/check.(*suiteRunner).forkCall
	/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/check.go:734 +0x5de

goroutine 2068 [select, 29 minutes]:
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore(0xc000841900, {0x6f19730?, 0xc000c05000?}, 0xc000c05040)
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:690 +0x21c
github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2()
	/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515 +0x70
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f

goroutine 2028 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 2027 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:174 +0x259
created by go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/raft.go:169 +0xed

goroutine 2029 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 2020 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).monitorDowngrade(0xc0000dee00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2586 +0x156
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2009 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncVictimsLoop(0xc000ac6c80)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:259 +0x209
created by go.etcd.io/etcd/server/v3/mvcc.newWatchableStore
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/watchable_store.go:96 +0xbf2

goroutine 2016 [select, 11 minutes]:
go.etcd.io/etcd/server/v3/etcdserver.monitorFileDescriptor(0xc0000dee00?, 0xc000f9b8c0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/metrics.go:237 +0x3f9
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).Start.func3()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:802 +0x65
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 1992 [select, 29 minutes]:
go.etcd.io/etcd/server/v3/wal.(*filePipeline).run(0xc000416340)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:97 +0x1e5
created by go.etcd.io/etcd/server/v3/wal.newFilePipeline
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/wal/file_pipeline.go:55 +0x4af

goroutine 2012 [select]:
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run(0xc0000dee00)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:1117 +0xb65
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).start
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:861 +0x159e

goroutine 2039 [IO wait, 29 minutes]:
internal/poll.runtime_pollWait(0x7f89b3fa3a98, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0004f2b18, 0xc000db6000?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0xbd
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004f2b00, {0xc000db6000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:167 +0x415
net.(*netFD).Read(0xc0004f2b00, {0xc000db6000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000011ef8, {0xc000db6000, 0x8000, 0x8000})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc000d16120, {0xc000d43380, 0x9, 0x9})
	/opt/hostedtoolcache/go/1.18.3/x64/src/bufio/bufio.go:236 +0x4db
io.ReadAtLeast({0x6ef6ce0, 0xc000d16120}, {0xc000d43380, 0x9, 0x9}, 0x9)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:331 +0xde
io.ReadFull(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc000d43380?, 0x9?, 0x9?}, {0x6ef6ce0?, 0xc000d16120?})
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x96
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000d43340)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x108
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007f4960)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x585
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x2772

goroutine 2042 [select, 29 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001b6c4b0, 0x1)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x1ad
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000d16540)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x11b
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:332 +0x1cd
created by google.golang.org/grpc/internal/transport.NewServerTransport
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:329 +0x269b

goroutine 2030 [select]:
go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:85 +0x2cf
created by go.etcd.io/etcd/client/pkg/v3/fileutil.purgeFile
	/home/runner/go/pkg/mod/go.etcd.io/etcd/client/pkg/[email protected]/fileutil/purge.go:44 +0x4f2

goroutine 1991 [select]:
go.etcd.io/etcd/server/v3/mvcc/backend.(*backend).run(0xc0000e5ec0)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:421 +0x235
created by go.etcd.io/etcd/server/v3/mvcc/backend.newBackend
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/mvcc/backend/backend.go:229 +0xd36

goroutine 1993 [select]:
go.etcd.io/etcd/raft/v3.(*node).run(0xc000bbae40)
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:341 +0xc1d
created by go.etcd.io/etcd/raft/v3.StartNode
	/home/runner/go/pkg/mod/go.etcd.io/etcd/raft/[email protected]/node.go:230 +0x31e

goroutine 2034 [select]:
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.monitorLeader.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/api/v3rpc/interceptor.go:316 +0x173
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1()
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2690 +0x7b
created by go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach
	/home/runner/go/pkg/mod/go.etcd.io/etcd/server/[email protected]/etcdserver/server.go:2688 +0x1d2

goroutine 2033 [select, 29 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000c04880)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x150
created by google.golang.org/grpc.newCCBalancerWrapper
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x4b7

DONE 223 tests, 1 failure in 2029.314s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

@github-actions
Copy link

github-actions bot commented Sep 2, 2022

This issue is stale because it has been open 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/stale type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[close #177] fix unstable client_test pingyu/tikv-migration
1 participant
@pingyu