Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestReplicateQueueRebalance failed #73482

Closed
cockroach-teamcity opened this issue Dec 5, 2021 · 2 comments · Fixed by #73484
Closed

kv/kvserver: TestReplicateQueueRebalance failed #73482

cockroach-teamcity opened this issue Dec 5, 2021 · 2 comments · Fixed by #73484
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

kv/kvserver.TestReplicateQueueRebalance failed with artifacts on master @ 8b696d17241b2eb2d333e23750cf6fdc433897d8:

F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0212cf900, {0x3bb4940, 0x7d8eac0})
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x479
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !runtime/pprof.Do({0x561f368, 0xc000074108}, {{0xc0005d0660, 0xc021bf2090, 0xc007b51f80}}, 0xc007b51fc0)
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/pebble/record.NewLogWriter.func2()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x5c
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/cockroachdb/pebble/record.NewLogWriter
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x329
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !goroutine 59337086 [select]:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func1()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:217 +0x8e
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:213 +0x285
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !goroutine 76054908 [runnable]:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !github.com/sasha-s/go-deadlock.lock.func1()
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:182
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !created by github.com/sasha-s/go-deadlock.lock
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:182 +0x16f
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !For more context, check log files in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueRebalance2770073773
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !****************************************************************************
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !This node experienced a fatal error (printed above), and as a result the
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !process is terminating.
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !will try to determine the root cause, recommend next steps, and we can
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !improve CockroachDB based on your report.
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !Please submit a crash report by following the instructions here:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !If you would rather not post publicly, please contact us directly at:
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !    [email protected]
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !
F211205 08:51:09.450053 75981868 kv/kvserver/store_raft.go:348  [n4,s4,r44/6:{-}] 1 !The Cockroach Labs team appreciates your feedback.


ERROR: exit status 7

9 runs completed, 1 failures, over 59m16s
context canceled
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • TAGS=deadlock

  • GOFLAGS=-parallel=4

/cc @cockroachdb/kv

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 5, 2021
@erikgrinaker erikgrinaker mentioned this issue Dec 5, 2021
Merged
@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestReplicateQueueRebalance failed with artifacts on master @ 39923c0b11d229b394fa5498ee58e455cae8ec99:

F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:284 +0xd1
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !goroutine 996601 [chan receive]:
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x5620498, 0xc01bae11d0})
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:293 +0x5b
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !runtime/pprof.Do({0x5620428, 0xc000074108}, {{0xc0005a4da0, 0xfa7280, 0x5620498}}, 0xc0055b67a8)
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0055b6790)
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:292 +0x58
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !created by github.com/cockroachdb/pebble.(*tableCacheShard).init
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:284 +0xd1
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !goroutine 996477 [chan receive]:
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x5620498, 0xc00fbd8ff0})
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:293 +0x5b
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !runtime/pprof.Do({0x5620428, 0xc000074108}, {{0xc0005a4da0, 0xfa7280, 0x5620498}}, 0xc0062edfa8)
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0062edf90)
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:292 +0x58
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !created by github.com/cockroachdb/pebble.(*tableCacheShard).init
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:284 +0xd1
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !For more context, check log files in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueRebalance865755404
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !****************************************************************************
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !This node experienced a fatal error (printed above), and as a result the
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !process is terminating.
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !will try to determine the root cause, recommend next steps, and we can
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !improve CockroachDB based on your report.
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !Please submit a crash report by following the instructions here:
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !If you would rather not post publicly, please contact us directly at:
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !    [email protected]
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !
F211207 08:39:29.494125 1145169 kv/kvserver/store_raft.go:346  [n5,s5,r67/4:{-}] 1 !The Cockroach Labs team appreciates your feedback.


ERROR: exit status 7

9 runs completed, 1 failures, over 33m56s
context canceled
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

  • GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Dec 7, 2021

This will be fixed by #73484 as well cc @erikgrinaker

@craig craig bot closed this as completed in #73484 Dec 7, 2021
@craig craig bot closed this as completed in 67662d8 Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: use background context when applying snapshots erikgrinaker/cockroach
2 participants
@tbg @cockroach-teamcity