kv/kvserver: TestReplicateQueueUpReplicateOddVoters failed

[cockroach-teamcity]

kv/kvserver.TestReplicateQueueUpReplicateOddVoters failed with artifacts on master @ 506d129f5f187134c35e2f71860490e044fde989:

F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc006344f90)
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:292 +0x58
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !created by github.com/cockroachdb/pebble.(*tableCacheShard).init
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:284 +0xd1
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !goroutine 71962476 [select]:
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func1()
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:217 +0x8e
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:213 +0x285
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !goroutine 72042089 [chan receive]:
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x561d8b8, 0xc01f282210})
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:293 +0x5b
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !runtime/pprof.Do({0x561d848, 0xc000074108}, {{0xc0005ee820, 0xfaaa80, 0x561d8b8}}, 0xc0082b2fa8)
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/usr/local/go/src/runtime/pprof/runtime.go:40 +0xa3
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0082b2f90)
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:292 +0x58
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !created by github.com/cockroachdb/pebble.(*tableCacheShard).init
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:284 +0xd1
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !For more context, check log files in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueUpReplicateOddVoters1027669390
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !****************************************************************************
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !This node experienced a fatal error (printed above), and as a result the
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !process is terminating.
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !will try to determine the root cause, recommend next steps, and we can
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !improve CockroachDB based on your report.
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !Please submit a crash report by following the instructions here:
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !If you would rather not post publicly, please contact us directly at:
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !    [email protected]
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !
F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348  [n2,s2,r36/2:{-}] 1 !The Cockroach Labs team appreciates your feedback.


ERROR: exit status 7

9 runs completed, 1 failures, over 58m46s
context canceled

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

• TAGS=deadlock

• GOFLAGS=-parallel=4

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

[tbg]

F211204 08:42:36.775340 72457976 kv/kvserver/store_raft.go:348 â‹® [n2,s2,r36/2:{-}] 787 while applying snapshot: while applying snapshot: error rate limiting bulk io write: context canceled

Will be fixed by #73484 cc @erikgrinaker