storage: TestMVCCResolveTxnRange failed

[cockroach-teamcity]

storage.TestMVCCResolveTxnRange failed on release-24.1.0-rc @ 5e4ca9e26f1a25681de9c944298cfa139c344466:

        github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
        	github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:19
        github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func2()
        	github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:161 +0x1a8
        github.com/cockroachdb/cockroach/pkg/storage.TestMVCCResolveTxnRange(0xc000540340)
        	github.com/cockroachdb/cockroach/pkg/storage/mvcc_test.go:3744 +0x1406
        testing.tRunner(0xc000540340, 0x2263168)
        	GOROOT/src/testing/testing.go:1689 +0xfb
        created by testing.(*T).Run in goroutine 1
        	GOROOT/src/testing/testing.go:1742 +0x390
        goroutine 1 [chan receive]:
        testing.(*T).Run(0xc00046dd40, {0x20c1bd6?, 0x0?}, 0x2263168)
        	GOROOT/src/testing/testing.go:1750 +0x3ab
        testing.runTests.func1(0xc00046dd40)
        	GOROOT/src/testing/testing.go:2161 +0x37
        testing.tRunner(0xc00046dd40, 0xc00037fb40)
        	GOROOT/src/testing/testing.go:1689 +0xfb
        testing.runTests(0xc000600900, {0xc00058bc08, 0x3b, 0x4a}, {0x45e90e?, 0xc000502a80?, 0x3a9d980?})
        	GOROOT/src/testing/testing.go:2159 +0x445
        testing.(*M).Run(0xc0003bdae0)
        	GOROOT/src/testing/testing.go:2027 +0x68b
        main.main()
        	main/bazel-out/k8-fastbuild/bin/pkg/storage/storage_test_/testmain.go:720 +0x4d7
        goroutine 21 [chan receive]:
        github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler.func1()
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:34 +0x1d
        created by github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler in goroutine 1
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:33 +0x98
        goroutine 7 [chan receive]:
        github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
        	github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:117 +0x4e
        created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
        	github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:83 +0x1a
        goroutine 8 [chan receive]:
        github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
        	github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:141 +0x89
        created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
        	github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:84 +0x26
        goroutine 10 [syscall]:
        os/signal.signal_recv()
        	GOROOT/src/runtime/sigqueue.go:152 +0x29
        os/signal.loop()
        	GOROOT/src/os/signal/signal_unix.go:23 +0x13
        created by os/signal.Notify.func1.1 in goroutine 8
        	GOROOT/src/os/signal/signal.go:151 +0x1f: Leaked goroutine: goroutine 21 [chan receive]:
        github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler.func1()
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:34 +0x1d
        created by github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler in goroutine 1
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:33 +0x98
--- FAIL: TestMVCCResolveTxnRange (5.01s)

Parameters:

• attempt=1
• run=15
• shard=3

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-39432

[aadityasondhi]

Not reproducible under stressrace.

Leaked goroutine: goroutine 21 [chan receive]:
        github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler.func1()
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:34 +0x1d
        created by github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler in goroutine 1
        	github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:33 +0x98

The leaked goroutine seems to be coming from bazel?

[rickystewart]

That leaked goroutine comes from rules_go. There is a Goroutine that is used to watch for signals and exit if the test times out: bazel-contrib/rules_go#3749

I'm not clear on how this happened/how it doesn't seem to be a recurring issue. If it comes up again, we can do something more aggressive about this (e.g., maybe our leaktest thing needs to be taught to ignore these goroutines).

[aadityasondhi]

Ack. Thanks for the context. Going to close this issue since it is not reproducible and the leaked goroutine is in rules.go which is monitoring test timeouts. If it comes up again, we could do what @rickystewart suggested above and have leaktest ignore those goroutines.