roachtest: clearrange/checks=true failed

[cockroach-teamcity]

roachtest.clearrange/checks=true failed with artifacts on release-21.1 @ c425111e138297bcacd1370cbe40263dd00e64ac:

		  |  3592.0s        0         8144.6         5835.2      3.4      8.4     14.2     24.1 write
		  |  3593.0s        0         8150.2         5835.9      3.4      8.1     13.6     23.1 write
		  |  3594.0s        0         8633.8         5836.7      3.1      7.9     12.6     23.1 write
		  |  3595.0s        0         8309.0         5837.3      3.3      8.1     14.2     23.1 write
		  |  3596.0s        0         8460.1         5838.1      3.3      7.9     13.6     23.1 write
		  |  3597.0s        0         8162.3         5838.7      3.4      8.4     14.7     29.4 write
		  |  3598.0s        0         8166.9         5839.4      3.4      8.9     14.2     23.1 write
		  |  3599.0s        0         8035.6         5840.0      3.4      8.4     13.6     30.4 write
		  |  3600.0s        0         8294.8         5840.7      3.4      7.9     14.7     24.1 write
		  | Highest sequence written: 21026391. Can be passed as --write-seq=R21026391 to the next run.
		  |
		  | _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
		  |  3600.0s        0       21026359         5840.7      5.5      4.2     12.6     27.3   6442.5  write
		  |
		  | _elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
		  |  3600.0s        0       21026359         5840.7      5.5      4.2     12.6     27.3   6442.5
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,clearrange.go:187,clearrange.go:34,test_runner.go:733: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:187
		  | main.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:34
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:733
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh clearrange/checks=true

Same failure on other branches

• roachtest: clearrange/zfs/checks=true failed #68303 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-master]
• roachtest: clearrange/checks=true failed #65092 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-20.1 release-blocker]
• roachtest: clearrange/checks=true failed #64764 roachtest: clearrange/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-master release-blocker]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

[erikgrinaker]

Error: ERROR: result is ambiguous (unable to determine whether command was applied via snapshot) (SQLSTATE 40003)
COMMAND_PROBLEM: exit status 1

This failure seems "fine". We should make workload resistant to ambiguous errors, but closing this for now.

@tbg Let me know if you think this error is unexpected or needs investigation.