roachtest: clearrange/checks=true failed

[cockroach-teamcity]

roachtest.clearrange/checks=true failed with artifacts on release-21.1 @ 55f6c679a1b87113bea998e15f991007a13c412f:

		  |   135.0s        0         2606.1         2758.6     11.5     23.1     33.6     44.0 write
		  |   136.0s        0         2668.1         2758.0     11.0     23.1     35.7     46.1 write
		  |   137.0s        0         3074.9         2760.3      9.4     19.9     33.6     88.1 write
		  |   138.0s        0         3412.4         2765.0      8.9     19.9     32.5     41.9 write
		  |   139.0s        0         3433.7         2769.8      8.4     21.0     31.5     56.6 write
		  |   140.0s        0         3569.9         2775.5      8.4     19.9     31.5     58.7 write
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   141.0s        0         2328.0         2772.4      7.1     18.9     31.5     58.7 write
		  |   142.0s        0          120.0         2753.7      2.6    104.9   1543.5   1543.5 write
		  |   143.0s        0         4006.5         2762.4      6.3     16.3     31.5   1744.8 write
		  |   144.0s        0         4279.0         2773.0      6.3     13.6     30.4    218.1 write
		  |   145.0s        0          354.0         2756.3      3.1      6.6     10.5    805.3 write
		  |   146.0s        0         2226.6         2752.7      4.5     11.0   1543.5   1744.8 write
		  |   147.0s        0          115.0         2734.7      2.8      7.1      9.4      9.4 write
		  | Error: ERROR: result is ambiguous (unable to determine whether command was applied via snapshot) (SQLSTATE 40003)
		  | COMMAND_PROBLEM: exit status 1
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2688,clearrange.go:187,clearrange.go:34,test_runner.go:733: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2676
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2684
		  | main.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:187
		  | main.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:34
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:733
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2732
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2646
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh clearrange/checks=true

Same failure on other branches

• roachtest: clearrange/zfs/checks=true failed #70306 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
• roachtest: clearrange/zfs/checks=true failed #68303 roachtest: clearrange/zfs/checks=true failed [C-test-failure O-roachtest O-robot T-storage branch-master]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

[jbowens]

    | Error: ERROR: result is ambiguous (unable to determine whether command was applied via snapshot) (SQLSTATE 40003)

@cockroachdb/kv — Is this failure unexpected?

[erikgrinaker]

I think it's expected. It means that it isn't safe to apply the command, because it may have been reordered wrt. other commands in a Raft snapshot we received, and we can't tell whether the snapshot already applied the command or not. I suppose the workload should really retry these kinds of errors, but we can just close this for now.