Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: clearrange/zfs/checks=true failed #70306

Closed
cockroach-teamcity opened this issue Sep 16, 2021 · 18 comments
Closed

roachtest: clearrange/zfs/checks=true failed #70306

cockroach-teamcity opened this issue Sep 16, 2021 · 18 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ ed81878bb4697c3a0a09e69b8a5053566297b142:

		Wraps: (2) output in run_093214.819996426_n1_cockroach_workload_fixtures_import_bank
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3452192-1631773211-40-n10cpu16:1 -- ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank returned
		  | stderr:
		  | I210916 09:32:15.864130 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3452192-1631773211-40-n10cpu16 --oneshot --ignore-empty-nodes: exit status 1 6: 13773
		1: dead (exit status 137)
		3: 13473
		5: 13788
		8: 13851
		7: 13596
		10: 13253
		4: 14140
		9: 13335
		2: 13344
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 1: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 16, 2021
@blathers-crl blathers-crl bot added the T-storage Storage Team label Sep 16, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ a29fd0ad538d51050d82e92583a8206983622f6b:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@jbowens jbowens removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 20, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 5202c97fcfe7e51e244335c3cd68bd33647860ff:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 57bb9046cd2a3090dc5f17b79912f34db8dde33e:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ d1231cff60125b397ccce6c79c9aeea771cdcca4:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ c5a6b266917ee3846dbd7ae1126c6a5d55cf439b:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 79e5979416cb426092a83beff0be1c20aebf84c6:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ ab508bd125bcf2fdb791d887e0aa26ebd64a8bcc:

		7: 14868
		9: 15206
		8: dead (exit status 137)
		10: 14503
		6: 14560
		3: 14659
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 8: dead (exit status 137)
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:225
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) 8: dead (exit status 137)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) 1: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ cec7f76e9e9daf50c545b4282c971139d9115b24:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ af7f257bc39988e6d98d516d7fe7cb842a22ad67:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 67ef976cb1ca1ca306669dff103a218eab4832f3:

		  | stderr:
		  | I211125 10:29:25.588146 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 1 tables
		  | Error: importing fixture: importing table bank: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./cockroach workload fixtures import bank --payload-bytes=10240 --ranges=10 --rows=65104166 --seed=4 --db=bigbank
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3795399-1637824840-41-n10cpu16 --oneshot --ignore-empty-nodes: exit status 1 1: dead (exit status 137)
		5: 13654
		9: 13789
		6: 14269
		3: 13579
		10: 13725
		2: 14346
		4: 13872
		7: 14449
		8: 14366
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:596
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:569
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (3) 1: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 81d49bd3d2e8ee3da4a5800043b94aa174127ab7:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 999d35029f33a9b35dde0599c952efc83ed726a7:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ c22ac33a023323cb76fde367b766ef988d6e27f7:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ dc72c17f280dc20fa59970209c0e975e4072f91e:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ 12c606395dc2cfb35bd82972ab65c4453f33c400:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ f26513f0b00c95e2e98d5ce3032abd7e65ff0e3d:

The test failed on branch=release-21.2, cloud=gce:
test timed out (see artifacts for details)
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.clearrange/zfs/checks=true failed with artifacts on release-21.2 @ b4a27f3634e3e41dc3966214cf05ff6870c055c5:

		8: 13895
		6: 13777
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 3: dead (exit status 137)
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:596
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:569
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:255
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		  | Wraps: (2) 3: dead (exit status 137)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:596
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:569
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:255
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) 1: dead (exit status 137)
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

@nicktrav
Copy link
Collaborator

This hasn't fired in a while, and the last instance looked like a transient issue with a dead node on import. Will close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Projects
None yet
Development

No branches or pull requests

3 participants