Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jobs: TestMetrics failed #65735

Closed
cockroach-teamcity opened this issue May 26, 2021 · 0 comments · Fixed by #65936
Closed

jobs: TestMetrics failed #65735

cockroach-teamcity opened this issue May 26, 2021 · 0 comments · Fixed by #65936
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

jobs.TestMetrics failed with artifacts on master @ 821d3153b6925104a692b28ad21bb2e82156f051:

=== RUN   TestMetrics
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestMetrics832649711
    test_log_scope.go:74: use -show-logs to present logs inline
=== CONT  TestMetrics
    jobs_test.go:2681: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestMetrics832649711
--- FAIL: TestMetrics (2.33s)
=== RUN   TestMetrics/restart,_pause,_resume,_then_success
E210526 18:21:11.442943 81432 jobs/adopt.go:284  [n1] 7  job 661857473033502721: adoption completed with error job 661857473033502721: : restarting in background
    jobs_test.go:2597: 
        	Error Trace:	jobs_test.go:2597
        	Error:      	Received unexpected error:
        	            	job with status pause-requested cannot be resumed
        	Test:       	TestMetrics/restart,_pause,_resume,_then_success
E210526 18:21:11.511114 80515 jobs/registry.go:697  [-] 8  error claiming jobs: could not query jobs table: claim-jobs: context canceled
E210526 18:21:11.511417 80515 jobs/registry.go:707  [-] 9  error processing claimed jobs: could not query for claimed jobs: select-running/get-claimed-jobs: context canceled
E210526 18:21:11.548571 80433 jobs/registry.go:687  [-] 10  failed to serve pause and cancel requests: node unavailable; try another peer
    --- FAIL: TestMetrics/restart,_pause,_resume,_then_success (0.52s)
Reproduce

To reproduce, try: 

make stressrace TESTS=TestMetrics PKG=./pkg/jobs TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

  • GOFLAGS=-json

/cc @cockroachdb/sql-schema ajwerner

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 26, 2021
@ajwerner ajwerner mentioned this issue Jun 1, 2021
Merged
craig bot pushed a commit that referenced this issue Jun 1, 2021
@sumeerbhola @RichardJCai
@adityamaru @ajwerner
Merge #65867 #65868 #65900 #65910 #65925 #65933 #65934 #65936
f5b21cf 
65867: changefeedccl: Fix flaky tests. r=miretskiy a=miretskiy

Fix flaky test and re-enable it to run under stress.
The problem was that the transaction executed by the table feed can
be restarted.  If that happens, then we would see the same keys again,
but because we had side effects inside transaction (marking the keys
seen), we would not emit those keys causing the test to be hung.
The stress race was failing because of both transaction restarts and
the 10ms resolved timestamp frequency (with so many resolved timestamps
being generated, the table feed transaction was always getting
restarted).

Fixes #57754
Fixes #65168

Release Notes: None

65868: storage: expose pebble.IteratorStats through {MVCC,Engine}Iterator r=sumeerbhola a=sumeerbhola

These will potentially be aggregated before exposing in trace
statements, EXPLAIN ANALYZE etc.

Release note: None

65900: roachtest: fix ruby-pg test suite r=rafiss a=RichardJCai

Update blocklist with passing test.
The not run test causing a failure is because the test is no longer failing.
Since it is not failing, it shows up under not run.

Release note: None

65910: sql/gcjob: retry failed GC jobs r=ajwerner a=sajjadrizvi

In the previous implementation, failed GC jobs were not being retried regardless
whether the failure is permanent or transient. As a result, a GC job's failure
risked orphaned data, which cannot be reclaimed.

This commit adds a mechanism to retry failed GC jobs that are not permanent. No
limit is set on the number of retries. For the time being, the failure type is
determined based on the failure categorization of schema-change jobs. This
behavior is expected to change once exponential backoff mechanism is
implemented for failed jobs (#44594).

Release note: None

Fixes: #65000

Release note (<category, see below>): <what> <show> <why>

65925: ccl/importccl: skip TestImportPgDumpSchemas/inject-error-ensure-cleanup r=tbg a=adityamaru

Refs: #65878

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

65933: kv/kvserver: skip TestReplicateQueueDeadNonVoters under race r=sumeerbhola a=sumeerbhola

Refs: #65932

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

65934: kv/kvserver: skip TestReplicateQueueSwapVotersWithNonVoters under race r=sumeerbhola a=sumeerbhola

Refs: #65932

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

65936: jobs: fix flakey TestMetrics r=fqazi a=ajwerner

Fixes #65735

The test needed to wait for the job to be fully marked as paused.

Release note: None

Co-authored-by: Yevgeniy Miretskiy <[email protected]>
Co-authored-by: sumeerbhola <[email protected]>
Co-authored-by: richardjcai <[email protected]>
Co-authored-by: Sajjad Rizvi <[email protected]>
Co-authored-by: Aditya Maru <[email protected]>
Co-authored-by: Andrew Werner <[email protected]>
@craig craig bot closed this as completed in 221c555 Jun 1, 2021
ajwerner added a commit to ajwerner/cockroach that referenced this issue Jun 21, 2021
@ajwerner
jobs: fix flakey TestMetrics
d3d5306 
Fixes cockroachdb#65735

The test needed to wait for the job to be fully marked as paused.

Release note: None
@ajwerner ajwerner mentioned this issue Jun 21, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

jobs: fix flakey TestMetrics ajwerner/cockroach
1 participant
@cockroach-teamcity