Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testWatcherRestart occasionally fails in REST tests #33753

Closed
DaveCTurner opened this issue Sep 17, 2018 · 15 comments
Closed

testWatcherRestart occasionally fails in REST tests #33753

DaveCTurner opened this issue Sep 17, 2018 · 15 comments
Assignees
@jakelandis @pgomulka
Labels
:Data Management/Watcher >test-failure Triaged test failures from CI v6.4.2 v7.2.0

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Sep 17, 2018 •
edited
Loading

I count 10 failures of WatchBackwardsCompatibilityIT#testWatcherRestart since June 13th, all in the 6.x series (i.e. 6.2, 6.3 or 6.4 as well as 6.x). They are not all the same as the failure described here, but some of them are. The latest one was in a BWC test but non-BWC tests also failed in this fashion.

The build link is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+bwc-tests/1/console

The reproduction line follows, but does not reproduce for me:

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0#oldClusterTestRunner \
  -Dtests.seed=A6F6BC1F82877D71 \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Etc/GMT+4

The output follows.

java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
	at __randomizedtesting.SeedInfo.seed([A6F6BC1F82877D71:33AEB290C99BA428]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:804)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$12(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:273)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.ensureWatcherStopped(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.testWatcherRestart(WatchBackwardsCompatibilityIT.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
		... 42 more
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@dliappis
Copy link
Contributor

dliappis commented Oct 2, 2018

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/111/consoleText

@danielmitterdorfer
Copy link
Member

Looking at the failure output in the original comment, we see that on one node (node id -34h6dtFQ2SsSYIJ_n0LKw), Watcher is in the state starting and then moves to started over time. So maybe the problem is that before we attempt to stop Watcher in:

elasticsearch/x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/WatchBackwardsCompatibilityIT.java

Line 183 in 230cd10

executeAgainstRandomNode(client -> assertOK(client.performRequest("POST", "/_xpack/watcher/_stop")));

we'd actually need to ensure that all watches are started?

@spinscale / @hub-cap can one of you please have a look at this one?

@andrershov
Copy link
Contributor

Occurred again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/58/console

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:without-system-key:v6.4.3-SNAPSHOT#oldClusterTestRunner \
  -Dtests.seed=EA5213BE918AF24A \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=el \
  -Dtests.timezone=Africa/Lagos \
  -Dcompiler.java=11 \
  -Druntime.java=8

@jakelandis jakelandis assigned jakelandis and unassigned jakelandis Oct 25, 2018
@colings86 colings86 added v6.6.0 and removed v6.5.0 labels Oct 25, 2018
@pgomulka pgomulka mentioned this issue Nov 5, 2018
Merged
@jaymode
Copy link
Member

jaymode commented Nov 13, 2018

Another instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/56/console

@albertzaharovits
Copy link
Contributor

albertzaharovits commented Nov 22, 2018 •
edited
Loading

And another one:
https://groups.google.com/a/elastic.co/d/msg/build-elasticsearch/LXzJGUAL0WQ/wD4F1v4TBAAJ

java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
	at __randomizedtesting.SeedInfo.seed([7AADDEF1EDDF9F81:EFF5D07EA6C346D8]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$12(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:273)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.ensureWatcherStopped(WatchBackwardsCompatibilityIT.java:297)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.testWatcherRestart(WatchBackwardsCompatibilityIT.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
		... 42 more
	Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
		at org.junit.Assert.assertThat(Assert.java:956)
		at org.junit.Assert.assertThat(Assert.java:923)
		at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)

@cbuescher
Copy link
Member

And another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/369/console

@cbuescher
Copy link
Member

And another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/357/console

@cbuescher
Copy link
Member

I start to think we should mute this, especially since there seems to be a PR on the way to fix: #35243

@cbuescher
Copy link
Member

And another one. This should definitely be muted IMHO: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/96/console

@cbuescher
Copy link
Member

I muted this on 6.x with c847953 and on 6.5 with 15632d7.

I haven't seen any failures on master since this seems to be a bwc thing, also there are three other similarly named WatchBackwardsCompatibilityIT in other qa modules which I haven't muted so far since the failures I've seen so far only were in rolling-upgrade:without-system-key

@jasontedor jasontedor added v6.7.0 and removed v6.6.0 labels Dec 19, 2018
pgomulka added a commit that referenced this issue Jan 10, 2019
@pgomulka
Fix failing failing testWatcherRestart (#35243)
7f20c0b 
Reenabling the test on 6.x as  #35271 looks like fixed the problem. If that helps on 6.x it will be back ported to 6.6, 6.5 and previous versions where this test is muted.

closes #33753
@Tim-Brooks
Copy link
Contributor

There was a failure on 6.x for this test today:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/926/console

@pgomulka - Is this still an issue?

@cbuescher
Copy link
Member

@tbrooks8 @pgomulka 7f20c0b apparently didn't solve the issue, I'll revert that commit on 6.x

@cbuescher
Copy link
Member

Reverted in 633aac1

@jakelandis
Copy link
Contributor

I am closing this issue since this specific test upgrading TO 6.x, and as such was removed from 7.x (a long time ago). The test is currently muted in 6.x (non-existent in 7+) and I don't believe it worth effort to track down a possible root cause of the flakey-ness of this test since 7.0 has been released and we haven't had real world issues w/r/t Watcher and upgrading TO 6.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jakelandis jakelandis

@pgomulka pgomulka

Labels
:Data Management/Watcher >test-failure Triaged test failures from CI v6.4.2 v7.2.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests