Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] XDocsClientYamlTestSuiteIT failed because of no .watches index #34448

Closed
DaveCTurner opened this issue Oct 15, 2018 · 2 comments · Fixed by #35271
Closed

[CI] XDocsClientYamlTestSuiteIT failed because of no .watches index #34448

DaveCTurner opened this issue Oct 15, 2018 · 2 comments · Fixed by #35271
Assignees
Labels

Comments

@DaveCTurner
Copy link
Contributor

Two seemingly-unrelated tests failed for the same reason in the same run:

   > Throwable #1: java.lang.AssertionError: 4 active tasks found:
   > indices:data/write/update      YJDcklJoRF-6apksAvIohw:845 -                          transport 1539506880310 08:48:00 10.4s       127.0.0.1 node-0 
   > indices:data/write/update[s]   YJDcklJoRF-6apksAvIohw:846 -                          direct    1539506880321 08:48:00 10.4s       127.0.0.1 node-0 
   > indices:data/write/bulk        YJDcklJoRF-6apksAvIohw:847 -                          transport 1539506880322 08:48:00 10.4s       127.0.0.1 node-0 requests[1], indices[.watches]
   > indices:data/write/bulk[s]     YJDcklJoRF-6apksAvIohw:848 YJDcklJoRF-6apksAvIohw:847 transport 1539506880322 08:48:00 10.4s       127.0.0.1 node-0 requests[1], index[.watches]
   >  expected:<0> but was:<4>
   > 	at __randomizedtesting.SeedInfo.seed([8E320CEFCF5C0656:666333561A06BAE]:0)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForPendingTasks$3(XPackRestTestHelper.java:134)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:847)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:821)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForPendingTasks(XPackRestTestHelper.java:110)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForPendingTasks(XPackRestTestHelper.java:97)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestIT.cleanup(XPackRestIT.java:246)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError: 4 active tasks found:
   > indices:data/write/update      YJDcklJoRF-6apksAvIohw:845 -                          transport 1539506880310 08:48:00 410ms       127.0.0.1 node-0 
   > indices:data/write/update[s]   YJDcklJoRF-6apksAvIohw:846 -                          direct    1539506880321 08:48:00 399.1ms     127.0.0.1 node-0 
   > indices:data/write/bulk        YJDcklJoRF-6apksAvIohw:847 -                          transport 1539506880322 08:48:00 397.9ms     127.0.0.1 node-0 requests[1], indices[.watches]
   > indices:data/write/bulk[s]     YJDcklJoRF-6apksAvIohw:848 YJDcklJoRF-6apksAvIohw:847 transport 1539506880322 08:48:00 397.8ms     127.0.0.1 node-0 requests[1], index[.watches]
   >  expected:<0> but was:<4>
   > 		at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForPendingTasks$3(XPackRestTestHelper.java:134)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:835)
   > 		... 38 more

The server logs indicate that the .watches index does not exist:

[2018-10-14T08:48:21,707][WARN ][o.e.a.b.TransportShardBulkAction] [node-0] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[.watches][0]] containing [index {[.watches][doc][my-watch], source[{"trigger":{"schedule":{"cron":"0 0/1 * * * ?"}},"input":{"search":{"request":{"search_type":"query_then_fetch","indices":["logstash*"],"types":[],"body":{"query":{"bool":{"filter":{"range":{"@timestamp":{"from":"{{ctx.trigger.scheduled_time}}||-5m","to":"{{ctx.trigger.triggered_time}}"}}},"must":{"match":{"response":404}}}}}}}},"condition":{"compare":{"ctx.payload.hits.total":{"gt":0}}},"actions":{"email_admin":{"email":{"profile":"standard","to":["[email protected]"],"subject":"404 recently encountered"}}},"status":{"state":{"active":true,"timestamp":"2018-10-14T08:47:59.870Z"},"actions":{"email_admin":{"ack":{"timestamp":"2018-10-14T08:47:59.870Z","state":"awaits_successful_execution"}}},"headers":{"_xpack_security_authentication":"s6LxAgAKdGVzdF9hZG1pbgEJc3VwZXJ1c2VyCgAAAAEABm5vZGUtMAxkZWZhdWx0X2ZpbGUEZmlsZQA="},"version":-1,"last_checked":"2018-10-14T08:48:00.169Z","execution_state":"execution_not_needed"}}]}]]
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:137) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:795) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:730) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:881) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:303) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:191) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:492) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3566) [?:?]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) [?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:489) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:472) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

The run was
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=amazon/60/console and apparently this can be reproduced with

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=8E320CEFCF5C0656 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/security/authorization/role-templates/line_53}" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-UY \
  -Dtests.timezone=America/Argentina/Tucuman \
  -Dcompiler.java=11 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=8E320CEFCF5C0656 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/security/authentication/saml-guide/line_636}" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-UY \
  -Dtests.timezone=America/Argentina/Tucuman \
  -Dcompiler.java=11 \
  -Druntime.java=8

However this does not reproduce for me.

I searched for similar issues and found #34013 and #33864 which relate to the security index. Perhaps this is similar? Pinging @nik9000, since you're looking at X-pack test cleanup code?

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@nik9000
Copy link
Member

nik9000 commented Oct 15, 2018

I'll add it to the pile, yeah.

@nik9000 nik9000 self-assigned this Oct 15, 2018
@colings86 colings86 added v6.6.0 and removed v6.5.0 labels Oct 25, 2018
@jakelandis jakelandis self-assigned this Nov 6, 2018
jakelandis added a commit that referenced this issue Nov 7, 2018
#35271)

Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
jakelandis added a commit that referenced this issue Nov 8, 2018
#35271)

Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
jakelandis added a commit that referenced this issue Nov 8, 2018
#35271)

Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
pgomulka pushed a commit to pgomulka/elasticsearch that referenced this issue Nov 13, 2018
elastic#35271)

Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes elastic#33291, closes elastic#29877, closes elastic#34462, closes elastic#30705, closes elastic#34448
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants