Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] ReactiveStorageIT testScaleDuringSplitOrClone failing #88478

Closed
pxsalehi opened this issue Jul 12, 2022 · 1 comment · Fixed by #88607
Closed

[CI] ReactiveStorageIT testScaleDuringSplitOrClone failing #88478

pxsalehi opened this issue Jul 12, 2022 · 1 comment · Fixed by #88607
Assignees
@fcofdez
Labels
:Distributed/Autoscaling Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@pxsalehi
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/7lj6thapivvd4/tests/:x-pack:plugin:autoscaling:internalClusterTest/org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT/testScaleDuringSplitOrClone

Reproduction line:
./gradlew ':x-pack:plugin:autoscaling:internalClusterTest' --tests "org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT.testScaleDuringSplitOrClone" -Dtests.seed=E99F6DC356CB8C62 -Dtests.locale=ru-RU -Dtests.timezone=America/Louisville -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT&tests.test=testScaleDuringSplitOrClone

Failure excerpt:

java.lang.NullPointerException: Cannot invoke "org.elasticsearch.xpack.autoscaling.capacity.AutoscalingCapacity.total()" because the return value of "org.elasticsearch.xpack.autoscaling.capacity.AutoscalingDeciderResults.requiredCapacity()" is null

  at __randomizedtesting.SeedInfo.seed([E99F6DC356CB8C62:F8CE624DFC822EC0]:0)
  at org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageIT.testScaleDuringSplitOrClone(ReactiveStorageIT.java:398)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@pxsalehi pxsalehi added :Distributed/Autoscaling >test-failure Triaged test failures from CI labels Jul 12, 2022
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Jul 12, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

This was referenced Jul 12, 2022
Merged
Merged
elasticsearchmachine pushed a commit that referenced this issue Jul 12, 2022
@pxsalehi
Mute ReactiveStorageIT#testScaleDuringSplitOrClone (#88480)
c636e43 
Relates to #88478.
@fcofdez fcofdez self-assigned this Jul 19, 2022
@fcofdez fcofdez mentioned this issue Jul 19, 2022
Merged
fcofdez added a commit to fcofdez/elasticsearch that referenced this issue Jul 19, 2022
@fcofdez
Fix ReactiveStorageIT#testScaleDuringSplitOrClone
ce447dc 
Sometimes the autoscaling decider returns an empty response when
the service does not have enough information to provide an autoscaling
decision, i.e. when a new node joins it tries to fetch the new node
memory info and this might take a while. This commits adds a busy
assertion to ensure that a valid autoscaling capacity is provided
eventually.

Closes elastic#88478
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fcofdez fcofdez

Labels
:Distributed/Autoscaling Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix ReactiveStorageIT#testScaleDuringSplitOrClone fcofdez/elasticsearch
3 participants
@fcofdez @pxsalehi @elasticmachine