Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] org.elasticsearch.xpack.ml.integration.ClassificationIT.testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows failure #50168

Closed
dliappis opened this issue Dec 13, 2019 · 3 comments · Fixed by #50322
Closed

[CI] org.elasticsearch.xpack.ml.integration.ClassificationIT.testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows failure #50168

dliappis opened this issue Dec 13, 2019 · 3 comments · Fixed by #50322
Assignees
@dimitris-athanasiou
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@dliappis
Copy link
Contributor

Example failure: https://gradle-enterprise.elastic.co/s/uohtdst7yu7lk

Started occurring after #50040 got merged.
Stack trace:

org.elasticsearch.xpack.ml.integration.ClassificationIT > testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: <1> |  
  | but: was <0> |  
  | at __randomizedtesting.SeedInfo.seed([6B678FC96FDF9BF5:7F732A40232997FC]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.assertModelStatePersisted(MlNativeDataFrameAnalyticsIntegTestCase.java:282) |  
  | at org.elasticsearch.xpack.ml.integration.ClassificationIT.testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows(ClassificationIT.java:98)

Fails in

elasticsearch/x-pack/plugin/ml/qa/native-multi-node-tests/src/test/java/org/elasticsearch/xpack/ml/integration/MlNativeDataFrameAnalyticsIntegTestCase.java

Line 282 in 68c739f

assertThat(searchResponse.getHits().getHits().length, equalTo(1));

returning 0 hits.

I can reproduce it locally with:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.ClassificationIT.testSingleNumericFeatureAndMixedTrainingAndNonTrainingRows" -Dtests.seed=6B678FC96FDF9BF5 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=fi-FI -Dtests.timezone=Europe/Isle_of_Man -Dcompiler.java=13 -Dlicense.key=<path_to_release_key>
@dliappis dliappis added >test-failure Triaged test failures from CI :ml Machine learning labels Dec 13, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@dliappis
Copy link
Contributor Author

@dimitris-athanasiou do you think you could take a look?

@martijnvg
Copy link
Member

martijnvg commented Dec 18, 2019 •
edited
Loading

Other instances of this failure:

@dimitris-athanasiou dimitris-athanasiou self-assigned this Dec 18, 2019
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[ML] Refresh state index before completing data frame analytics job
b8e864c 
In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in elastic#50168 and elastic#50313 where
the model state appears missing.

Closes elastic#50168
Closes elastic#50313
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Dec 18, 2019
Merged
dimitris-athanasiou added a commit that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[ML] Refresh state index before completing data frame analytics job (#…
4d8d8d9 
…50322)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in #50168 and #50313 where
the model state appears missing.

Closes #50168
Closes #50313
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[7.x][ML] Refresh state index before completing data frame analytics …
d8d9cc7 
…job (elastic#50322)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in elastic#50168 and elastic#50313 where
the model state appears missing.

Closes elastic#50168
Closes elastic#50313

Backport of elastic#50322
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Dec 18, 2019
Merged
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[7.5][ML] Refresh state index before completing data frame analytics …
32050d4 
…job (elastic#50322)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in elastic#50168 and elastic#50313 where
the model state appears missing.

Closes elastic#50168
Closes elastic#50313

Backport of elastic#50322
@dimitris-athanasiou dimitris-athanasiou mentioned this issue Dec 18, 2019
Merged
dimitris-athanasiou added a commit that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[7.5][ML] Refresh state index before completing data frame analytics …
f41bea0 
…job (#50322) (#50325)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in #50168 and #50313 where
the model state appears missing.

Closes #50168
Closes #50313

Backport of #50322
dimitris-athanasiou added a commit that referenced this issue Dec 18, 2019
@dimitris-athanasiou
[7.x][ML] Refresh state index before completing data frame analytics …
d3c83cd 
…job (#50322) (#50324)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in #50168 and #50313 where
the model state appears missing.

Closes #50168
Closes #50313

Backport of #50322
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
@dimitris-athanasiou @SivagurunathanV
[ML] Refresh state index before completing data frame analytics job (e…
65f7a01 
…lastic#50322)

In order to ensure any persisted model state is searchable by the moment
the job reports itself as `stopped`, we need to refresh the state index
before completing.

This should fix the occasional failures we see in elastic#50168 and elastic#50313 where
the model state appears missing.

Closes elastic#50168
Closes elastic#50313
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimitris-athanasiou dimitris-athanasiou

Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@martijnvg @dliappis @dimitris-athanasiou @elasticmachine