[BUG] flaky test index/80_geo_point/Single point test

[peternied]

Describe the bug
Looks like there is a flaky test case index/80_geo_point/Single point test inside of org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test

java.lang.AssertionError: Failure at [index/80_geo_point:60]: hits.total didn't match expected value:
hits.total: expected Integer [6] but was Integer [5]

To Reproduce
Steps to reproduce the behavior:

1. REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v2.4.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=index/80_geo_point/Single point test}" -Dtests.seed=9E49FBE4E90B87AB -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-AR -Dtests.timezone=Asia/Tehran -Druntime.java=17

Expected behavior
There should be no flaky tests in the gradle check

Host/Environment (please complete the following information):

• Jenkins build environment

Additional context

• Gist of the test failure https://gist.github.com/peternied/22c8fec19528bf6bc79683923c15f5b4
• Jenkins Gradle Check workflow https://build.ci.opensearch.org/job/gradle-check/5071/consoleFull
• Impacted pull request Add project health badges #4843 (comment)

Please react with 🤔 if you see further instances and paste a link to the impacted pull request in this issue.

[ketanv3]

Another PR impacted by this - #4805

[kotwanikunal]

Test consistently failing for #4870

[heemin32]

Test succeeded in #4805. #4870 will succeed once it is rebased.

[andrross]

Fixed by #4860

[peternied]

It looks like the test has been disabled - yay for not failing gradle checks! - but it has not been fixed. I think this issue should stay open until the test is fixed or deleted.

[heemin32]

The test is not disabled. The confusion is coming from the naming of the test. The test is newly added with new feature of geojson format support for point type in version 2.4. Therefore, it fails with previous version which does not support the geojson format for point type.

Let me separate the two cases out instead of combining it...

[peternied]

@heemin32 Good detail!

I am curious if the test was running against versions of OpenSearch that did not support the feature, then I would expect gradle check to be blocked since the test was merged - do you know how the test presented itself as intermittently failing?

[andrross]

@peternied My understanding is that in the mixed cluster test case the requests may go to any node. If the requests requiring the new feature happen to hit only the nodes with the newer version of OpenSearch, then the tests will pass. Hence in the case of a compatibility issue you might see intermittent successes and failures.

Let me separate the two cases out instead of combining it...

@heemin32 What action are you taking here?

[heemin32]

@andrross Creating test case for testing all previous formats and creating another test case testing geojson format for only version 2.4 and above. Do you think we need that?

[peternied]

I am going to close out this issue since its tracking the test failure itself, up to @heemin32 if you'd like to re-work the test case to be more obvious about its dependencies.

If the requests requiring the new feature happen to hit only the nodes with the newer version of OpenSearch, then the tests will pass.

This sounds like tests are inherently non-deterministic - seems like something we should try to fix as it might be the root cause of many other intermittent failures. @andrross Could you open an issue to track separately?

[andrross]

@peternied Opened an issue here #5257