[BUG] org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats is flaky #10735

gbbafna · 2023-10-19T06:50:03Z

Describe the bug
https://build.ci.opensearch.org/job/gradle-check/28391/

So many runs in gradle are failing due to same.

gbbafna · 2023-10-19T07:08:33Z

We have just muted this . Will need to fix this test as it is critical for repository stats to work

mch2 · 2023-10-19T16:54:52Z

FYI have a reproducible seed here with #10730

akolarkunnu · 2024-05-21T13:43:31Z

I analyzed this bug in detail and got the solution.
@rramachand21 Can you please assign this bug to me, if you are not working on this.

peternied · 2024-05-21T15:54:52Z

@akolarkunnu Thanks for jumping on this issue!

akolarkunnu · 2024-05-22T05:35:44Z

Main part of error log is:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([F562E26F95091BFA:EAE65676A0DA30C5]:0)
at org.opensearch.repositories.s3.S3BlobStore.extendedStats(S3BlobStore.java:243)
at org.opensearch.repositories.blobstore.BlobStoreRepository.stats(BlobStoreRepository.java:859)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:558)
at org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats(S3BlobStoreRepositoryTests.java:210)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)

It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue.

akolarkunnu · 2024-05-22T05:38:34Z

As a solution, we can set a proper GenericStatsMetricPublisher object from test.

rramachand21 · 2024-05-23T03:24:11Z

@akolarkunnu thanks for your contribution on this. Let me know if you are blocked on any of the processes up until your fix is merged and I can help follow up.

…Stats fails with NullPointerException It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves opensearch-project#10735 Signed-off-by: Abdul Muneer Kolarkunnu <[email protected]>

…Stats fails with NullPointerException It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves opensearch-project#10735 Signed-off-by: akolarkunnu <[email protected]>

…Stats fails with NullPointerException (#13814) It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves #10735 Signed-off-by: akolarkunnu <[email protected]> Co-authored-by: akolarkunnu <[email protected]>

reta · 2024-05-28T20:16:22Z

The issue is not fixed:

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([1B7179E203DFF64:1E33A38715EED45B]:0)
	at org.opensearch.repositories.RepositoryStats.merge(RepositoryStats.java:88)
	at java.base/java.util.stream.ReduceOps$2ReducingSink.accept(ReduceOps.java:123)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:662)
	at org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats(S3BlobStoreRepositoryTests.java:209)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

…Stats fails with NullPointerException It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves opensearch-project#10735 Signed-off-by: Abdul Muneer Kolarkunnu <[email protected]> Signed-off-by: akolarkunnu <[email protected]>

peternied · 2024-05-29T15:24:19Z

[Triage - attendees 1 2 3 4 5 6
@reta Thanks for reopening this issue

prudhvigodithi · 2024-05-29T17:56:56Z

Coming from the metrics dashboard, I can see the S3BlobStoreRepositoryTests as top hitter and failed on multiple PR's.

msfroh · 2024-05-30T05:48:30Z

I added an extra check that seems to fix a reliably reproducible seed on my machine: #13887

…Stats fails with NullPointerException (#13866) It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves #10735 Signed-off-by: Abdul Muneer Kolarkunnu <[email protected]> Signed-off-by: akolarkunnu <[email protected]> Co-authored-by: akolarkunnu <[email protected]>

…Stats fails with NullPointerException (opensearch-project#13814) It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves opensearch-project#10735 Signed-off-by: akolarkunnu <[email protected]> Co-authored-by: akolarkunnu <[email protected]>

…Stats fails with NullPointerException (opensearch-project#13866) It's a NullPointerException from S3BlobStore.extendedStats() method, where 'genericStatsMetricPublisher' is null. This parameter sets through S3Repository constructor from test and in this test it sets as null. This is the root cause of the issue. If we set valid a GenericStatsMetricPublisher, test works fine without any issue. This was a consistent failure, not a random failure. Resolves opensearch-project#10735 Signed-off-by: Abdul Muneer Kolarkunnu <[email protected]> Signed-off-by: akolarkunnu <[email protected]> Co-authored-by: akolarkunnu <[email protected]> Signed-off-by: kkewwei <[email protected]>

gbbafna added bug Something isn't working untriaged labels Oct 19, 2023

gbbafna mentioned this issue Oct 19, 2023

Muting s3 request stats test #10736

Merged

7 tasks

sachinpkale closed this as completed in #10736 Oct 19, 2023

gbbafna reopened this Oct 19, 2023

gbbafna added the Storage Issues and PRs relating to data and metadata storage label Oct 19, 2023

mch2 mentioned this issue Oct 19, 2023

[BUG] S3BlobStoreRepositoryTests.testRequestStats is flaky #10730

Closed

rramachand21 removed the untriaged label Oct 20, 2023

gbbafna mentioned this issue Oct 24, 2023

[Repository Plugin] Fix stats test #10880

Closed

8 tasks

reta mentioned this issue Dec 4, 2023

[Backport] [2.x] Bump actions/setup-java from 3 to 4 (#11447) #11453

Merged

ankitkala assigned rramachand21 Dec 16, 2023

saratvemulapalli mentioned this issue Jan 24, 2024

Fixing Minio docker compose fixture #12014

Merged

6 tasks

Bukhtawar added flaky-test Random test failure that succeeds on second run good first issue Good for newcomers labels May 16, 2024

peternied assigned akolarkunnu May 21, 2024

rramachand21 removed their assignment May 23, 2024

jed326 mentioned this issue May 25, 2024

org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats fails with NullPointerException #13814

Merged

9 tasks

jed326 closed this as completed in #13814 May 25, 2024

reta reopened this May 28, 2024

github-actions bot added the untriaged label May 28, 2024

reta mentioned this issue May 28, 2024

Remove deprecated constant META_FIELDS_BEFORE_7DOT8 #13860

Merged

9 tasks

akolarkunnu mentioned this issue May 29, 2024

Backport of 13814 - org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequest… #13866

Merged

shiv0408 mentioned this issue May 29, 2024

[AUTOCUT] Gradle Check Failure on push to main #13867

Closed

This was referenced May 29, 2024

[AUTOCUT] Gradle Check Failure on push to main #13872

Closed

[Streaming Indexing] Enhance RestAction with request / response streaming support #13772

Merged

peternied removed good first issue Good for newcomers untriaged labels May 29, 2024

This was referenced May 29, 2024

Avoid negative scores returned from multi_match query with cross_fields #13829

Merged

Fix flaky test S3BlobStoreRepositoryTests.testRequestStats #13887

Merged

rayshrey mentioned this issue May 30, 2024

[Writable Warm] Composite Directory implementation and integrating it with FileCache #12782

Merged

8 tasks

reta closed this as completed in #13887 May 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats is flaky #10735

[BUG] org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats is flaky #10735

gbbafna commented Oct 19, 2023

gbbafna commented Oct 19, 2023

mch2 commented Oct 19, 2023

akolarkunnu commented May 21, 2024 •

edited

Loading

peternied commented May 21, 2024

akolarkunnu commented May 22, 2024 •

edited

Loading

akolarkunnu commented May 22, 2024 •

edited

Loading

rramachand21 commented May 23, 2024

reta commented May 28, 2024

peternied commented May 29, 2024

prudhvigodithi commented May 29, 2024 •

edited

Loading

msfroh commented May 30, 2024 •

edited

Loading

[BUG] org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats is flaky #10735

[BUG] org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testRequestStats is flaky #10735

Comments

gbbafna commented Oct 19, 2023

gbbafna commented Oct 19, 2023

mch2 commented Oct 19, 2023

akolarkunnu commented May 21, 2024 • edited Loading

peternied commented May 21, 2024

akolarkunnu commented May 22, 2024 • edited Loading

akolarkunnu commented May 22, 2024 • edited Loading

rramachand21 commented May 23, 2024

reta commented May 28, 2024

peternied commented May 29, 2024

prudhvigodithi commented May 29, 2024 • edited Loading

msfroh commented May 30, 2024 • edited Loading

akolarkunnu commented May 21, 2024 •

edited

Loading

akolarkunnu commented May 22, 2024 •

edited

Loading

akolarkunnu commented May 22, 2024 •

edited

Loading

prudhvigodithi commented May 29, 2024 •

edited

Loading

msfroh commented May 30, 2024 •

edited

Loading