Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Segment Replication] Flaky test SegmentReplicationTargetServiceTests.testReplicationOnDone #4312

Closed
dreamer-89 opened this issue Aug 27, 2022 · 2 comments
Closed

[Segment Replication] Flaky test SegmentReplicationTargetServiceTests.testReplicationOnDone #4312

dreamer-89 opened this issue Aug 27, 2022 · 2 comments
Labels
distributed framework flaky-test Random test failure that succeeds on second run

Comments

@dreamer-89
Copy link
Member

dreamer-89 commented Aug 27, 2022 •
edited
Loading

Describe the bug

Flaky test failure last reported here in #4309. This test has caused CI check failure few times.

Error Message
org.mockito.exceptions.verification.TooFewActualInvocations: 
segmentReplicationTargetService.onNewCheckpoint(
    ReplicationCheckpoint{shardId=[index][0], primaryTerm=19, segmentsGen=2, seqNo=-1, version=7},
    <any>
);
Wanted 2 times:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone(SegmentReplicationTargetServiceTests.java:257)
But was 1 time:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone(SegmentReplicationTargetServiceTests.java:251)
Stacktrace
org.mockito.exceptions.verification.TooFewActualInvocations: 
segmentReplicationTargetService.onNewCheckpoint(
    ReplicationCheckpoint{shardId=[index][0], primaryTerm=19, segmentsGen=2, seqNo=-1, version=7},
    <any>
);
Wanted 2 times:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone(SegmentReplicationTargetServiceTests.java:257)
But was 1 time:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone(SegmentReplicationTargetServiceTests.java:251)

	at __randomizedtesting.SeedInfo.seed([7845FD34BB0C54B3:628B2DF971F0BB11]:0)
	at app//org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone(SegmentReplicationTargetServiceTests.java:257)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at [email protected]/java.lang.Thread.run(Thread.java:833)

Related
#3969

Additional context
Add any other context about the problem here.
@dreamer-89 dreamer-89 added untriaged flaky-test Random test failure that succeeds on second run labels Aug 27, 2022
@dreamer-89
Copy link
Member Author

This test fails roughly 50% times on local.

./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testReplicationOnDone" -Dtests.seed=7845FD34BB0C54B3 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=vi-VN -Dtests.timezone=Asia/Aqtobe -Druntime.java=17 -Dtests.iters=20

This was referenced Aug 27, 2022
Merged
Merged
@dreamer-89
Copy link
Member Author

Relevant change merged into main. Resolving this issue.

Side-note: This test is also removed in one of the latest segrep PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
distributed framework flaky-test Random test failure that succeeds on second run
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adnapibar @dreamer-89