Adds counter metrics for leader and follower check failures #12439

gargharsh3134 · 2024-02-23T09:24:49Z

Description

Emits counter metrics to track the number of failures in leader checks made by the follower nodes and follower checks performed by the leader node.

Related Issues

Resolves #[12711]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-02-23T09:39:04Z

❌ Gradle check result for 67c3ca7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-23T09:45:34Z

Compatibility status:

Checks if related components are compatible with change 6166519

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/sql.git]

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java

...framework/src/main/java/org/opensearch/cluster/coordination/AbstractCoordinatorTestCase.java

github-actions · 2024-02-29T22:06:25Z

❌ Gradle check result for f2e731e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-15T05:57:27Z

❌ Gradle check result for bf79851: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-03-26T05:29:25Z

❌ Gradle check result for 4fd90ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-26T05:34:49Z

❌ Gradle check result for 48e38b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/coordination/ClusterManagerMetrics.java

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java

server/src/test/java/org/opensearch/cluster/coordination/FollowersCheckerTests.java

server/src/test/java/org/opensearch/telemetry/TestInMemoryMetricsRegistry.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-04-10T05:48:08Z

❌ Gradle check result for 6166519: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

rajiv-kv

LGTM ! Some minor comments.

server/src/main/java/org/opensearch/cluster/coordination/ClusterManagerMetrics.java

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-05-22T13:49:45Z

✅ Gradle check result for dd39aee: SUCCESS

codecov · 2024-05-22T13:53:16Z

Codecov Report

Attention: Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.

Project coverage is 71.60%. Comparing base (b15cb0c) to head (5246ed7).
Report is 351 commits behind head on main.

Files	Patch %	Lines
.../org/opensearch/cluster/ClusterManagerMetrics.java	66.66%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12439      +/-   ##
============================================
+ Coverage     71.42%   71.60%   +0.18%     
- Complexity    59978    61377    +1399     
============================================
  Files          4985     5071      +86     
  Lines        282275   288403    +6128     
  Branches      40946    41764     +818     
============================================
+ Hits         201603   206521    +4918     
- Misses        63999    64848     +849     
- Partials      16673    17034     +361

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-05-22T13:55:46Z

✅ Gradle check result for c295e40: SUCCESS

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-05-23T12:02:13Z

❌ Gradle check result for 13954dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

gargharsh3134 · 2024-05-23T14:31:44Z

❌ Gradle check result for 13954dc: FAILURE

#10006 Failing Test is Flaky

github-actions · 2024-05-23T20:03:18Z

❌ Gradle check result for 13954dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-05T12:39:09Z

✅ Gradle check result for 13954dc: SUCCESS

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-06-06T06:59:00Z

✅ Gradle check result for 5246ed7: SUCCESS

opensearch-trigger-bot · 2024-06-06T07:04:26Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12439-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0b2e012c266f4d954e3c3875276880c15efd5e2c
# Push it to GitHub
git push --set-upstream origin backport/backport-12439-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12439-to-2.x.

…ch-project#12439) * Adds counter metrics for leader and follower check failures Signed-off-by: Harsh Garg <[email protected]>

rajiv-kv reviewed Feb 29, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java Outdated Show resolved Hide resolved

...framework/src/main/java/org/opensearch/cluster/coordination/AbstractCoordinatorTestCase.java Outdated Show resolved Hide resolved

gargharsh3134 force-pushed the leaderCheckMetrics branch from f2e731e to bf79851 Compare March 15, 2024 05:31

Harsh Garg added 2 commits March 26, 2024 10:45

Adds counter metrics for leader and follower check failures

04b4cb6

Signed-off-by: Harsh Garg <[email protected]>

Refactoring UTs to assert counters

63eba93

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 force-pushed the leaderCheckMetrics branch from bf79851 to 4fd90ac Compare March 26, 2024 05:20

Adding TestMetricsRegistry for UTs

48e38b0

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 force-pushed the leaderCheckMetrics branch from 4fd90ac to 48e38b0 Compare March 26, 2024 05:22

rajiv-kv reviewed Mar 27, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/coordination/ClusterManagerMetrics.java Outdated Show resolved Hide resolved

rajiv-kv reviewed Mar 27, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java Outdated Show resolved Hide resolved

rajiv-kv reviewed Mar 27, 2024

View reviewed changes

server/src/test/java/org/opensearch/cluster/coordination/FollowersCheckerTests.java Show resolved Hide resolved

rajiv-kv reviewed Mar 27, 2024

View reviewed changes

server/src/test/java/org/opensearch/cluster/coordination/FollowersCheckerTests.java Outdated Show resolved Hide resolved

rajiv-kv reviewed Mar 27, 2024

View reviewed changes

server/src/test/java/org/opensearch/telemetry/TestInMemoryMetricsRegistry.java Outdated Show resolved Hide resolved

Adding asserts in unit tests for no leader or failure checks

6166519

Signed-off-by: Harsh Garg <[email protected]>

rajiv-kv reviewed Apr 23, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/coordination/ClusterManagerMetrics.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/coordination/LeaderChecker.java Outdated Show resolved Hide resolved

rwali-aws assigned gargharsh3134 May 16, 2024

Harsh Garg added 3 commits May 22, 2024 16:40

Merge branch 'main' into leaderCheckMetrics

83f3028

Signed-off-by: Harsh Garg <[email protected]>

Moving leaderCheckFailureCounter outside of Transport call

dd39aee

Signed-off-by: Harsh Garg <[email protected]>

Added ChangeLog

c295e40

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 marked this pull request as ready for review May 22, 2024 13:04

gargharsh3134 requested review from anasalkouz, andrross, Bukhtawar and CEHENKLE as code owners May 22, 2024 13:04

gargharsh3134 requested review from tlfeng and VachaShah as code owners May 22, 2024 13:04

Moving followerChecksFailureCounter outside of Transport call

13954dc

Signed-off-by: Harsh Garg <[email protected]>

rajiv-kv approved these changes May 23, 2024

View reviewed changes

sachinpkale added the backport 2.x Backport to 2.x branch label Jun 5, 2024

sachinpkale approved these changes Jun 5, 2024

View reviewed changes

Merge branch 'main' into leaderCheckMetrics

5246ed7

Signed-off-by: Harsh Garg <[email protected]>

sachinpkale approved these changes Jun 6, 2024

View reviewed changes

sachinpkale merged commit 0b2e012 into opensearch-project:main Jun 6, 2024
30 checks passed

opensearch-trigger-bot bot added the backport-failed label Jun 6, 2024

gargharsh3134 mentioned this pull request Jun 10, 2024

[Backport] ClusterManager metrics instrumentation changes #14118

Merged

3 tasks

finnegancarroll mentioned this pull request Jun 14, 2024

Remove composeBuild fixture tasks when docker support not found #14357

Merged

3 tasks

prudhvigodithi mentioned this pull request Jun 18, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

BrewTestBot mentioned this pull request Jun 26, 2024

opensearch 2.15.0 Homebrew/homebrew-core#175637

Merged

This was referenced Jun 27, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds counter metrics for leader and follower check failures #12439

Adds counter metrics for leader and follower check failures #12439

gargharsh3134 commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 29, 2024

github-actions bot commented Mar 15, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 10, 2024

rajiv-kv left a comment

github-actions bot commented May 22, 2024

codecov bot commented May 22, 2024 •

edited

Loading

github-actions bot commented May 22, 2024

github-actions bot commented May 23, 2024

gargharsh3134 commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 6, 2024

opensearch-trigger-bot bot commented Jun 6, 2024

Adds counter metrics for leader and follower check failures #12439

Adds counter metrics for leader and follower check failures #12439

Conversation

gargharsh3134 commented Feb 23, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Feb 29, 2024

github-actions bot commented Mar 15, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 10, 2024

rajiv-kv left a comment

Choose a reason for hiding this comment

github-actions bot commented May 22, 2024

codecov bot commented May 22, 2024 • edited Loading

Codecov Report

github-actions bot commented May 22, 2024

github-actions bot commented May 23, 2024

gargharsh3134 commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 6, 2024

opensearch-trigger-bot bot commented Jun 6, 2024

gargharsh3134 commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 23, 2024 •

edited

Loading

codecov bot commented May 22, 2024 •

edited

Loading

gargharsh3134 commented May 23, 2024 •

edited

Loading