Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support wildcard/regex for indices param in _remotestore/_restore #8922

Merged

Conversation

linuxpi
Copy link
Collaborator

@linuxpi linuxpi commented Jul 27, 2023

Description

  • _remotestore/_restore will now support wildcards/regex for indices param.
  • Specifying body - { "indices": ["my-index-*"] } will match all indices whose name start with my-index- and trigger remote store restore.

Related Issues

Resolves #8480

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@linuxpi linuxpi force-pushed the restore-api-index-wildcard branch from 85c230d to 61aa234 Compare July 27, 2023 18:17
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

@codecov
Copy link

codecov bot commented Jul 27, 2023

Codecov Report

Attention: Patch coverage is 77.19298% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 71.66%. Comparing base (b15cb0c) to head (488ab9c).
Report is 259 commits behind head on main.

Files Patch % Lines
...in/java/org/opensearch/common/util/IndexUtils.java 84.61% 4 Missing and 4 partials ⚠️
...arch/index/recovery/RemoteStoreRestoreService.java 0.00% 4 Missing ⚠️
...ava/org/opensearch/snapshots/SnapshotsService.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8922      +/-   ##
============================================
+ Coverage     71.42%   71.66%   +0.24%     
- Complexity    59978    61106    +1128     
============================================
  Files          4985     5052      +67     
  Lines        282275   287122    +4847     
  Branches      40946    41602     +656     
============================================
+ Hits         201603   205771    +4168     
- Misses        63999    64322     +323     
- Partials      16673    17029     +356     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 1, 2023

Gradle Check (Jenkins) Run Completed with:

Copy link
Contributor

@BhumikaSaini-Amazon BhumikaSaini-Amazon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @linuxpi!
Please could you also enable the ITs in RemoteStoreIT that are currently awaiting fix on #8480? We need to pass those tests as part of this enhancement.

@opensearch-trigger-bot
Copy link
Contributor

This PR is stalled because it has been open for 30 days with no activity. Remove stalled label or comment or this will be closed in 7 days.

@linuxpi
Copy link
Collaborator Author

linuxpi commented May 2, 2024

@linuxpi I've cleaned up the conflict, and verify this is in a green state, before we merge this improvement please add a test case(s) that verifies the regex/wildcard scenario.

Integ tests for this already existed before my change but were skipped due to AwaitsFix annotation. I've remove this and now these tests are run as part of gradle check.

@linuxpi linuxpi added v2.15.0 Issues and PRs related to version 2.15.0 and removed v2.14.0 labels May 2, 2024
Copy link
Contributor

github-actions bot commented May 2, 2024

❌ Gradle check result for 56d100a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi
Copy link
Collaborator Author

linuxpi commented May 2, 2024

@peternied
Copy link
Member

peternied commented May 2, 2024

@linuxpi Thanks for checking on the test failure, I've restarted the gradle-check workflow

Copy link
Contributor

github-actions bot commented May 2, 2024

❌ Gradle check result for 56d100a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bansal <[email protected]>
@peternied
Copy link
Member

@linuxpi Please investigate the tests failures after the results come back:

* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.14.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.x/distribution/archives/linux-tar/build/distributions/opensearch-min-2.14.0-SNAPSHOT-linux-x64.tar.gz

Check will keep failing until you've rebased.

Copy link
Contributor

github-actions bot commented May 2, 2024

❌ Gradle check result for e1f55c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented May 3, 2024

❌ Gradle check result for e1f55c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented May 3, 2024

❕ Gradle check result for 488ab9c: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@sachinpkale sachinpkale merged commit 29ebd80 into opensearch-project:main May 3, 2024
28 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-8922-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 29ebd80fdd6771ccc2023bf432977c38c76bfd55
# Push it to GitHub
git push --set-upstream origin backport/backport-8922-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8922-to-2.x.

@linuxpi linuxpi deleted the restore-api-index-wildcard branch May 3, 2024 10:02
linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request May 3, 2024
…ensearch-project#8922)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: bansvaru <[email protected]>
(cherry picked from commit 29ebd80)
@linuxpi
Copy link
Collaborator Author

linuxpi commented May 3, 2024

backport pr #13527

linuxpi added a commit to linuxpi/OpenSearch that referenced this pull request May 3, 2024
…ensearch-project#8922)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: Varun Bansal <[email protected]>
(cherry picked from commit 29ebd80)
sachinpkale pushed a commit that referenced this pull request May 3, 2024
) (#13527)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: Varun Bansal <[email protected]>
(cherry picked from commit 29ebd80)
finnegancarroll pushed a commit to finnegancarroll/OpenSearch that referenced this pull request May 10, 2024
…ensearch-project#8922)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: bansvaru <[email protected]>
deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request May 17, 2024
…ensearch-project#8922)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: bansvaru <[email protected]>
parv0201 pushed a commit to parv0201/OpenSearch that referenced this pull request Jun 10, 2024
…ensearch-project#8922)

* Support wildcard/regex for indices param in _remotestore/_restore

Signed-off-by: bansvaru <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed bug Something isn't working enhancement Enhancement or improvement to existing feature or request Storage Issues and PRs relating to data and metadata storage v2.15.0 Issues and PRs related to version 2.15.0
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[BUG] [Remote Store] Wildcards and regex are not inferred by _remotestore/_restore API
9 participants