Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create documentation for snapshots with hashed prefix path type #8196

Merged
merged 7 commits into from
Sep 11, 2024
Merged

Create documentation for snapshots with hashed prefix path type #8196

merged 7 commits into from
Sep 11, 2024

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Sep 9, 2024 •
edited by kolchfa-aws
Loading

Description

This PR aims to add documentation for the following features -

  1. RFC proposed here - [RFC] Optimized Prefix Pattern for Shard-Level Files for Efficient Snapshots OpenSearch#15146.
  2. Add path prefix support to hashed prefix snapshots OpenSearch#15664
  3. Add prefix support to hashed prefix & infix path types on remote store OpenSearch#15557

Issues Resolved

Closes #8143

Version

This PR applies to version 2.17 and later.

Frontend features

If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.
NONE

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 9, 2024

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Naarcha-AWS
Naarcha-AWS requested changes Sep 10, 2024
View reviewed changes
Copy link
Collaborator

@Naarcha-AWS Naarcha-AWS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of follow up questions for you @gbbafna and @ashking94.

@Naarcha-AWS Naarcha-AWS added 4 - Doc review PR: Doc review in progress v2.17.0 release-notes PR: Include this PR in the automated release notes labels Sep 10, 2024
@ashking94
Copy link
Member Author

A couple of follow up questions for you @gbbafna and @ashking94.

Thanks for working on this PR @Naarcha-AWS. I have responded to your question.

@Naarcha-AWS Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress 6 - Done but waiting to merge PR: The work is done and ready to merge and removed 4 - Doc review PR: Doc review in progress labels Sep 10, 2024
natebower
natebower reviewed Sep 11, 2024
View reviewed changes
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashking94 @Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md Outdated Show resolved Hide resolved
_api-reference/snapshots/create-repository.md

- `FIXED`: Keeps the path structure in the existing hierarchical manner, such as, `<ROOT>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`
- `HASHED_PREFIX`: Prepends a hashed prefix at the start of path for each unique shard ID, for example, `<ROOT>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/<BASE_PATH>/indices/<index-id>/0/<SHARD_BLOBS>`.
- `HASHED_INFIX`: Appends a hashed prefix after the base path for each unique shard ID, for example, `<ROOT>/<BASE-PATH>/<HASH-OF-INDEX-ID-AND-SHARD-ID>/indices/<index-id>/0/<SHARD_BLOBS>`. The hash method used is `fnv_1a_composite_1, which uses the `FNV1a` hash function and generates a custom-encoded 64-bit hash value that scales well with most remote store options. `FNV1a` takes the most significant 6 bits to create a url-safe base64 character and the next 14 bits to create a binary string.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Same question re: fnv_1a_composite_1. I believe it may have been in all caps in a previous PR. Also, it looks like you have an unclosed tic mark here.

_install-and-configure/configuring-opensearch/index-settings.md Outdated Show resolved Hide resolved
_install-and-configure/configuring-opensearch/index-settings.md Outdated Show resolved Hide resolved
_install-and-configure/configuring-opensearch/index-settings.md Outdated Show resolved Hide resolved
@Naarcha-AWS @natebower
Apply suggestions from code review
5e7ca66 
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
@Naarcha-AWS Naarcha-AWS merged commit 9bc06e4 into opensearch-project:main Sep 11, 2024
6 checks passed
epugh pushed a commit to o19s/documentation-website that referenced this pull request Sep 12, 2024
@ashking94 @Naarcha-AWS
@natebower @epugh
Create documentation for snapshots with hashed prefix path type (open…
426cd65 
…search-project#8196)

* Create documentation for snapshots with hashed prefix path type

Signed-off-by: Ashish Singh <[email protected]>

* Add documentation on new cluster settings for fixed prefix

Signed-off-by: Ashish Singh <[email protected]>

* Update create-repository.md

* Update create-repository.md

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Eric Pugh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@natebower natebower natebower left review comments

@Naarcha-AWS Naarcha-AWS Naarcha-AWS approved these changes

@gbbafna gbbafna gbbafna approved these changes

@kolchfa-aws kolchfa-aws Awaiting requested review from kolchfa-aws kolchfa-aws is a code owner

@vagimeli vagimeli Awaiting requested review from vagimeli vagimeli is a code owner

@AMoo-Miki AMoo-Miki Awaiting requested review from AMoo-Miki AMoo-Miki is a code owner

@dlvenable dlvenable Awaiting requested review from dlvenable dlvenable is a code owner

@stephen-crawford stephen-crawford Awaiting requested review from stephen-crawford

@epugh epugh Awaiting requested review from epugh epugh is a code owner

Assignees

@Naarcha-AWS Naarcha-AWS

Labels
5 - Editorial review PR: Editorial review in progress 6 - Done but waiting to merge PR: The work is done and ready to merge release-notes PR: Include this PR in the automated release notes v2.17.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[DOC] Create documentation for snapshots with hashed prefix path type
5 participants
@ashking94 @gbbafna @Naarcha-AWS @natebower @kolchfa-aws