docs(RFC): Object Versioning #2602

suyanhanx · 2023-07-06T13:05:23Z

This PR introduces a new feature object_versioning.

Originally posted by Xuanwo #2156

Some storage services, such as Amazon S3, have built-in support for versioning.

This is achieved through a feature called ObjectVersion, which allows the same object to exist in multiple versions and be accessed separately even after deletion. With this feature, users can ensure the safety of their data by rolling back to previous versions in case of unintended deletions or changes.

Signed-off-by: suyanhanx <[email protected]>

core/src/docs/rfcs/2602_object_versioning.md

Signed-off-by: suyanhanx <[email protected]>

core/src/docs/rfcs/2602_object_versioning.md

Xuanwo · 2023-07-07T11:10:35Z

I invite GPT-4 to read this proposal as a newbie of rust and storage systems, he has the following problems. Would you like to answer them in the RFC's rational section?

The following questions is from GPT-4:

This RFC proposes the object versioning feature for OpenDAL. Here are some aspects that I do not quite understand:

How are Version IDs generated? The RFC mentions that each version will have a unique Version ID. However, it does not specify how these IDs are generated nor does it clarify whether there is an inherent order in these IDs.
If a delete operation removes the latest version of an object, which version is returned when a read or stat operation is performed without specifying a Version ID? Is it the second latest version or would it return an error?

These are the areas in the RFC that I am unclear about. I would appreciate if these could be clarified.

Signed-off-by: suyanhanx <[email protected]>

Xuanwo

Mostly LGTM, this RFC is ready to merge. Please fix some minor issues and create the tracking issue for this RFC.

core/src/docs/rfcs/mod.rs

core/src/docs/rfcs/2602_object_versioning.md

Signed-off-by: suyanhanx <[email protected]>

Xuanwo

Thanks, let's go!

suyanhanx · 2023-07-09T13:17:46Z

Any reason for delaying implementing list_versions?
Without it, users would have no way from us to get the versions list. @Xuanwo

Xuanwo · 2023-07-09T13:42:51Z

Any reason for delaying implementing list_versions?

The API is not user-friendly and does not meet any user requests.

suyanhanx · 2023-07-09T15:58:51Z

Any reason for delaying implementing list_versions?

The API is not user-friendly and does not meet any user requests.

🥲 Weird.

Xuanwo · 2023-07-09T16:09:03Z

Weird.

Changing my mind is easy: bring me a user and their use case.

@oowl

* feat(binding/lua): add rename and create_dir operator function by @oowl in apache#2564 * feat(services/azblob): support sink by @suyanhanx in apache#2574 * feat(services/gcs): support sink by @suyanhanx in apache#2576 * feat(services/oss): support sink by @suyanhanx in apache#2577 * feat(services/obs): support sink by @suyanhanx in apache#2578 * feat(services/cos): impl sink by @suyanhanx in apache#2587 * feat(service): Support stat for Dropbox by @Zheaoli in apache#2588 * feat(services/dropbox): impl create_dir and polish error handling by @suyanhanx in apache#2600 * feat(services/dropbox): Implement refresh token support by @Xuanwo in apache#2604 * feat(service/dropbox): impl batch delete by @suyanhanx in apache#2606 * feat(CI): set Kvrocks test for service redis by @suyanhanx in apache#2613 * feat(core): object versioning APIs by @suyanhanx in apache#2614 * feat(oay): actually read configuration from `oay.toml` by @messense in apache#2615 * feat(services/webdav): impl sink by @suyanhanx in apache#2622 * feat(services/fs): impl Sink for Fs by @Ji-Xinyou in apache#2626 * feat(core): impl `delete_with` on blocking operator by @suyanhanx in apache#2633 * feat(bindings/C): add support for list in C binding by @Ji-Xinyou in apache#2448 * refactor(core): Add ErrorKind InvalidInput to indicate users input error by @dqhl76 in apache#2637 * fix(doc): fix codeblock rendering by @xxchan in apache#2592 * fix(service/minitrace): should set local parent by @andylokandy in apache#2620 * fix(service/minitrace): update doc by @andylokandy in apache#2621 * doc(bindings/haskell): add module document by @silver-ymz in apache#2566 * docs: Update license related comments by @Prashanth-Chandra in apache#2573 * docs: add hdfs namenode High Availability related troubleshoot by @wcy-fdu in apache#2601 * docs: polish release doc by @PsiACE in apache#2608 * docs(blog): add Apache OpenDAL(Incubating): Access Data Freely by @PsiACE in apache#2607 * docs(RFC): Object Versioning by @suyanhanx in apache#2602 * ci: Disable bindings/java deploy for now by @tisonkun in apache#2560 * ci: Disable the failed stage-release job instead by @tisonkun in apache#2561 * ci: add haddock generator for haskell binding by @silver-ymz in apache#2569 * ci(binding/lua): add luarocks package manager support by @oowl in apache#2558 * build(deps): bump predicates from 2.1.5 to 3.0.1 by @dependabot in apache#2583 * build(deps): bump tower-http from 0.4.0 to 0.4.1 by @dependabot in apache#2582 * build(deps): bump chrono from 0.4.24 to 0.4.26 by @dependabot in apache#2581 * build(deps): bump redis from 0.22.3 to 0.23.0 by @dependabot in apache#2580 * build(deps): bump cbindgen from 0.24.3 to 0.24.5 by @dependabot in apache#2579 * ci: upgrade hawkeye to v3 by @tisonkun in apache#2585 * ci(services/webdav): Setup integration test for nextcloud by @Xuanwo in apache#2631 * chore: add haskell binding link to website by @silver-ymz in apache#2571 * chore: fix cargo warning for resolver by @xxchan in apache#2590 * chore: bump log to 0.4.19 by @xxchan in apache#2591 * chore(deps): update deps to latest version by @suyanhanx in apache#2596 * chore: Add release 0.38.0 to download by @PsiACE in apache#2597 * chore(service/minitrace): automatically generate span name by @andylokandy in apache#2618 * @Prashanth-Chandra made their first contribution in apache#2573 * @andylokandy made their first contribution in apache#2618 **Full Changelog**: apache/opendal@v0.38.0...v0.38.1 Signed-off-by: ClSlaid <[email protected]>

@oowl

* Bump to version 0.38.1 * feat(binding/lua): add rename and create_dir operator function by @oowl in #2564 * feat(services/azblob): support sink by @suyanhanx in #2574 * feat(services/gcs): support sink by @suyanhanx in #2576 * feat(services/oss): support sink by @suyanhanx in #2577 * feat(services/obs): support sink by @suyanhanx in #2578 * feat(services/cos): impl sink by @suyanhanx in #2587 * feat(service): Support stat for Dropbox by @Zheaoli in #2588 * feat(services/dropbox): impl create_dir and polish error handling by @suyanhanx in #2600 * feat(services/dropbox): Implement refresh token support by @Xuanwo in #2604 * feat(service/dropbox): impl batch delete by @suyanhanx in #2606 * feat(CI): set Kvrocks test for service redis by @suyanhanx in #2613 * feat(core): object versioning APIs by @suyanhanx in #2614 * feat(oay): actually read configuration from `oay.toml` by @messense in #2615 * feat(services/webdav): impl sink by @suyanhanx in #2622 * feat(services/fs): impl Sink for Fs by @Ji-Xinyou in #2626 * feat(core): impl `delete_with` on blocking operator by @suyanhanx in #2633 * feat(bindings/C): add support for list in C binding by @Ji-Xinyou in #2448 * refactor(core): Add ErrorKind InvalidInput to indicate users input error by @dqhl76 in #2637 * fix(doc): fix codeblock rendering by @xxchan in #2592 * fix(service/minitrace): should set local parent by @andylokandy in #2620 * fix(service/minitrace): update doc by @andylokandy in #2621 * doc(bindings/haskell): add module document by @silver-ymz in #2566 * docs: Update license related comments by @Prashanth-Chandra in #2573 * docs: add hdfs namenode High Availability related troubleshoot by @wcy-fdu in #2601 * docs: polish release doc by @PsiACE in #2608 * docs(blog): add Apache OpenDAL(Incubating): Access Data Freely by @PsiACE in #2607 * docs(RFC): Object Versioning by @suyanhanx in #2602 * ci: Disable bindings/java deploy for now by @tisonkun in #2560 * ci: Disable the failed stage-release job instead by @tisonkun in #2561 * ci: add haddock generator for haskell binding by @silver-ymz in #2569 * ci(binding/lua): add luarocks package manager support by @oowl in #2558 * build(deps): bump predicates from 2.1.5 to 3.0.1 by @dependabot in #2583 * build(deps): bump tower-http from 0.4.0 to 0.4.1 by @dependabot in #2582 * build(deps): bump chrono from 0.4.24 to 0.4.26 by @dependabot in #2581 * build(deps): bump redis from 0.22.3 to 0.23.0 by @dependabot in #2580 * build(deps): bump cbindgen from 0.24.3 to 0.24.5 by @dependabot in #2579 * ci: upgrade hawkeye to v3 by @tisonkun in #2585 * ci(services/webdav): Setup integration test for nextcloud by @Xuanwo in #2631 * chore: add haskell binding link to website by @silver-ymz in #2571 * chore: fix cargo warning for resolver by @xxchan in #2590 * chore: bump log to 0.4.19 by @xxchan in #2591 * chore(deps): update deps to latest version by @suyanhanx in #2596 * chore: Add release 0.38.0 to download by @PsiACE in #2597 * chore(service/minitrace): automatically generate span name by @andylokandy in #2618 * @Prashanth-Chandra made their first contribution in #2573 * @andylokandy made their first contribution in #2618 **Full Changelog**: v0.38.0...v0.38.1 Signed-off-by: ClSlaid <[email protected]> * refactor: bump npm version Signed-off-by: ClSlaid <[email protected]> * chore: include #2634 Signed-off-by: 蔡略 <[email protected]> --------- Signed-off-by: ClSlaid <[email protected]> Signed-off-by: 蔡略 <[email protected]>

doc(rfc): object versioning

3aebdfa

Signed-off-by: suyanhanx <[email protected]>

suyanhanx requested review from Xuanwo, PsiACE, ClSlaid and sundy-li as code owners July 6, 2023 13:05

github-actions bot added the releases-note/docs label Jul 6, 2023

update

db76183

Signed-off-by: suyanhanx <[email protected]>

Xuanwo reviewed Jul 6, 2023

View reviewed changes

suyanhanx marked this pull request as draft July 6, 2023 13:24

suyanhanx added 2 commits July 7, 2023 18:19

polish

c7b4204

Signed-off-by: suyanhanx <[email protected]>

polish

a5ffc86

Signed-off-by: suyanhanx <[email protected]>

Xuanwo reviewed Jul 7, 2023

View reviewed changes

core/src/docs/rfcs/2602_object_versioning.md Show resolved Hide resolved

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

Xuanwo reviewed Jul 7, 2023

View reviewed changes

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

update

b5347d5

Signed-off-by: suyanhanx <[email protected]>

suyanhanx requested a review from Xuanwo July 9, 2023 08:59

Xuanwo reviewed Jul 9, 2023

View reviewed changes

core/src/docs/rfcs/mod.rs Outdated Show resolved Hide resolved

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

Xuanwo reviewed Jul 9, 2023

View reviewed changes

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

Xuanwo reviewed Jul 9, 2023

View reviewed changes

core/src/docs/rfcs/2602_object_versioning.md Outdated Show resolved Hide resolved

polish

4c3566f

Signed-off-by: suyanhanx <[email protected]>

suyanhanx mentioned this pull request Jul 9, 2023

Tracking issues of RFC: Object Versioning #2611

Open

8 tasks

suyanhanx added 2 commits July 9, 2023 19:49

create tracking issue

1ef56e7

Signed-off-by: suyanhanx <[email protected]>

polish

997d041

Signed-off-by: suyanhanx <[email protected]>

Xuanwo changed the title ~~doc(rfc): object versioning~~ docs(RFC): Object Versioning Jul 9, 2023

Xuanwo approved these changes Jul 9, 2023

View reviewed changes

Xuanwo marked this pull request as ready for review July 9, 2023 12:14

Xuanwo merged commit 97cf73f into main Jul 9, 2023
25 checks passed

Xuanwo deleted the object-version branch July 9, 2023 12:14

ClSlaid mentioned this pull request Jul 14, 2023

Bump to version 0.38.1 #2638

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(RFC): Object Versioning #2602

docs(RFC): Object Versioning #2602

suyanhanx commented Jul 6, 2023

Xuanwo commented Jul 7, 2023

Xuanwo left a comment •

edited

Loading

Xuanwo left a comment

suyanhanx commented Jul 9, 2023 •

edited

Loading

Xuanwo commented Jul 9, 2023

suyanhanx commented Jul 9, 2023

Xuanwo commented Jul 9, 2023

docs(RFC): Object Versioning #2602

docs(RFC): Object Versioning #2602

Conversation

suyanhanx commented Jul 6, 2023

Xuanwo commented Jul 7, 2023

Xuanwo left a comment • edited Loading

Choose a reason for hiding this comment

Xuanwo left a comment

Choose a reason for hiding this comment

suyanhanx commented Jul 9, 2023 • edited Loading

Xuanwo commented Jul 9, 2023

suyanhanx commented Jul 9, 2023

Xuanwo commented Jul 9, 2023

Xuanwo left a comment •

edited

Loading

suyanhanx commented Jul 9, 2023 •

edited

Loading