feat: add soft delete feature #2403

JesseLovelace · 2024-02-08T23:15:27Z

Implements the soft delete feature as requested in go/gcs-soft-delete-client-request

Also adds the get(String blob, Long generation, BlobGetOption...options) method signature to Bucket.java. It's already present in Storage.java, I'm adding it now because it's necessary to get a soft deleted object.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…into softdel

…tdel

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…tdel

frankyn

Have a couple of questions. Thanks Jesse

frankyn · 2024-02-12T17:24:21Z

google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpc.java

-        .setUserProject(Option.USER_PROJECT.getString(options));
+        .setUserProject(Option.USER_PROJECT.getString(options))
+        .setSoftDeleted(Option.SOFT_DELETED.getBoolean(options));
+    // .setGeneration(Option.GENERATION.getLong(options));


Might not be intentional

frankyn · 2024-02-12T17:38:57Z

google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpc.java

+          .setIfMetagenerationNotMatch(Option.IF_METAGENERATION_NOT_MATCH.getLong(options))
+          .setIfGenerationMatch(Option.IF_GENERATION_MATCH.getLong(options))
+          .setIfGenerationNotMatch(Option.IF_GENERATION_NOT_MATCH.getLong(options))
+          .setCopySourceAcl(Option.COPY_SOURCE_ACL.getBoolean(options))


This is an interesting addition to the API; surprised it needs to be added if you're restoring an object.

frankyn · 2024-02-12T17:45:36Z

google-cloud-storage/src/main/java/com/google/cloud/storage/Bucket.java

+   * @throws StorageException upon failure
+   */
+  @TransportCompatibility({Transport.HTTP, Transport.GRPC})
+  public Blob get(String blob, Long generation, BlobGetOption... options) {


IIUC this is needed because you can't specify generation on Bucket.get() that exists today; Would recommend using BlobId instead of adding Long generation overload.

Do you think this is hard requirement to support this feature since there's storage.get(BlobId..)?

I think it would feel pretty weird that soft deleted would become the only state an object can be in where you can get it from Storage but can't get it from the Bucket that the object lives in.

I also think using BlobId here would feel weird, because you're already calling from a Bucket, so it's weird to have to supply the same Bucket name into the BlobId. There are no other methods in Bucket.java that have a BlobId in the signature

Bleh, missed that bucket name is also part of BlobId. thanks.
We haven't had generation in Bucket.get() at all which is required for version enabled buckets, that's why i asked this question as well.

For the sake of discussion, WDYT about having generation under BlobGetOption's? It's a query paramter just like IfGenerationMatch etc.

frankyn · 2024-02-12T17:46:08Z

google-cloud-storage/src/main/java/com/google/cloud/storage/GrpcConversions.java

+    ifNonNull(from.getSoftDeletePolicy(), softDeletePolicyCodec::encode, to::setSoftDeletePolicy);
+    if (from.getModifiedFields().contains(SOFT_DELETE_POLICY)
+        && from.getSoftDeletePolicy() == null) {
+      System.out.println("this is happening");


frankyn · 2024-02-12T17:50:02Z

google-cloud-storage/src/test/java/com/google/cloud/storage/it/ITBlobReadMaskTest.java

@@ -201,7 +201,9 @@ public ImmutableList<?> parameters() {
              new Args<>(BlobField.UPDATED, LazyAssertion.equal()),
              new Args<>(
                  BlobField.RETENTION,
-                  LazyAssertion.skip("TODO: jesse fill in buganizer bug here")));
+                  LazyAssertion.skip("TODO: jesse fill in buganizer bug here")),


I understand this code already exists; just wondering why is Retention field mask test skipped?

This is related to object retention being JSON only. This will be removed once it's in the grpc api as well

frankyn · 2024-02-12T17:51:18Z

google-cloud-storage/src/main/java/com/google/cloud/storage/GrpcStorageImpl.java

+        RestoreObjectRequest.newBuilder()
+            .setBucket(bucketNameCodec.encode(blobId.getBucket()))
+            .setObject(blobId.getName());
+    ifNonNull(blobId.getGeneration(), builder::setGeneration);


Is the API failure helpful when generation isn't supplied?

Yeah, it says "you must specify a generation"

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…tdel

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

frankyn · 2024-02-29T18:50:34Z

google-cloud-storage/src/main/java/com/google/cloud/storage/GrpcConversions.java

-import static com.google.cloud.storage.Utils.projectNameCodec;
-import static com.google.cloud.storage.Utils.topicNameCodec;
+import static com.google.cloud.storage.Storage.BucketField.SOFT_DELETE_POLICY;
+import static com.google.cloud.storage.Utils.*;


Revert com.google.cloud.storage.Utils.*;

frankyn · 2024-02-29T18:53:04Z

google-cloud-storage/src/main/java/com/google/cloud/storage/HttpRetryAlgorithmManager.java

@@ -213,6 +213,11 @@ public ResultRetryAlgorithm<?> getForObjectsGet(
    return retryStrategy.getIdempotentHandler();
  }

+  public ResultRetryAlgorithm<?> getForObjectsRestore(
+      StorageObject pb, Map<StorageRpc.Option, ?> optionsMap) {
+    return retryStrategy.getIdempotentHandler();


Missed this on first pass; ObjectRestore is idempotent because it can only succeed once which is similar to Bucket; is that your understanding as well?

Yes, restore has the same idempotency as Copy and similar operations

frankyn · 2024-02-29T18:55:15Z

google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/HttpStorageRpcSpans.java

@@ -30,6 +30,8 @@ class HttpStorageRpcSpans {
  static final String SPAN_NAME_LIST_OBJECTS = getTraceSpanName("list(String,Map)");
  static final String SPAN_NAME_GET_BUCKET = getTraceSpanName("get(Bucket,Map)");
  static final String SPAN_NAME_GET_OBJECT = getTraceSpanName("get(StorageObject,Map)");
+


remove extra line

frankyn · 2024-02-29T18:55:27Z

google-cloud-storage/src/main/java/com/google/cloud/storage/spi/v1/StorageRpc.java

-    OVERRIDE_UNLOCKED_RETENTION("overrideUnlockedRetention");
+    OVERRIDE_UNLOCKED_RETENTION("overrideUnlockedRetention"),
+    SOFT_DELETED("softDeleted"),
+


remove extra line

…tdel

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…tdel

feat: add soft delete feature

26326a4

JesseLovelace requested a review from a team as a code owner February 8, 2024 23:15

product-auto-label bot added size: l Pull request size is large. api: storage Issues related to the googleapis/java-storage API. labels Feb 8, 2024

🦉 Updates from OwlBot post-processor

407af36

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

JesseLovelace requested a review from a team as a code owner February 8, 2024 23:18

gcf-owl-bot bot and others added 5 commits February 8, 2024 23:19

🦉 Updates from OwlBot post-processor

22ae88d

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'softdel' of https://github.com/googleapis/java-storage …

bd4bb67

…into softdel

add softDeleteTime and hardDeleteTime object fields

d77aa58

Merge branch 'softdel' of github.com:googleapis/java-storage into sof…

a888e80

…tdel

🦉 Updates from OwlBot post-processor

2a85a65

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

🦉 Updates from OwlBot post-processor

4f1b6de

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

JesseLovelace requested a review from frankyn February 9, 2024 18:46

JesseLovelace added 3 commits February 9, 2024 11:41

add new fields to field tests

fd22008

Merge branch 'softdel' of github.com:googleapis/java-storage into sof…

7142bcb

…tdel

clirr ignore

0f837d3

frankyn reviewed Feb 12, 2024

View reviewed changes

JesseLovelace and others added 2 commits February 13, 2024 16:21

remove debug comments

c2884aa

🦉 Updates from OwlBot post-processor

ac6022e

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

frankyn approved these changes Feb 14, 2024

View reviewed changes

JesseLovelace and others added 3 commits February 27, 2024 16:01

Merge branch 'main' into softdel

1c363ff

🦉 Updates from OwlBot post-processor

dbe0d92

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

add softdeletetime and harddeletetime to grpc codec

fb88608

gcf-owl-bot bot and others added 4 commits February 28, 2024 00:15

🦉 Updates from OwlBot post-processor

04975fc

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

fix read mask test

1de29b8

Merge branch 'softdel' of github.com:googleapis/java-storage into sof…

c98d8c7

…tdel

🦉 Updates from OwlBot post-processor

3a4a00f

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

frankyn requested changes Feb 29, 2024

View reviewed changes

JesseLovelace and others added 5 commits February 29, 2024 11:06

fix style issues

b3d70db

Merge branch 'softdel' of github.com:googleapis/java-storage into sof…

7986f9c

…tdel

Merge branch 'main' into softdel

a93e162

updates to apiary library

5a744ca

🦉 Updates from OwlBot post-processor

0b8fe3f

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

frankyn approved these changes Mar 12, 2024

View reviewed changes

JesseLovelace added do not merge Indicates a pull request not ready for merge, due to either quality or timing. and removed do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Mar 12, 2024

JesseLovelace and others added 5 commits March 14, 2024 13:26

Merge branch 'main' into softdel

be3e975

🦉 Updates from OwlBot post-processor

5e65c40

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

fix read mask test

938e00c

Merge branch 'softdel' of github.com:googleapis/java-storage into sof…

8f38221

…tdel

fix typo

68ce8c0

JesseLovelace merged commit 989f36f into main Mar 14, 2024
22 of 23 checks passed

JesseLovelace deleted the softdel branch March 14, 2024 22:40

release-please bot mentioned this pull request Mar 14, 2024

chore(main): release 2.36.0 #2443

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add soft delete feature #2403

feat: add soft delete feature #2403

JesseLovelace commented Feb 8, 2024

frankyn left a comment

frankyn Feb 12, 2024

frankyn Feb 12, 2024

frankyn Feb 12, 2024

JesseLovelace Feb 12, 2024

frankyn Feb 12, 2024

frankyn Feb 12, 2024

frankyn Feb 12, 2024

JesseLovelace Feb 12, 2024

frankyn Feb 12, 2024

JesseLovelace Feb 12, 2024

frankyn Feb 29, 2024

frankyn Feb 29, 2024

JesseLovelace Feb 29, 2024

frankyn Feb 29, 2024

frankyn Feb 29, 2024

feat: add soft delete feature #2403

feat: add soft delete feature #2403

Conversation

JesseLovelace commented Feb 8, 2024

frankyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment