Storage v4 POST Policy #4866

quartzmo · 2020-02-25T16:22:36Z

Acceptance tests are passing locally with my development project.

quartzmo · 2020-02-25T16:49:54Z

@frankyn Currently failing on the signature (test run currently limited to just the first conformance test.)

Do you see anything missing in the signature calculation?

  1) Failure:
SignerV4PostObjectTest#test_bucket_0: POST Policy Simple [/Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/test/google/cloud/storage/file/signer_v4_post_object_test.rb:87]:
--- expected
+++ actual
@@ -1 +1,2 @@
-"47ed5e45ea9bf72b889c0ba80b4424a0a8765d3c93558f4766e29a188c4ee70b7e5a287dc478e58087ae89e829324b0d3306d33bdd589e7092eb1b12a171023c37f28f2379f0d49eb9a9db62e3faa996ebff605a84318038fc4b2134532e50617709362b92f02cc9623acbc480dd964ee091121f8a5a5b074a625231ab507c6d223c09d727f56c5036ff56b719696a9958926274e7612e523716918aee351379feaffb92a6c111cee5d63d7d770fdb4a4e5d0f9e28971463cbf602015250e4b48e6822bc0cb550634da4b7d41e4d6caa817c7ed7bd4e30d87da81985cfc7eedee729b3221ff3b85f33efb92dfd24e7d5f6c135c1d93454a6364a1a4b8bf4b047"
+# encoding: US-ASCII
+"KCGiPKiIaunp1n6MCUwGYmpzPzc83KdtnYwyXmDtgje1ALLzmTehWh673vftFW/j9BvDRW1FO7aNZ1WHLrQpwymuTg37Xgq+9MqXY5RDCtez0fFOUPYEDCz4GiiAmAwm+a64AKZ0/INLDiHkFK3D0aCDXaZBB8zmoUvSs5tNZicCn4IQCeFf86dTjdwn3PjBie0ZfSXVDwvd9Y+Z4CZa31Yv/39mifq6YJINc+amcEw9K+NKQkdmOYOWxV02gPjf0ZUT4oYfpO+nrpkAGHfwQ1ZxMtQT3Dxp4Ep+9dMKMlwvm8ZOGd1vBXZ4HlRjgzeOek8pBEEe4y1cWnY2gzM91w=="

frankyn

Thanks @quartzmo! Mostly have nits.

google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb

google-cloud-storage/lib/google/cloud/storage/bucket.rb

google-cloud-storage/conformance/v1/v4_signatures.json

google-cloud-storage/lib/google/cloud/storage/bucket.rb

google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb

closes googleapis#4739

…post_object

google-cloud-storage/lib/google/cloud/storage/bucket.rb

frankyn

This looks much better from a usability stand-point.

One nit on documentation, and the PR is still missing integration tests which will be unblocked ~EOW.

Thanks for your patience @quartzmo

google-cloud-storage/lib/google/cloud/storage/bucket.rb

frankyn

Thanks @quartzmo, LGTM, now it's pending integration tests and surface checkoff.

LGTM by comment until that's complete.

frankyn

Two nits, I'm waiting on confirmation for consistent naming across languages.

google-cloud-storage/acceptance/storage/bucket_post_object_v4_test.rb

frankyn

Got confirmation on method name for v4.

frankyn · 2020-03-24T00:41:55Z

google-cloud-storage/lib/google/cloud/storage/bucket.rb

+        #   post.fields["x-goog-date"] #=> "20200128T000000Z"
+        #   post.fields["x-goog-signature"] #=> "4893a0e...cd82"
+        #
+        def post_object_v4 path,


Using consistent naming across languages, please use: generate_signed_post_policy_v4.

Will do this after I get a passing conformance test for character escaping.

quartzmo · 2020-03-24T17:50:07Z

@frankyn Here is the current test failure and debug output. The actual policy_str value and expected expectedDecodedPolicy value appear to me to have no visible differences. But the test fails on the comparison of actual vs expected encoded policy.

Run options: "-n=/^(SignerV4PostObjectTest\\#test_bucket_9:\\ POST\\ Policy\\ Character\\ Escaping)$/" --seed 44

# Running:



policy_str:

{"conditions":[{"x-goog-meta":"$test-object-é-metadata"},{"success_action_redirect":"http://www.google.com/"},{"key":"$test-object-é"},{"x-goog-date":"20200123T043530Z"},{"x-goog-credential":"[email protected]/20200123/auto/storage/goog4_request"},{"x-goog-algorithm":"GOOG4-RSA-SHA256"}],"expiration":"2020-01-23T04:35:40Z"}

expectedDecodedPolicy:

{"conditions":[{"x-goog-meta":"$test-object-é-metadata"},{"success_action_redirect":"http://www.google.com/"},{"key":"$test-object-é"},{"x-goog-date":"20200123T043530Z"},{"x-goog-credential":"[email protected]/20200123/auto/storage/goog4_request"},{"x-goog-algorithm":"GOOG4-RSA-SHA256"}],"expiration":"2020-01-23T04:35:40Z"}

F

Finished in 0.041164s, 24.2931 runs/s, 145.7584 assertions/s.

  1) Failure:
SignerV4PostObjectTest#test_bucket_9: POST Policy Character Escaping [/Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/test/google/cloud/storage/file/signer_v4_post_object_test.rb:86]:
--- expected
+++ actual
@@ -1 +1 @@
-"eyJjb25kaXRpb25zIjpbeyJ4LWdvb2ctbWV0YSI6IiR0ZXN0LW9iamVjdC1cdTAwZTktbWV0YWRhdGEifSx7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjoiaHR0cDovL3d3dy5nb29nbGUuY29tLyJ9LHsia2V5IjoiJHRlc3Qtb2JqZWN0LVx1MDBlOSJ9LHsieC1nb29nLWRhdGUiOiIyMDIwMDEyM1QwNDM1MzBaIn0seyJ4LWdvb2ctY3JlZGVudGlhbCI6InRlc3QtaWFtLWNyZWRlbnRpYWxzQGR1bW15LXByb2plY3QtaWQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20vMjAyMDAxMjMvYXV0by9zdG9yYWdlL2dvb2c0X3JlcXVlc3QifSx7IngtZ29vZy1hbGdvcml0aG0iOiJHT09HNC1SU0EtU0hBMjU2In1dLCJleHBpcmF0aW9uIjoiMjAyMC0wMS0yM1QwNDozNTo0MFoifQ=="
+"eyJjb25kaXRpb25zIjpbeyJ4LWdvb2ctbWV0YSI6IiR0ZXN0LW9iamVjdC3DqS1tZXRhZGF0YSJ9LHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiJodHRwOi8vd3d3Lmdvb2dsZS5jb20vIn0seyJrZXkiOiIkdGVzdC1vYmplY3Qtw6kifSx7IngtZ29vZy1kYXRlIjoiMjAyMDAxMjNUMDQzNTMwWiJ9LHsieC1nb29nLWNyZWRlbnRpYWwiOiJ0ZXN0LWlhbS1jcmVkZW50aWFsc0BkdW1teS1wcm9qZWN0LWlkLmlhbS5nc2VydmljZWFjY291bnQuY29tLzIwMjAwMTIzL2F1dG8vc3RvcmFnZS9nb29nNF9yZXF1ZXN0In0seyJ4LWdvb2ctYWxnb3JpdGhtIjoiR09PRzQtUlNBLVNIQTI1NiJ9XSwiZXhwaXJhdGlvbiI6IjIwMjAtMDEtMjNUMDQ6MzU6NDBaIn0="

frankyn · 2020-03-24T17:56:14Z

Thanks @quartzmo, one issue I see is that expectedDecodedPolicy is not consistent with the PR: googleapis/conformance-tests#27.

The expectedDecodedPolicy and encodedPolicy are the following:

"policy": "eyJjb25kaXRpb25zIjpbeyJ4LWdvb2ctbWV0YSI6IiR0ZXN0LW9iamVjdC1cdTAwZTktbWV0YWRhdGEifSx7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjoiaHR0cDovL3d3dy5nb29nbGUuY29tLyJ9LHsia2V5IjoiJHRlc3Qtb2JqZWN0LVx1MDBlOSJ9LHsieC1nb29nLWRhdGUiOiIyMDIwMDEyM1QwNDM1MzBaIn0seyJ4LWdvb2ctY3JlZGVudGlhbCI6InRlc3QtaWFtLWNyZWRlbnRpYWxzQGR1bW15LXByb2plY3QtaWQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20vMjAyMDAxMjMvYXV0by9zdG9yYWdlL2dvb2c0X3JlcXVlc3QifSx7IngtZ29vZy1hbGdvcml0aG0iOiJHT09HNC1SU0EtU0hBMjU2In1dLCJleHBpcmF0aW9uIjoiMjAyMC0wMS0yM1QwNDozNTo0MFoifQ=="
"expectedDecodedPolicy": "{\"conditions\":[{\"x-goog-meta\":\"$test-object-\u00e9-metadata\"},{\"success_action_redirect\":\"http://www.google.com/\"},{\"key\":\"$test-object-\u00e9\"},{\"x-goog-date\":\"20200123T043530Z\"},{\"x-goog-credential\":\"[email protected]/20200123/auto/storage/goog4_request\"},{\"x-goog-algorithm\":\"GOOG4-RSA-SHA256\"}],\"expiration\":\"2020-01-23T04:35:40Z\"}"

quartzmo · 2020-03-31T22:00:27Z

Output from bundle exec rake test using commit 1b12e3f:


# Running:



policy_str:

{"conditions":[{"x-goog-meta":"$test-object-é-metadata"},{"success_action_redirect":"http://www.google.com/"},{"key":"$test-object-é"},{"x-goog-date":"20200123T043530Z"},{"x-goog-credential":"[email protected]/20200123/auto/storage/goog4_request"},{"x-goog-algorithm":"GOOG4-RSA-SHA256"}],"expiration":"2020-01-23T04:35:40Z"}


policy_str_fake:

{\"conditions\":[{\"success_action_redirect\":\"http://www.google.com/\"},{\"x-goog-meta\":\"$test-object-\u00e9-metadata\"},{\"key\":\"$test-object-\u00e9\"},{\"x-goog-date\":\"20200123T043530Z\"},{\"x-goog-credential\":\"[email protected]/20200123/auto/storage/goog4_request\"},{\"x-goog-algorithm\":\"GOOG4-RSA-SHA256\"}],\"expiration\":\"2020-01-23T04:35:40Z\"}


test_bucket_9: POST Policy Character Escaping:
policyOutput

expectedDecodedPolicy:

{"conditions":[{"success_action_redirect":"http://www.google.com/"},{"x-goog-meta":"$test-object-é-metadata"},{"key":"$test-object-é"},{"x-goog-date":"20200123T043530Z"},{"x-goog-credential":"[email protected]/20200123/auto/storage/goog4_request"},{"x-goog-algorithm":"GOOG4-RSA-SHA256"}],"expiration":"2020-01-23T04:35:40Z"}

F

Finished in 0.117436s, 8.5153 runs/s, 51.0917 assertions/s.

  1) Failure:
SignerV4PostObjectTest#test_bucket_9: POST Policy Character Escaping [/Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/test/google/cloud/storage/file/signer_v4_post_object_test.rb:86]:
--- expected
+++ actual
@@ -1 +1 @@
-"eyJjb25kaXRpb25zIjpbeyJzdWNjZXNzX2FjdGlvbl9yZWRpcmVjdCI6Imh0dHA6Ly93d3cuZ29vZ2xlLmNvbS8ifSx7IngtZ29vZy1tZXRhIjoiJHRlc3Qtb2JqZWN0LVx1MDBlOS1tZXRhZGF0YSJ9LHsia2V5IjoiJHRlc3Qtb2JqZWN0LVx1MDBlOSJ9LHsieC1nb29nLWRhdGUiOiIyMDIwMDEyM1QwNDM1MzBaIn0seyJ4LWdvb2ctY3JlZGVudGlhbCI6InRlc3QtaWFtLWNyZWRlbnRpYWxzQGR1bW15LXByb2plY3QtaWQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20vMjAyMDAxMjMvYXV0by9zdG9yYWdlL2dvb2c0X3JlcXVlc3QifSx7IngtZ29vZy1hbGdvcml0aG0iOiJHT09HNC1SU0EtU0hBMjU2In1dLCJleHBpcmF0aW9uIjoiMjAyMC0wMS0yM1QwNDozNTo0MFoifQ=="
+"e1wiY29uZGl0aW9uc1wiOlt7XCJzdWNjZXNzX2FjdGlvbl9yZWRpcmVjdFwiOlwiaHR0cDovL3d3dy5nb29nbGUuY29tL1wifSx7XCJ4LWdvb2ctbWV0YVwiOlwiJHRlc3Qtb2JqZWN0LVx1MDBlOS1tZXRhZGF0YVwifSx7XCJrZXlcIjpcIiR0ZXN0LW9iamVjdC1cdTAwZTlcIn0se1wieC1nb29nLWRhdGVcIjpcIjIwMjAwMTIzVDA0MzUzMFpcIn0se1wieC1nb29nLWNyZWRlbnRpYWxcIjpcInRlc3QtaWFtLWNyZWRlbnRpYWxzQGR1bW15LXByb2plY3QtaWQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20vMjAyMDAxMjMvYXV0by9zdG9yYWdlL2dvb2c0X3JlcXVlc3RcIn0se1wieC1nb29nLWFsZ29yaXRobVwiOlwiR09PRzQtUlNBLVNIQTI1NlwifV0sXCJleHBpcmF0aW9uXCI6XCIyMDIwLTAxLTIzVDA0OjM1OjQwWlwifQ=="


1 runs, 6 assertions, 1 failures, 0 errors, 0 skips
rake aborted!

google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb

google-cloud-storage/test/google/cloud/storage/file/signer_v4/escape_characters_test.rb

frankyn

LGTM @quartzmo, thank you!

Confirming escape table, LGTM.

dazuma

🎉

google-cloud-storage/lib/google/cloud/storage/bucket.rb

quartzmo · 2020-04-06T22:49:56Z

rubygems: https://rubygems.org/gems/google-cloud-storage/versions/1.26.0

docs: https://googleapis.dev/ruby/google-cloud-storage/latest/Google/Cloud/Storage/Bucket.html#generate_signed_post_policy_v4-instance_method

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Feb 25, 2020

quartzmo requested a review from frankyn February 25, 2020 16:22

quartzmo added api: storage Issues related to the Cloud Storage API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Feb 25, 2020

frankyn requested changes Feb 25, 2020

View reviewed changes

quartzmo force-pushed the storage-v4-post branch 2 times, most recently from a5fe591 to ed98c7e Compare February 25, 2020 23:03

frankyn reviewed Feb 26, 2020

View reviewed changes

google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb Show resolved Hide resolved

quartzmo added 6 commits March 5, 2020 16:25

Remove unused conformance test data from test dir

4c32ea2

Add SignerV4#post_object and update conformance data

55f2459

closes googleapis#4739

Update conformance data, add expires, fields, conditions

65d2d21

Update conformance tests for fix gcs -> storage (googleapis#24)

5d7afdf

Fix rubocop

7645496

Update conformance tests from googleapis#25

8de976a

quartzmo force-pushed the storage-v4-post branch from ff416e9 to 88ba5c8 Compare March 6, 2020 01:30

Add scheme, virtual_hosted_style and bucket_bound_hostname to Bucket#…

c891e4e

…post_object

quartzmo force-pushed the storage-v4-post branch from 88ba5c8 to c891e4e Compare March 6, 2020 19:09

Add Bucket#post_object docs and refactor SignerV4

5223c9a

frankyn reviewed Mar 7, 2020

View reviewed changes

google-cloud-storage/lib/google/cloud/storage/bucket.rb Show resolved Hide resolved

Add Bucket#post_object_v4

9508f8b

quartzmo removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 7, 2020

quartzmo self-assigned this Mar 9, 2020

frankyn requested changes Mar 9, 2020

View reviewed changes

google-cloud-storage/lib/google/cloud/storage/bucket.rb Outdated Show resolved Hide resolved

Update PostObject V4 examples

aa02e4d

frankyn reviewed Mar 10, 2020

View reviewed changes

Add POST Policy V4 acceptance tests

610390c

quartzmo force-pushed the storage-v4-post branch from 91b9cc8 to 610390c Compare March 17, 2020 23:56

frankyn requested changes Mar 20, 2020

View reviewed changes

google-cloud-storage/acceptance/storage/bucket_post_object_v4_test.rb Outdated Show resolved Hide resolved

google-cloud-storage/acceptance/storage/bucket_post_object_v4_test.rb Outdated Show resolved Hide resolved

Update acceptance test assertions

7ce5904

quartzmo added 2 commits March 23, 2020 14:53

Add trailing slash to #post_object URLs for latest conformance tests

4b16f5c

Update conformance tests for escaping special characters

3b917fb

frankyn requested changes Mar 24, 2020

View reviewed changes

Use latest conformance test from PR 27, currently still failing

50bcf66

add focus and debug output

6c55a61

frankyn mentioned this pull request Mar 31, 2020

[Storage] Add Post Policy V4 Sample GoogleCloudPlatform/ruby-docs-samples#539

Merged

4 tasks

quartzmo added 2 commits March 31, 2020 15:33

Use latest conformance test from master, currently still failing

fbea916

Use expectedDecodedPolicy from test inline in impl. Test still failing

1b12e3f

quartzmo added 3 commits April 1, 2020 17:38

Add working #escape_characters method

5babdb0

Finish SignerV4#escape_characters

5822314

Rename Bucket#post_object_v4 to #generate_signed_post_policy_v4

fe3ef60

quartzmo commented Apr 2, 2020

View reviewed changes

google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb Show resolved Hide resolved

quartzmo commented Apr 2, 2020

View reviewed changes

google-cloud-storage/test/google/cloud/storage/file/signer_v4/escape_characters_test.rb Show resolved Hide resolved

frankyn approved these changes Apr 2, 2020

View reviewed changes

quartzmo requested a review from dazuma April 3, 2020 00:01

dazuma approved these changes Apr 3, 2020

View reviewed changes

google-cloud-storage/lib/google/cloud/storage/bucket.rb Outdated Show resolved Hide resolved

Fix errors in docs

41f5772

quartzmo force-pushed the storage-v4-post branch from a024990 to 41f5772 Compare April 3, 2020 22:37

Update conformance test data from conformance-tests#28

711b627

quartzmo merged commit 626b72f into googleapis:master Apr 4, 2020

quartzmo deleted the storage-v4-post branch April 4, 2020 02:12

quartzmo mentioned this pull request Apr 4, 2020

Unlock minitest version of google-cloud-storage #5207

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage v4 POST Policy #4866

Storage v4 POST Policy #4866

quartzmo commented Feb 25, 2020 •

edited

Loading

quartzmo commented Feb 25, 2020 •

edited

Loading

frankyn left a comment

frankyn left a comment

frankyn left a comment

frankyn left a comment

frankyn left a comment

frankyn Mar 24, 2020

quartzmo Mar 24, 2020

quartzmo commented Mar 24, 2020

frankyn commented Mar 24, 2020

quartzmo commented Mar 31, 2020

frankyn left a comment

dazuma left a comment

quartzmo commented Apr 6, 2020

Storage v4 POST Policy #4866

Storage v4 POST Policy #4866

Conversation

quartzmo commented Feb 25, 2020 • edited Loading

quartzmo commented Feb 25, 2020 • edited Loading

frankyn left a comment

Choose a reason for hiding this comment

frankyn left a comment

Choose a reason for hiding this comment

frankyn left a comment

Choose a reason for hiding this comment

frankyn left a comment

Choose a reason for hiding this comment

frankyn left a comment

Choose a reason for hiding this comment

frankyn Mar 24, 2020

Choose a reason for hiding this comment

quartzmo Mar 24, 2020

Choose a reason for hiding this comment

quartzmo commented Mar 24, 2020

frankyn commented Mar 24, 2020

quartzmo commented Mar 31, 2020

frankyn left a comment

Choose a reason for hiding this comment

dazuma left a comment

Choose a reason for hiding this comment

quartzmo commented Apr 6, 2020

quartzmo commented Feb 25, 2020 •

edited

Loading

quartzmo commented Feb 25, 2020 •

edited

Loading