Cloud storage rewrite with GCloud?

[hohaichi]

Does GCloud support Google Cloud Storage rewrite?

[daspecster]

I don't believe we do at this time.

[hohaichi]

Do you have any plan to add rewrite? It is hard for the user to rotate their encryption key without rewrite.

[daspecster]

That's a great use case!

@tseaver & @dhermes WDYT? I can start this today or tomorrow if there are not blockers.

[theacodes]

@tseaver @dhermes @jgeewax this is also a blocking issue for us to be able to re-write storage usage samples using this library.

[tseaver]

Based on the rewrite API docs, it seems like this should be a method on Bucket, parallel to Bucket.copy_blob. Something like:

    def rewrite_blob(self, blob, destination_bucket, new_name, client=None):
        """Rewrite the given blob to the given bucket, with a new name.

        :type blob: :class:`gcloud.storage.blob.Blob`
        :param blob: The blob to be copied.

        :type destination_bucket: :class:`gcloud.storage.bucket.Bucket`
        :param destination_bucket: The bucket into which the blob should be
                                   copied.

        :type new_name: string
        :param new_name: (optional) the new name for the copied file.

        :type client: :class:`gcloud.storage.client.Client` or ``NoneType``
        :param client: Optional. The client to use.  If not passed, falls back
                       to the ``client`` stored on the current bucket.

        :rtype: bytes
        :returns: The new Blob.
        """
        client = self._require_client(client)
        if new_name is None:
            new_name = blob.name
        new_blob = Blob(bucket=destination_bucket, name=new_name)
        api_path = blob.path + '/rewriteTo' + new_blob.path
        resource = dict(blob._properties)
        result = client.connection.api_request(
            method='POST', path=api_path, data=resource,
            _target_object=new_blob)
        token = result.get('rewriteToken')
        while token not in (None, ''):
            query_params = {'rewriteToken': token}
            result = client.connection.api_request(
                method='POST', path=api_path, query_params=query_params,
                _target_object=new_blob)
            token = result.get('rewriteToken')
        new_blob._set_properties(result['resource'])
        return new_blob

[theacodes]

That would be fine, alternatively blob.rewrite_to(destination_name, source_encryption_key=..., encryption_key=...) would be acceptable as well.

N.B. this method must take two encryption key parameters, one for the source and one for the destination. This a the primary use case of the rewrite method, other than just moving objects between buckets.

[tseaver]

@jonparrott Where in the rewrite request are the keys passed? I see fields in the Object resource which are metadata about the encryption, but not the key itself.

[theacodes]

In the headers, hilariously enough.

See the current sample here.

[tseaver]

@jonparrott Why are those values not part of the payload? Worse, why is that not documented?

[theacodes]

@tseaver I have no idea - this API predates me joining and I'm not part of the API review team. My hunch is that it was a way to get around releasing a new API version. Maybe @Capstan can provide some background.

[tseaver]

/me grumbles This is exactly the kind of thing that is supposed to make you release a new API version.

[theacodes]

@tseaver don't shoot the messenger. I thought it was whack when I had to write the samples for it using google-api-python-client, because it's awkward to add headers using that library.

[tseaver]

@jonparrott I'm grumbling about that (hypothetical) motive, which predates you. New API release or not, the API docs should still be updated to define the (added?) behaviors: how did you learn about those headers in the first place?

[theacodes]

https://cloud.google.com/storage/docs/encryption

[Capstan]

@tseaver I've filed an internal bug about Rewrite docs not saying enough about how to use it with customer-supplied encryption keys. I'm not entirely sure why it couldn't have been added to the post body, but I do know that ensuring that no key is ever persisted anywhere is of paramount importance and may have guided that decision.

[theacodes]

Bump, any update on this?

[daspecster]

It looks like it was last updated August 19th,
And now it talks about the headers. I'm not sure if that was there before or if that's what we were looking for?

https://cloud.google.com/storage/docs/encryption#request

[hohaichi]

Re: object resource having encryption key metadata but not key materials: It is a little bit confusing in the first glance, as object resources can appear in both requests and responses. But if you look more closely, the table denotes which fields are "writable". Only those fields can be updated. Key metadata cannot be updated, they are only helpful in responses.

Re: why the keys are passed in as headers rather than the post body: Actually adding the keys to the request body is not as straightforward as it seems. In our current API, a request body either contains object data or object metadata. And key materials are neither.

[theacodes]

@tseaver @dhermes can I get a release of the storage client so I can update the sample?

[tseaver]

@dhermes I'm not quite sure how you've made sub-project releases after the umbrella-0.20.0 tag. I just ran $ git log umbrella-0.20.0..HEAD -- storage/ and got the following:

• Making max_results part of the base Iterator class. (Fixing bucket.list_blobs(max_results=n) has no effect #1467)
• Add 'Blob.compose' API wrapper method. (Closing Cloud storage is missing 'compose' functionality #1982)
• Add 'Blob.rewrite' API wrapper method. (Closing Cloud storage rewrite with GCloud? #1960)
• Add new storage classes. (PR Update Google Cloud Storage bucket classes. #2571)
• Your iterator work.

[dhermes]

@tseaver I think it's more robust just to use the GitHub UI (since people can merge PRs however they like now, and we haven't disabled the other two options, though we can):

https://github.com/GoogleCloudPlatform/google-cloud-python/commits/master/storage?since=2016-09-27

[tseaver]

@dhermes FWIW, I wasn't grepping for "Merged" commits: I looked at the whole history for /storage since that tag.

[dhermes]

Haha DERP. I should've read closely.

[theacodes]

Bump - I can has release?

[dhermes]

@jonparrott It's not as easy as it seems. It requires a core release as well, which has implications on all packages.

[theacodes]

The core change being the iterator work? If so, I can wait.

[dhermes]

The core change being the iterator work?

Yes