Possible Gcloud Java bug in signurl?

[OverclockedTim]

So I'm using gcloud storage in a managed VM app and generally loving it, but I've hit something that I really think it is a bug so I wanted to throw it out there and see what people think.

I'm doing a few things with Storage and most of them are working perfectly Reads work perfectly, writes work perfectly, creates work perfectly. However, when I attempt to sign a URL like so:

Storage storage = StorageOptions.defaultInstance().service();

URL url = storage.signUrl(
BlobInfo.builder(BUCKET_NAME,
OBJECT_NAME).build(),
30,
TimeUnit.MINUTES,
Storage.SignUrlOption.httpMethod(HttpMethod.GET));

I end up getting a "Signing key not provided" exception:

java.lang.IllegalArgumentException: Signing key was not provided and could not be derived
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at com.google.gcloud.storage.StorageImpl.signUrl(StorageImpl.java:531)
...

This occurs on a deployed Managed VM server. It also occurs locally when gcloud is authorized either to myself or to a service account. In other words, it appears that storage.signUrl is not picking up signing authorization from the standard locations. I have not yet tried to manually add a signing option from a generated key, but that seems like it should not required. Additionally, the signUrl example from the gcloud-java source in maven has no additional signingKey in the SignUrlOptions which leads me to believe that it is intended for signUrl to pick up its authorization from the environment.

[mziccard]

Hi @OverclockedTim thanks for the report and the kind words:)

What signUrl does is looking in the credentials you provided to StorageOptions for a private key. If you are authenticated with default credentials you provided no private key (because you don't need to) hence the complaint Signing key not provided.

To use signUrl when authenticated with default credentials you can pass a private key using the SignUrlOption.serviceAccount option. Your code would become:

// Create ServiceAccountAuthCredentials from a stream
ServiceAccountAuthCredentials credentials =
    AuthCredentials.createForJson(new FileInputStream("/path/to/key.json"));
// or from a String (account) and PrivateKey
ServiceAccountAuthCredentials credentials =
    AuthCredentials.createFor(ACCOUNT, PRIVATE_KEY);
// Create the signed URL
URL url = storage.signUrl(BlobInfo.builder(BUCKET_NAME, OBJECT_NAME).build(), 30,
    TimeUnit.MINUTES, Storage.SignUrlOption.httpMethod(HttpMethod.GET),
    Storage.SignUrlOption.serviceAccount(credentials));

Let us know if this helps!

PS: to understand how authentication (and default credentials) work you can have a look at the authentication section in our README.

[OverclockedTim]

Hi @mziccard - Thank you! The method you mentioned does in fact work, but that does leave me with some questions: If a private key is always required for this function to succeed, why is it not required in the function signature? (it is optional)

Also, if a private key is always required, is the example in the comment for signURL (https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java#L1478-L1498) simply always guaranteed to throw the Signing key not provided exception?

Finally, does this mean that any application that uses signUrl must keep a copy of the JSON key around? This is something that I was hoping to avoid due to the difficulty of keeping that key secure. I would prefer to let Google's internal security handle it if that were possible.

[mziccard]

Thank you! The method you mentioned does in fact work, but that does leave me with some questions: If a private key is always required for this function to succeed, why is it not required in the function signature? (it is optional)

If you set the GOOGLE_DEFAULT_CREDENTIALS variable to point to a JSON key then default credentials you are using detect and use that JSON key to authenticate. In that case providing a key to signUrl is not required, it will detect the one used to authenticate requests and use it. Alternatively you could have created your StorageOptions by providing some ServiceAccountAuthCredentials:

StorageOptions options = StorageOptions.builder()
    .authCredentials(AuthCredentials.createForJson(new FileInputStream("/path/to/key.json")))
    .build();

Also in this case passing a key to signUrl is not required, as the one already provided to StorageOptions is used.

Finally, does this mean that any application that uses signUrl must keep a copy of the JSON key around?

Signing a URL involves signing with a private key. If the credentials used to authenticate do not expose one (this is the case for App Engine credentials, Compute Engine credentials and Google Cloud SDK credentials) there is nothing we can do to help.

[aozarov]

I think we should make that clear in the Storage.signUrl javadoc (credentials for signing is needed and will try to get it from the environment unless explicitly provided). In both method description and @throws tag.

Also, FYI #701 plans to make the signing credentials options a bit more flexible and we should be able to support it on AE.

[mziccard]

I think we should make that clear in the Storage.signUrl javadoc

I agree, this is tricky indeed. I'm assigning this to myself and I'll take care of adding a better javadoc. I'll post back here the PR if @OverclockedTim wants to give us some feedback on the readability of the javadoc.

[OverclockedTim]

@mziccard - Certainly, if that is helpful, I would be happy to!

[OverclockedTim]

@aozarov Just as a heads up, the thing that I did not understand after reading the javadocs was that the key required for signUrl was different than the credentials required for things like Upload or Read.

[mziccard]

Closed with #743

[lov001]

Hey @mziccard ,
I'm also stuck with a similar kind of issue, wherein I'm able to perform upload, download, list and delete a file from GCS but when obtaining signedURL I'm getting the above error message. The point I'm still confused is that using the same below code I'm able to perform all other operations but not table to get signedURL. I'm already authenticated with default credentials and the GOOGLE_DEFAULT_CREDENTIALS env variable is also set.
`

        Storage storage = StorageOptions.getDefaultInstance().getService();
        BlobId blobId = BlobId.of(bucketName, "objectName, Long.getLong(generationID));
        BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
       return storage.signUrl(blobInfo, 5, TimeUnit.MINUTES, Storage.SignUrlOption.httpMethod(HttpMethod.GET)
            Storage.SignUrlOption.withV4Signature());`

Would be great if you can point out what I'm doing wrong here.
Thanks