Tasks: Update queue, "x-goog-request-params" header

[alexcwatt]

Environment details

Running google-cloud-tasks==1.2.1.

Steps to reproduce

1. Create a Cloud Tasks queue.
2. Attempt to update the queue using the update_queue method with a dictionary to specify the queue name.
3. Observe exception: google.api_core.exceptions.InvalidArgument: 400 "x-goog-request-params" header is either missing or misformatted. "x-goog-request-params" must contain "queue.name=projects/[projectname]/locations/[location]/queues/[name]"
4. Replace the queue object in update_queue with an instance of google.cloud.tasks_v2beta3.types.Queue and observe success.

Code example

import google
from google.cloud import tasks_v2beta3

TASKS_CLIENT = tasks_v2beta3.CloudTasksClient()
TASKS_QUEUE_NAME = 'queuenamehere'
TASKS_CLIENT.update_queue({'name': TASKS_QUEUE_NAME})

vs.

queue = google.cloud.tasks_v2beta3.types.Queue(name=TASKS_QUEUE_NAME)
TASKS_CLIENT.update_queue(queue)

Note: After I resolved this issue, I was using this to update the retry policy, but this is the simplest example to demonstrate the failure -- namely that update_queue doesn't actually work if a dictionary is provided for the queue, even though the type annotation says that the Union[dict, Queue] is supported.

Stack trace

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = ""x-goog-request-params" header is either missing or misformatted. "x-goog-request-params" must contain "queue.name=projects/gecko-ops-sandbox/locations/us-east4/queues/loader-mapper""
        debug_error_string = "{"created":"@1567636043.780459442","description":"Error received from peer ipv4:74.125.141.95:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":""x-goog-request-params" header is either missing or misformatted. "x-goog-request-params" must contain "queue.name=projects/gecko-ops-sandbox/locations/us-east4/queues/loader-mapper"","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/google/cloud/tasks_v2beta3/gapic/cloud_tasks_client.py", line 640, in update_queue
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 "x-goog-request-params" header is either missing or misformatted. "x-goog-request-params" must contain "queue.name=[redacted]"

Fortunately creating the Queue object resolves this but if the dict is not supported then the annotation should be updated and ideally there should be a nice error that lets the API user know that a dict isn't supported. Right now the AttributeError exception for accessing .name is silently passed but perhaps should be raised.

[tseaver]

@alexcwatt I'm not sure of the semantics of calling the UpdateQueue API passing a Queue instance without any attributes set other than name. Normally, the point of calling the API would be to actually update one or more of the other writable attributes (rate_limits, retry_config, stackdriver_logging_config).

All that aside, I can see the source of the exception you are reporting. See: https://github.com/googleapis/google-cloud-python/blob/9e8bbb8ea3933dc0fc6a15da123476c12a9bb7e3/tasks/google/cloud/tasks_v2beta3/gapic/cloud_tasks_client.py#L613-L627

The code there successfully constructs an UpdateQueueRequest using the queue dictionary, but fails to create the routing metadata header: that bit expects an actual Queue message, not an equivalent dictionary.

@lukesneeringer, @busunkim96 ISTM that any "message" arguments advertised as taking equivalent dictionaries ought to marshal such dictionaries into the equivalent message before use (even before passing to another message constructory). We can't fix this locally: it needs to happen in the gapic-generator repository.

I guess as an alternative, we could make all that icky generated try: except: code willing to handle dict-key access as a fallback. :(

[alexcwatt]

@tseaver Yeah, this evening I was just trying to update the retry policy for a queue I had created earlier. For the sake of pinpointing the precise issue and noting that the exception isn't a function of any of the other arguments, and only has to do with the dictionary, I created the example that I did, but in my usage of the API earlier I used update_queue to modify the retry policy.

Prior to this work (last week), I had used create_queue and that accepted a dictionary just fine, as the docs suggested. I was then a little surprised when the dictionary didn't work the same way for update_queue despite the same typing signature Union[Queue, dict].

[preston-hf]

One of the big reasons to use this API is that it seems to be idempotent whereas the create_queue method returns an exception if it already exists. I just want a way to ensure a queue exists with a certain config, update_queue seems to fit the bill according to the docs:

This method creates the queue if it does not exist and updates the queue
if it does exist.

Thanks for the workaround

[busunkim96]

Hmm, from a closer read this looks like it needs to be addressed in the generator. @software-dov could you take a look?

[engelke]

Does this issue still exist in current version(s) of the library? Please comment if so, for fix, or close as obsolete.

[busunkim96]

This was taken care of in googleapis/gapic-generator-python#401 and is not an issue in the latest version of the library.