Wrong protobuf type in ClusterControllerClient.diagnose_cluster

[turbaszek]

Environment details

• OS type and version: MacOS 10.15.4
• Python version: 3.6.8
• pip version: pip --version
• google-cloud-dataproc version: 1.0.0

Steps to reproduce

1. Create cluster
2. Initialize ClusterControllerClient
3. run diagnose_cluster on the cluster
4. use .result() to wait for the operation to complete

Code example

client = ClusterControllerClient()
operation = client.diagnose_cluster(
    project_id=project_id,
    region=region,
    cluster_name=cluster_name,
    retry=retry,
    timeout=timeout,
    metadata=metadata,
)
operation.result()

Stack trace

  File "~/airflow/airflow/providers/google/cloud/hooks/dataproc.py", line 387, in diagnose_cluster
    operation.result()
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/future/polling.py", line 125, in result
    self._blocking_poll(timeout=timeout)
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/future/polling.py", line 104, in _blocking_poll
    retry_(self._done_or_raise)()
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/future/polling.py", line 83, in _done_or_raise
    if not self.done():
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/operation.py", line 169, in done
    self._refresh_and_update(retry)
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/operation.py", line 158, in _refresh_and_update
    self._set_result_from_operation()
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/operation.py", line 131, in _set_result_from_operation
    self._result_type, self._operation.response
  File "~/.virtualenvs/airflow/lib/python3.6/site-packages/google/api_core/protobuf_helpers.py", line 71, in from_any_pb
    any_pb.__class__.__name__, pb_type.__name__
TypeError: Could not convert Any to Empty

I tried to debug this code and use google.protobuf.any_pb2.Any but then I get TypeError: Could not convert Any to Any.

[turbaszek]

@tseaver @busunkim96 would you mind taking a look? I can help to solve it but would like to be sure that real bug not only me :)

[busunkim96]

@bradmiro Could you take a look?

@turbaszek Did this behavior start recently?

[turbaszek]

@turbaszek Did this behavior start recently?

It's the first time I used this functionality so I cannot help here

[bradmiro]

Thanks @turbaszek for raising this, I was able to reproduce this in a non-Airflow environment as well. I'll take a look.

This issue appears to go as far back as at least 0.7.0 (I didn't test earlier versions than this).

[turbaszek]

Thanks @bradmiro !

[busunkim96]

The trace is very similar googleapis/google-cloud-python#9150, so perhaps this is the same issue?

I no longer remember how that was resolved, so I'll need to dig through my email a bit more.

[busunkim96]

I think the cause is a mismatch between the annotated response/metadata type for this LRO and what is returned by the API https://github.com/googleapis/googleapis/blob/a7ed6d42ebf4f4b9862c94e096d37aef92728212/google/cloud/dataproc/v1/clusters.proto#L100-L118

Opened internal PR 321053288.

[busunkim96]

A Dataproc engineer confirmed that the current annotation is incorrect. This should be fixed once the library is regenerated with the tweaked protos.

Lowering the priority since this only prevents retrieval of the response object.

ETA for a new release is by the end of this week.

[bradmiro]

Thanks @busunkim96!

[turbaszek]

Thank you @busunkim96 !