GAX support: Python3 JSON load errors

[tseaver]

/cc @bjwatson, @tbetbetbe

Recent grcpcio releases actually install and work on Python 3.4+, so I'm trying to get our system tests running with GAX enabled. Both pubsub and logging show errors like:

======================================================================
ERROR: test_create_metric (logging_.TestLogging)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/system_tests/logging_.py", line 156, in test_create_metric
    self.assertFalse(metric.exists())
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/logging/metric.py", line 128, in exists
    client.metrics_api.metric_get(self.project, self.name)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/logging/client.py", line 117, in metrics_api
    generated = GeneratedMetricsAPI()
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/google/logging/v2/metrics_service_v2_api.py", line 168, in __init__
    __name__, 'metrics_service_v2_client_config.json'))
  File "/opt/Python-3.4.3/lib/python3.4/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

To reproduce:

$ git clone git@github:GoogleCloudPlatform/gcloud-python
$ cd gcloud-python
$ tox -e system-tests3 --notest
$ GCLOUD_ENABLE_GAX=1 .tox/system-tests3/bin/python system_tests/run_system_test.py --package=pubsub # or logging

[bjwatson]

@geigerj Can you take a look?

[geigerj]

We currently develop/test GAX and GAPIC only on Python 2.7. @bjwatson @tbetbetbe -- we should discuss support for Python 3. I'll update this afterward.

[bjwatson]

Thanks @geigerj. I just noticed that, too.

@tseaver Do you need to support python3 now? Do you have an alternative to using the GAPIC surface for now?

[tseaver]

@bjwatson We have JSON-over-HTTP support as the default for both pubsub and logging (including Python2). We require that the user explicitly enable gax/grpc with an environment variable (which I forgot to show in the "to reproduce" example until just now). The packages all install cleanly on Python3, which is nice; getting them to work cleanly there would be nicer. :) But it isn't an emergency.

[bjwatson]

@tseaver Cool, thanks for the context.

[tseaver]

@rok987 this issue.

[bjwatson]

@tseaver Can you see if this is still an issue? Thanks!

[tseaver]

The 0.8.0 wrappers for pubsub/logging still don't work on Py3k. See this gist for details.

[geigerj]

I've uploaded 0.8.1 wrappers for logging and pubsub (with other APIs to follow) that should fix the JSON load error. Please try again and update.

(I wasn't able to repro; on the last step, I'm getting a Missing variables: GOOGLE_APPLICATION_CREDENTIALS even after following the instructions at https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/CONTRIBUTING.rst#running-system-tests and setting that environment variable explicitly.)

[tseaver]

@geigerj did you set the GCLOUD_TESTS_PROJECT_ID variable?

[geigerj]

Yup, both GCLOUD_TESTS_PROJECT_ID and GOOGLE_APPLICATION_CREDENTIALS are set.

[tseaver]

With the 0.8.1 wrappers, both pubsub and logging system tests pass for me on Python 3.4 with GCLOUD_ENABLE_GAX set:

$ GCLOUD_ENABLE_GAX=1 .tox/system-tests3/bin/python system_tests/run_system_test.py --package=pubsub
D0810 15:28:12.477279068    6716 ev_posix.c:106]             Using polling engine: poll
test_create_subscription_defaults (pubsub.TestPubsub) ... ok
test_create_subscription_w_ack_deadline (pubsub.TestPubsub) ... ok
test_create_topic (pubsub.TestPubsub) ... ok
test_fetch_delete_subscription_w_deleted_topic (pubsub.TestPubsub) ... ok
test_list_subscriptions (pubsub.TestPubsub) ... ok
test_list_topics (pubsub.TestPubsub) ... ok
test_message_pull_mode_e2e (pubsub.TestPubsub) ... ok
test_subscription_iam_policy (pubsub.TestPubsub) ... ok
test_topic_iam_policy (pubsub.TestPubsub) ... ok

----------------------------------------------------------------------
Ran 9 tests in 21.146s

OK
$ GCLOUD_ENABLE_GAX=1 .tox/system-tests3/bin/python system_tests/run_system_test.py --package=logging
D0810 15:28:55.697065457    6722 ev_posix.c:106]             Using polling engine: poll
test_create_metric (logging_.TestLogging) ... ok
test_create_sink_bigquery_dataset (logging_.TestLogging) ... ok
test_create_sink_pubsub_topic (logging_.TestLogging) ... ok
test_create_sink_storage_bucket (logging_.TestLogging) ... ok
test_list_metrics (logging_.TestLogging) ... ok
test_list_sinks (logging_.TestLogging) ... ok
test_log_struct (logging_.TestLogging) ... _has_entries. Trying again in 1 seconds...
ok
test_log_struct_w_metadata (logging_.TestLogging) ... _has_entries. Trying again in 1 seconds...
ok
test_log_text (logging_.TestLogging) ... _has_entries. Trying again in 1 seconds...
ok
test_log_text_w_metadata (logging_.TestLogging) ... _has_entries. Trying again in 1 seconds...
ok
test_reload_metric (logging_.TestLogging) ... ok
test_reload_sink (logging_.TestLogging) ... ok
test_update_metric (logging_.TestLogging) ... ok
test_update_sink (logging_.TestLogging) ... ok

----------------------------------------------------------------------
Ran 14 tests in 32.349s

OK

[tseaver]

@geigerj Can you show the failing output? And the result of $ env | grep GCLOUD?

[geigerj]

Excellent, I'm glad the tests are passing now.

I've figured my problem out. It was my fault: the path that I stored in GOOGLE_APPLICATION_CREDENTIALS was wrong. (Although, the error message Missing variables: GOOGLE_APPLICATION_CREDENTIALS doesn't distinguish between the variable being unset and the credentials file not being found.)

[tseaver]

Testing envvar issue tracked at #2084.