Add support for query parameters #2776

tseaver · 2016-11-29T21:24:04Z

Closes #2551.

Holds name, type, and value for scalar query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

Holds name, type, and value for array query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

Holds name, types, and values for Struct query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

It is just a list comprehension, really.

bigquery/google/cloud/bigquery/__init__.py

@@ -28,3 +28,6 @@
 from google.cloud.bigquery.dataset import Dataset
 from google.cloud.bigquery.schema import SchemaField
 from google.cloud.bigquery.table import Table
+from google.cloud.bigquery._helpers import ArrayQueryParameter


bigquery/google/cloud/bigquery/_helpers.py

+    """Named / positional query parameters for scalar values.
+
+    :type name: str or None
+    :param name: Parameter name, used via `@foo` syntax.  If None, the


bigquery/google/cloud/bigquery/_helpers.py

+    @classmethod
+    def positional(cls, type_, value):
+        """Factory for positional paramters.
+


bigquery/google/cloud/bigquery/_helpers.py

+        :param value: the scalar parameter value.
+
+        :rtype: :class:`ScalarQueryParameter`
+        :returns: instance w/o name


bigquery/google/cloud/bigquery/_helpers.py

+        """Factory for positional paramters.
+
+        :type sub_parms: tuple of :class:`ScalarQueryParameter`
+        :param sub_parms:s the sub-parameters for the struct


bigquery/unit_tests/test__helpers.py

+
+    @staticmethod
+    def _get_target_class():
+        from google.cloud.bigquery._helpers import AbstractQueryParameter


bigquery/google/cloud/bigquery/_helpers.py

+    """Named / positional query parameters for scalar values.
+
+    :type name: str or None
+    :param name: Parameter name, used via `@foo` syntax.  If None, the


bigquery/google/cloud/bigquery/_helpers.py

+class ScalarQueryParameter(object):
+    """Named / positional query parameters for scalar values.
+
+    :type name: str or None


bigquery/google/cloud/bigquery/_helpers.py

+
+    :type name: str or None
+    :param name: Parameter name, used via `@foo` syntax.  If None, the
+                 paramter can only be addressed via position (`?`).


bigquery/google/cloud/bigquery/_helpers.py

+        """
+        name = resource.get('name')
+        type_ = resource['parameterType']['type']
+        value = resource['parameterValue']['value']


bigquery/google/cloud/bigquery/_helpers.py

+                 :class:`datetime.date`.
+    :param value: the scalar parameter value.
+    """
+    def __init__(self, name, type_, value):


bigquery/google/cloud/bigquery/job.py

+                query_parameter.to_api_repr()
+                for query_parameter in self._query_parameters
+            ]
+            if self._query_parameters[0].name is None:


bigquery/unit_tests/test_job.py

@@ -1287,12 +1287,31 @@ def _verifyIntegerResourceProperties(self, job, config):
        else:
            self.assertIsNone(job.maximum_bytes_billed)

+    def _verifyUDFResources(self, job, config):


bigquery/unit_tests/test_job.py

+    def test_begin_w_named_query_parameter(self):
+        from google.cloud.bigquery._helpers import ScalarQueryParameter
+        query_parameters = [ScalarQueryParameter('foo', 'INT64', 123)]
+        PATH = 'projects/%s/jobs' % self.PROJECT


docs/bigquery_snippets.py

-        if token is None:
-            break
-        jobs, token = client.list_jobs(page_token=token)  # API request
+    job_iterator = client.list_jobs()   # API request


docs/bigquery_snippets.py

@@ -490,6 +486,30 @@ def client_run_sync_query(client, _):


 @snippet
+def client_run_sync_query_w_param(client, _):
+    """Run a synchronous query using a query parameter"""
+    QUERY_W_PARM = (


Addresses: #2776 (comment)

Addresses: #2776 (comment).

Addresses: #2776 (comment)

bigquery/google/cloud/bigquery/_helpers.py

+
+    :type type_: str
+    :param type_: name of parameter type.  One of `'STRING'`, `'INT64'`,
+                  `'FLOAT64'`, `'BOOLEAN'`, `'TIMESTAMP'`, or `'DATE'`.


The back-end raises an exception when passed the 'BOOLEAN' type to define a query parameter. Addresses: #2776 (comment)

bigquery/google/cloud/bigquery/_helpers.py

-            {'name': name, 'type': self.struct_types[name]}
-            for name in self._order
+            {'name': key, 'type': value}
+            for key, value in self.struct_types.items()


bigquery/google/cloud/bigquery/_helpers.py

+                         the class).
+
+        :type value: list of instances of classes derived from
+                :class:`AbstractQueryParameter`.


Addresses: #2776 (comment).

dhermes

LGTM, though I don't understand why you'd have the constructor and the API rep so far apart, especially if no human is every going to use the constructor

tseaver · 2016-12-02T22:50:26Z

especially if no human is every going to use the constructor

I do expect humans to use the constructor, e.g.:

from google.cloud.bigquery import Client
from google.cloud.bigquery import ScalarQueryParameter
from google.cloud.bigquery import StructQueryParameter


birth_state = ScalarQueryParameter('birth_state', 'STRING', 'VA')
death_state = ScalarQueryParameter('death_state', 'STRING', 'MS')
birth_death = StructQueryParameter(
    'birth_death', birth_state, death_state)

client = Client()

query = client.run_sync_query(
    'SELECT * FROM mortality WHERE birth_death = @birth_death')

* Add 'ScalarQueryParameter' class. Holds name, type, and value for scalar query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. * Factor out 'AbstractQueryParameter. * Add 'ArrayQueryParameter' class. Holds name, type, and value for array query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. * Add 'StructQueryParameter' class. Holds name, types, and values for Struct query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. * Add 'QueryParametersProperty' descriptor class. * Add 'query_parameters' property to 'QueryResults' and 'QueryJob'. * Plumb 'udf_resources'/'query_parameters' through client query factories. * Expose concrete query parameter classes as package APIs. Closes googleapis#2551.

avitzavi · 2018-10-31T17:36:18Z

does this mean we can use @parameters in the BigQuery UI ?? I cant seem to figure out how to declare them correctly.

tseaver · 2018-10-31T18:10:06Z

@avitzavi I'm not sure what you are asking. Does the example in the docs help?

avitzavi · 2018-10-31T18:38:55Z

I’m trying to use variables like I do in sql server: “Declare @variable INT = 10” Can I do this in the BigQuery UI?

…

On Oct 31, 2018, at 2:25 PM, Tres Seaver ***@***.***> wrote: @avitzavi I'm not sure what you are asking. Does the example in the docs help? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

tseaver · 2018-10-31T21:30:08Z

The @parameter placeholders added by this PR are used for passing in values from the client to be substituted into a query: they aren't meant to be used that way.

Also, I don't know what you mean by "BigQuery UI" -- is that the https://console.cloud.google.com/ interface?

avitzavi · 2018-10-31T22:10:05Z

Yes it is the console. The Declare/Set methods from SQL are very valuable for me as a data analyst. Seems like a really big missing feature... I create many reports (based off scheduled queries) and not having these parameters makes it much more challenging to do. Especially given the Console's lack of basic editor features like find/replace.

…

Oct 31, 2018, at 5:30 PM, Tres Seaver ***@***.***> wrote: The @parameter placeholders added by this PR are used for passing in values from the client to be substituted into a query: they aren't meant to be used that way. Also, I don't know what you mean by "BigQuery UI" -- is that the https://console.cloud.google.com/ interface? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

tseaver · 2018-10-31T22:34:11Z

@avitzavi This repository is solely about the Python client libraries which wrap the various Google Cloud APIs. Please check out the BigQuery support page for ways to get support for the back-end API itself.

tseaver added 10 commits November 28, 2016 16:50

Add 'ScalarQueryParameter' class.

128db62

Holds name, type, and value for scalar query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

Factor out 'AbstractQueryParameter.

3421d7a

Add 'ArrayQueryParameter' class.

57c61c1

Holds name, type, and value for array query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

Add 'StructQueryParameter' class.

6650c59

Holds name, types, and values for Struct query parameters, and handles marshalling them to / from JSON representation mandated by the BigQuery API. Toward #2551.

Remove '_build_udf_resources' helper.

fcff570

It is just a list comprehension, really.

Add 'QueryParametersProperty' descriptor class.

2e06228

Add 'query_parameters' property to 'QueryResults' and 'QueryJob'.

968582f

Plumb 'udf_resources'/'query_parameters' through client query factories.

52b25ec

Expose concrete query parameter classes as package APIs.

9089907

Add a snippet documenting use of query parameters.

9fd7f76

tseaver added the api: bigquery Issues related to the BigQuery API. label Nov 29, 2016

tseaver assigned daspecster and dhermes Nov 29, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 29, 2016

daspecster reviewed Nov 29, 2016

View reviewed changes

Address code review.

1df82a0

dhermes reviewed Nov 29, 2016

View reviewed changes

daspecster added the PR: Needs to be squashed label Nov 29, 2016

tseaver added 5 commits November 30, 2016 10:58

Document expected types for QueryParametersProperty setter/getter.

6191ca6

Addresses: #2776 (comment)

Use 'collections.OrderedDict' vs. hand-rolling order.

89f5a58

Addresses: #2776 (comment).

PEP8.

a4ff964

Addresses: #2776 (comment).

Correct notation of where API requests occur.

d4d451e

Addresses: #2776 (comment)

s/parm/param/.

0f61970

Addresses: #2776 (comment)

tseaver mentioned this pull request Nov 30, 2016

BigQuery: Add support for query parameters #2551

Closed

blowmage reviewed Dec 1, 2016

View reviewed changes

tseaver mentioned this pull request Dec 1, 2016

Add support for standard SQL 'BOOL' type. #2789

Merged

Update docstring to show standard-SQL 'BOOL' type.

524baf1

The back-end raises an exception when passed the 'BOOLEAN' type to define a query parameter. Addresses: #2776 (comment)

danoscarmike mentioned this pull request Dec 1, 2016

bigquery: support SQL parameters googleapis/google-cloud-node#1787

Closed

dhermes reviewed Dec 2, 2016

View reviewed changes

tseaver added 3 commits December 2, 2016 14:41

Indentation fix.

4b2cb13

Preserve OrderedDict for struct_types.

99bdcd1

Addresses: #2776 (comment).

Normalize '%' formatting.

132741a

Addresses: #2776 (comment).

dhermes approved these changes Dec 2, 2016

View reviewed changes

tseaver merged commit 75c4115 into googleapis:master Dec 2, 2016

tseaver deleted the 2551-bigquery-query_parameters branch December 2, 2016 22:52

tseaver removed the PR: Needs to be squashed label Dec 2, 2016

dhermes mentioned this pull request Dec 5, 2016

Cutting new releases for BigQuery, Language and Core. #2815

Merged

tswast mentioned this pull request Dec 5, 2016

BigQuery samples using query parameters. GoogleCloudPlatform/python-docs-samples#699

Merged

theacodes unassigned daspecster and dhermes Sep 28, 2018

Add support for query parameters #2776

Add support for query parameters #2776

Conversation

tseaver commented Nov 29, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes left a comment

Choose a reason for hiding this comment

tseaver commented Dec 2, 2016

avitzavi commented Oct 31, 2018

tseaver commented Oct 31, 2018

avitzavi commented Oct 31, 2018 via email

tseaver commented Oct 31, 2018

avitzavi commented Oct 31, 2018 via email

tseaver commented Oct 31, 2018