BigQuery: Client.list_rows should prefetch the first page and populate total_rows and other attributes

[bits01]

https://googlecloudplatform.github.io/google-cloud-python/latest/_modules/google/cloud/bigquery/table.html#Table.fetch_data

Return value doc:

:returns: Iterator of row data :class:`tuple`s. During each page, the
                  iterator will have the ``total_rows`` attribute set,
                  which counts the total number of rows **in the table**
                  (this is distinct from the total number of rows in the
                  current page: ``iterator.page.num_items``).

The iterator does not have the total_rows attribute or a page attribute either.

Can fetch_data() get a count of all rows or the only way is to iterate and count?

[bits01]

QueryResults.fetch_data() too.

https://googlecloudplatform.github.io/google-cloud-python/latest/_modules/google/cloud/bigquery/query.html#QueryResults.fetch_data

[lukesneeringer]

Summoning @tswast since he has been in this code a lot recently and probably knows the answer off hand. :-)

[tswast]

There's definitely a pages property. Not sure about page.

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/99c4498339e8cddb628962215e7fa7953a77c957/bigquery/tests/system.py#L296-L299

Looking at https://github.com/GoogleCloudPlatform/google-cloud-python/blob/99c4498339e8cddb628962215e7fa7953a77c957/core/google/api/core/page_iterator.py#L248, I think you're right that the docs need to be updated.

[dhermes]

@tswast The iterator docs are great, they just got unpublished when the module moved from google.cloud.iterator to google.api.core.page_iterator.

/cc @jonparrott

[bits01]

#4153 is related.

[tswast]

The iterator is using

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/e6cc4b4a6486971f910139d766c7807a588ea53b/bigquery/google/cloud/bigquery/_helpers.py#L507-L522

which appears to be populating total_rows. I'll investigate further and make sure our system tests are checking for this property.

[bits01]

It works on QueryResult, but not on the iterator returned by the destination table fetch_data():
AttributeError: 'HTTPIterator' object has no attribute 'total_rows'

[tswast]

My guess as to what is happening is that total_rows only gets populated after you start iterating through, because it gets that information from the first page of results.

I agree that this is confusing behavior.

@jonparrott Do you think we could prefetch the first page when we create the iterator?

[bits01]

Indeed. Once you start iterating total_rows shows up. Should be made consistent with QueryResult, assuming that QueryResult even needs to exist at all.

[theacodes]

@jonparrott Do you think we could prefetch the first page when we create the iterator?

That's a better question for @dhermes, who designed the iterator.

[dhermes]

Yes. The easiest way to do this would be to just subclass the relevant iterator class and over-ride the _next_page method.

Or you could monkey-patch the _next_page method on your instances. This way you wouldn't have to subclass both the HTTP and gRPC iterators, but then you have the problem of figuring out how to fetch the first page via HTTP or gRPC.

[tseaver]

@tswast Is this really a P1 issue? If so, we need to get it into SLO, i.e., fix it ASAP. If not, can you please switch it to P2?

[tswast]

We can bump it down. I put it at P1 thinking it might be a breaking change for GA, but then forgot about it.

[chemelnucfin]

I'm actually going to track it through here for the time being.
Feature Requests

[ashutoshkumars]

IMHO, total_rows should be accessible irrsepective of whether we start iterating the page or not.

[tswast]

@ashutoshkumars I agree, which is why I opened this feature request.

I have changed my mind on the implementation details. I don't think we should prefetch the first page, because that could be an unnecessary API request.

Instead, total_rows is available both in the query results API response and the get table API response. It could be copied to the object in most cases. The only problem with that is with table references. We would still need to make an API request to get the table size since it's not available in just a reference.

[tswast]

#7622 should address this request in the client for most cases.