feat: add wrapper for reading table data using Storage API

[alvarowolfx]

Add support for easily reading Tables using the BigQuery Storage API instead of the BigQuery API. This will provide increased performance and reduced memory usage for most use cases and will allow users to keep using the same interface as they used to use on our main library or fetch data directly via a new veneer on BigQuery Storage Read API

[alvarowolfx]

Some early results:

1. SELECT repository_url as url, repository_owner as owner, repository_forks as forks FROM `bigquery-public-data.samples.github_timeline` where repository_url is not null LIMIT 300000

• fetchGetQueryResultsI: 31.135s 🔴
• fetchStorageAPI: 20.033s ⬆️ 36% speedup

2. SELECT repository_url as url, repository_owner as owner, repository_forks as forks FROM `bigquery-public-data.samples.github_timeline` where repository_url is not null LIMIT 1000000

• fetchGetQueryResultsI: 1:32.622 (m:ss.mmm) 🔴
• fetchStorageAPI: 1:07.363 (m:ss.mmm) ⬆️ 27% faster

3. SELECT name, number, state from `bigquery-public-data.usa_names.usa_1910_current

• fetchGetQueryResults: 5:00.514 (m:ss.mmm) 🔴
• fetchStorageAPI: 3:20.987 (m:ss.mmm) ⬆️ 33% faster

[shollyman]

minor feedback thus far:

• it might be good to pull out the logger refactor to it's own PR, was surprised to see managedwriter in scope for changes
• This appears to be only the direct table read bit, are you landing query functionality in a followup, or is this PR going to keep growing?
• Consider testing using a CTAS that leverages generate_array or its cousins, should allow you much larger test data without hawving to stream it all in from the test env

[alvarowolfx]

minor feedback thus far:

• it might be good to pull out the logger refactor to it's own PR, was surprised to see managedwriter in scope for changes
• This appears to be only the direct table read bit, are you landing query functionality in a followup, or is this PR going to keep growing?
• Consider testing using a CTAS that leverages generate_array or its cousins, should allow you much larger test data without hawving to stream it all in from the test env

@shollyman

1. good point, I'll split that.
2. The query functionality lives in the main bq client repo: feat: use Storage Read API for faster data fetching nodejs-bigquery#1368
3. I'll check to use that for the integration tests.

[shollyman]

Approving for now, but please get other reviewer comments to a reasonable resolution before submitting.

[alvarowolfx]

@leahecole answered some other comments here that I missed. I'll try to address then later this week

[alvarowolfx]

Sorry missed this one:

• when using bq storage read, if the query is ordered, the service side only returns one stream, so results can be fetched with the correct order.
• We can add tests for that, I added on the Go side, but missed here.
• Do you have an good way of testing this ? just calling stream.pause ? if you have an example of that would be nice.

[alvarowolfx]

yeah, I'll add a test with more rows and use generation functions like @shollyman recommended

[leahecole]

Got it. Re third bullet point, something like this test I am working on in gax - in this test in gax, I have two streams piped together (I'm using pipeline) - attemptStream is the stream that's making the request to the server, and userStream the consuming stream. They are being piped together

Some things to note

• Idk if you'd need the error checking I have here - I'm specifically writing a retry test so the .on('error') stuff may be irrelevant - I'm also firing errors using the showcase server at very specific points in the data stream (mainly after the 85th item)
• This test has listeners for data on both the attemptStream and userStream - each of those appends the data content to an array. At the end, the two arrays should be the same length. I think a test of yours could use pipeline to connect streams and check that the lengths agree at the end.
• If you experiment with this test and can't find a way to ever make it fail/be confident it's doing anything, don't sweat it. It may be more important on the retries case and when I originally wrote this comment I was very much in the weeds with that CBT stuff
• the pausing happens in the user stream, but not in the attemptStream. This forces data to build up in the buffer. You could also change that userStream.on('data') bit to only pause when results2 is a certain length, but I think pausing every time is more of a surefire way to force that buffer to fill.

let results: string[] = [];
let results2: string[] = [];

attemptStream.on('data', (data: {content: string}) => {
    results.push(data.content);
  });
  attemptStream.on('end', () => {
    assert.strictEqual(results.length, 100);
  });

  attemptStream.on('error', (e: GoogleError) => {
    assert.strictEqual(e.code, 13);
  });
  userStream.on('data', (data: {content: string}) => {
    results2.push(data.content);
    userStream.pause();
    setTimeout(() => {
      userStream.resume();
    }, 100);
  });
  userStream.on('end', () => {
    assert.strictEqual(results.length, 100);
    assert.strictEqual(results2.length, 100);
  });
  userStream.on('error', (e: GoogleError) => {
    // assert.strictEqual(results2.length, 85)
    assert.strictEqual(results.length, 85);

    assert.strictEqual(e.code, 14);
  });

[alvarowolfx]

Added some more comments, address some minor issues and pushed a new test using CTA and generate_array to test it with more data cc @shollyman @leahecole

[alvarowolfx]

Tried to use pipeline, but is meant to be used when we have a destination. In this case where, we are just applying a bunch of transforms and we don't know the destination beforehand.

The error that I got:

  TypeError [ERR_INVALID_ARG_TYPE]: The "streams[stream.length - 1]" property must be of type function. Received an instance of ArrowRecordBatchTransform

[alvarowolfx]

Errors are handled by the consumer of the stream and when used internally like here, we handle the errors.