Is createQueryStream removed or is it just not working in v2

[RubenLaube-Pohto]

I was using createQueryStream in the 1.3 version. After updating to v2 the call does not work correctly anymore. It seems to start but then something goes wrong (see pic). The query string is passed in and is valid as it can be logged before the call and v1.3 works.

The v2 documentation does not mention createQueryStream anymore. Does that mean it was removed? The removal is not marked as a breaking change in the changelog.

Environment details

• OS: Running in GAE
• Node.js version: 10
• npm version:
• @google-cloud/bigquery version: 2

Steps to reproduce

1. Use createQueryStream in v1.3
2. Update to v2. Call seems to start but soon causes errors.

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[nynexman4464]

This is really frustrating after #120 was closed and not announced what release it would appear in.

[jkwlui]

This is really frustrating after #120 was closed and not announced what release it would appear in.

@nynexman4464 it was released as part of v2.0.0. You can view the release notes here. Sorry for the confusion!

As for the createQueryStream method, we haven't removed it in v2, rather there was a mistake in the generation of the docs that causes it to be missing from the documentation. The signature of the method should have remained unchanged between v1.3 and v2.0.

To help us figure out what is wrong, @RubenLaube-Pohto could you provide some code showing how you're calling createQueryStream?

[nynexman4464]

This simple example breaks with the "A SQL query string is required" message

  let bigquery = new BigQuery({
    projectId,
    keyFilename: /*omit*/,
  });
  const query = `
#standardSQL
SELECT country_code, name
FROM \`countries\`;`;
  bigquery.createQueryStream({ query }).on('error', error => {
      console.error(error);
    }).on('data', row => {
      console.log(row);
    }).on('end', () => {
      console.log('end');
    });

I ended up switching to createQueryJob and getQueryResultsStream combo which seems to work.

[RubenLaube-Pohto]

I was unfortunately unable to recreate the issue. I tried to recreate the basic idea of my app as simply as I could but it did run without errors. The actual app is a bit complex, maybe overly so. The issue I had instead was that on GAE the smallest instance ran out of memory with the example. My deployed app would get that too but I thought it would just be the complexity of the app.

I was able to run my deployed app locally (Ubuntu 16.04 + node10 + bq v2 with Datastore saves disabled) but the deploy is the one breaking if using node10 + bq v2.

const express = require('express');
const { BigQuery } = require('@google-cloud/bigquery');

const app = express();
const client = new BigQuery({
    projectId
});

const validQuery;
 
function runQuery() {
    return new Promise((resolve, reject) => {
        client.createQueryStream(validQuery)
            .on('error', (err) => {
                console.error(err);
                reject(err);
            })
            .on('data', (data) => {
                console.log(data);
            })
            .on('end', () => {
                console.log('complete');
                resolve();
            });
    });
}

app.get('/', 
    async (req, res, next) => {
        await runQuery();
        res.sendStatus(200);
    }
);

app.listen(process.env.PORT || 8080, () => {
    console.log('server started');
});

[jkwlui]

v2.0.5 has just been released which should resolve this issue, sorry for the trouble!

[RubenLaube-Pohto]

Just updated and the issue is not showing up anymore. Thanks! 👍

[oliverwoodings]

@kinwa91 v2.0.5 didn't fix this for me unfortunately. I ended up following what @nynexman4464 did and switched to using createQueryJob + getQueryResultsStream, which does work. I am unable to provide a reproduceable test case, but i did do some code diving. This is what i found:

• My code calls bigquery.createQueryStream() with a query. Note that this function has been wrapped by @google-cloud/paginator
• This calls queryAsStream_ (https://github.com/googleapis/nodejs-bigquery/blob/master/src/index.ts#L1515), which ends up calling createQueryJob
• After we have a job, we call job.getQueryResults (https://github.com/googleapis/nodejs-bigquery/blob/master/src/index.ts#L1505).
• After getting the job status, the job i'm running hasn't completed, so we call back with no rows and no page token (https://github.com/googleapis/nodejs-bigquery/blob/master/src/job.ts#L418-L428)
• The callback given to getQueryResults that is called at this point is actually the results handler from @google-cloud/paginator (https://github.com/googleapis/nodejs-paginator/blob/master/src/index.ts#L315)
• Because there is a nextQuery given to onResultSet of the paginator, and there are still rows to send, the paginator executes the request again (https://github.com/googleapis/nodejs-paginator/blob/master/src/index.ts#L332-L335)
• makeRequest in the paginator calls originalMethod, which in this case is actually BigQuery.queryAsStream_, which means it tries to create an entire new query job, instead of just polling for more results?!

This is as far as I got before resorting to manually using createQueryJob and getQueryResultsStream.

[stephenplusplus]

Thanks for digging-- let me take a look 🔎