-
Notifications
You must be signed in to change notification settings - Fork 210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bigquery:createWriteStream() should propogate insertErrors #8
Comments
Well, I tried something simple, and it worked! Can you try replacing the |
From @chrishiestand on September 29, 2017 18:23 @stephenplusplus thanks for the quick reply, it's so nice that you regularly get back so quickly. So I tried your suggestion (side note, I often wondered why gcloud-node used the complete event instead of the standard finish): writeStream.on('finish', resolve) And it didn't work for me. So my guess is that this is one of those "hard to reproduce" features of this issue. I also checked the BQ UI and double-checked that the rows didn't stream in later. Give it another try with |
From @chrishiestand on October 2, 2017 8:32 I've now discovered that the issue doesn't simply require many parallel streams, it can also be reproduced with many serial streams. I've pushed some new commits to the test repo - tagged If I stream 100 streams, one at a time ( If I stream 100 streams, two at a time ( It seems unlikely that I'd be the first to hit an issue like this if it were the same for everyone. I can't be the only one setting up parallel or serial streams to the same table. Perhaps there is some issue with the how I am wrapping the stream in a promise? Or something else specific to my environment? But as far as I can tell, the code ought to work as written. |
Looked into this, and came up with this: sometimes, the data just fails to insert. This isn't necessary, but using the This also gives us what turned out to be the key component of this issue, the logging of the
So, a couple questions:
|
From @chrishiestand on October 2, 2017 19:4 Thank you very much @stephenplusplus. I've tagged your latest version of the reproduction project as Now I am finally seeing rate limit errors with ...
DONE 150
DONE 150
TypeError: Cannot read property 'outputRows' of undefined
at Job.job.on.job (/private/tmp/gcloud-node-bigquery-manystreams-bug/test/many-streams.test.js:158:37)
at emitOne (events.js:115:13)
at Job.emit (events.js:210:7)
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/operation.js:188:10
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/bigquery/src/job.js:331:5
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/service-object.js:272:5
at Object.handleResp (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/util.js:135:3)
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/util.js:465:12
at Request.onResponse [as _callback] (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/retry-request/index.js:160:7)
at Request.self.callback (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:186:22)
at emitTwo (events.js:125:13)
at Request.emit (events.js:213:7)
at Request.<anonymous> (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:1163:10)
at emitOne (events.js:115:13)
at Request.emit (events.js:210:7)
at Gunzip.<anonymous> (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:1085:12)
at Object.onceWrapper (events.js:314:30)
at emitNone (events.js:110:20)
at Gunzip.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1059:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
{ kind: 'bigquery#job',
etag: '"Z8nD8CIuj_TqmPrS_MNV-O9x2rU/RoVaRZnuqFcy-MsZbrpdHrL7cy4"',
id: 'my-project:job_xwxZ4JQ7ESbaCf9Aw0xkFyfVcxl7',
selfLink: 'https://www.googleapis.com/bigquery/v2/projects/my-project/jobs/job_xwxZ4JQ7ESbaCf9Aw0xkFyfVcxl7',
jobReference:
{ projectId: 'my-project',
jobId: 'job_xwxZ4JQ7ESbaCf9Aw0xkFyfVcxl7' },
configuration:
{ load:
{ destinationTable: [Object],
sourceFormat: 'NEWLINE_DELIMITED_JSON' } },
status:
{ state: 'DONE',
errorResult:
{ reason: 'rateLimitExceeded',
location: 'table.write',
message: 'Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors' },
errors: [ [Object] ] },
statistics:
{ creationTime: '1506969989800',
startTime: '1506969989869',
endTime: '1506969989869' },
user_email: '[email protected]' }
/private/tmp/gcloud-node-bigquery-manystreams-bug/test/many-streams.test.js:40
throw error;
^
TypeError: Cannot read property 'outputRows' of undefined
at Job.job.on.job (/private/tmp/gcloud-node-bigquery-manystreams-bug/test/many-streams.test.js:163:39)
at emitOne (events.js:115:13)
at Job.emit (events.js:210:7)
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/operation.js:188:10
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/bigquery/src/job.js:331:5
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/service-object.js:272:5
at Object.handleResp (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/util.js:135:3)
at /private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/@google-cloud/common/src/util.js:465:12
at Request.onResponse [as _callback] (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/retry-request/index.js:160:7)
at Request.self.callback (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:186:22)
at emitTwo (events.js:125:13)
at Request.emit (events.js:213:7)
at Request.<anonymous> (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:1163:10)
at emitOne (events.js:115:13)
at Request.emit (events.js:210:7)
at Gunzip.<anonymous> (/private/tmp/gcloud-node-bigquery-manystreams-bug/node_modules/request/request.js:1085:12)
at Object.onceWrapper (events.js:314:30)
at emitNone (events.js:110:20)
at Gunzip.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1059:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
✖ test/many-streams.test.js exited with a non-zero exit code: 7
0 tests passed [11:46:30]
1 uncaught exception
npm ERR! Test failed. See above for more details. |
From @chrishiestand on October 2, 2017 19:40 It's unclear to me which rate limit I am hitting. It looks like I shouldn't be hitting any of the streaming insert limits: My guess is I'm hitting the "Maximum rate of table update operations: 1 operation every 2 seconds" query. But it isn't obvious to me that this applies to streaming inserts too. Does this mean that I have to wait 2 seconds between the end of one streaming request and the start of another? Or two seconds after I start one, I can start another, even if the first hasn't finished? |
From @chrishiestand on October 3, 2017 1:23 Regarding solving this issue: I believe that errors like quotas/rate limits ought to throw an error or emit an error event. But, if for some reason behavior cannot be changed, at the very least they should be logged to the error console. The lack of any errors being thrown in the typical and documented fashion, along with a lack of documentation of how to see such errors at https://googlecloudplatform.github.io/google-cloud-node/#/docs/google-cloud/0.56.0/bigquery/table?method=createWriteStream has burned a lot of my time. Let's spare the next people from this problem :-) |
Fixes #8 **This would be a breaking change** Currently, if you call `table.createWriteStream()`, the stream will emit "complete" with a Job object. You can then use that Job object to assign event handlers, as usual with an LRO. Judging from #8, I'm not sure this pattern is clear to the users. And aside from that, I'm not sure what we're doing is ideal, either. Ideally, we would not emit "complete" until the Job finished, instead of making the user do it manually. This PR shows how easy it would be for us to implement. However, I *believe* this would be a breaking change, although it's hard to say if it's actually a bugfix. I'm leaving this open for discussion on how we should handle this. Please share your thoughts! --- ### Before ```js table.createWriteStream(...) .on('complete', job => { job .on('error', err => { // Job failed. }) .on('complete', () => { // Job completed successfully. }); }); ``` ### After ```js table.createWriteStream(...) .on('error', err => { // Job failed. }) .on('job', job => { // `job` is a Job object. }) .on('finish', () => { // Job completed successfully. }); ``` And, `complete` still works to combine those. I added the "finish" event, because that's the natural writable stream closing event. ```js table.createWriteStream(...) .on('error', err => { // Job failed. }) .on('complete', job => { // `job` is a Job object. // Job completed successfully. }); ```
From @chrishiestand on September 29, 2017 4:30
The gist is bigquery sometimes silently drops one or more streams when many streams are used in parallel (42 in this example). If I'm hitting a quota an error ought to be thrown, but no errors are thrown the program runs as expected but in the end there are rows missing from the bigquery data.
If there is a bug, it might be in gcloud-node, or it might be in the bigquery api. Both seem less likely than me having made a mistake, so I hope you can find something I've done wrong.
The tricky part is the bug doesn't always reproduce. When the bug does reproduce
n
streams of sizes
are dropped so bigquery will haven
*s
missing rows. So ifn=2
ands=150
bigquery will be missing 300 rows. In other words, the problem does not appear to be that a subset of stream data is missing, but rather one or more entire streams are missing.This seems to reproduce reliably sometimes, and other times it reliably does not reproduce. To try and get the opposite result, try again later and/or change the stream load with the env variables.
This small bug reproduction project https://github.com/chrishiestand/gcloud-node-bigquery-manystreams-bugis the result of troubleshooting missing big query data in a production system where 50 streams are processed in parallel.
Below is a screenshot of the reproduction repository showing a reproduction of the issue.
In contrast, if I reduce the number of streams from 42 to 10, the tests pass as below:
Environment details
Steps to reproduce
Please go here for a reproduction project: https://github.com/chrishiestand/gcloud-node-bigquery-manystreams-bug
Copied from original issue: googleapis/google-cloud-node#2635
The text was updated successfully, but these errors were encountered: